Tooling Go Microservices With Consul Service Discovery and KV Store

Consul, at its core, is a service networking solution. It provides a service mesh solution, network configuration automation, service discovery, a simple Key-Value store, etc. In this article, we will focus on the key-value store (KV store) for our service configuration and one of the core features of consul, service discovery. In a word, we are covering an essential microservice tooling with Consul.

Service Registration and Distributed Configuration /w Consul and Go

TL;DR

GitHub - by-sabbir/consul-kv-discovery

Get the code and run

❯ docker compose up -d --build

That should expose Consul UI at localhost:8500 and the UI should look like below if you browse this link.

Consul Service Discovery with Health Check

If you are more interested in how to structure your code and make it ready for production, let’s dive in…

We will first start with one monolithic main.go file and then break it down into pieces to create an idiomatic go project. First, we need to start the consul service. Let’s create a docker-compose.yml file for that,

Also, we need to add the server configuration, from conf/server.json

❯ mkdir conf/ && curl -o conf/server.json https://raw.githubusercontent.com/by-sabbir/consul-kv-discovery/master/conf/server.json

Now, we are ready to start the consul server,

❯ docker compose up -d consul

Let’s focus on the application, we will be making some design decisions-

• First and foremost, we want this package to be used by all the microservices.
• Service should be registered when the application starts. (and should fail first, but for sake of simplicity let’s ignore this)
• We want to use KV store as an external dependency manager like service host and port, API version, not for security configs ie, password manager, API key, client secret, etc.

Our project structure should look like this,

.
├── cmd
│   └── server
├── conf
├── internal
|   └── api
└── pkg
    └── consul

For consul service interactions, we will be using pkg directory instead of internal as we want this to be reusable for other developers and services as part of go standard project layout.

Now, create a new file pkg/consul/service-discovery.go

This is a fairly simple code containing only two functions

• NewClient — initiates a new consul API client for the given address
• Register — Registers the service to consul service discovery with the given name and predefined tag.

For production, tags are significant to searching services in the Consul UI quickly. Once the service is registered, monitoring the service health is going to be the job of the consul server, line 30-34 denotes that the consul server will check for the service's health every 30 seconds.

We are done with service discovery. Now let’s discuss some dynamic configurations with Consul KV Store.

To integrate the Consul KV, let's look into the documentation a bit to better understand the situation. From the doc, we can see that,

func (c *Client) KV() *KV

So, KV is a pointer receiver to the *api.Client struct. We already have a method NewClient that returns *api.Client and we have initiated a client from the method we can reuse that client if we create a separate repository for KV Store. Let’s do that… create a file pkg/consul/kv-store.go and paste the following lines —

The function NewKVClient reuses our ConsulClient and returns a KV client. For our use case, we only created Get and Put functionality assuming all the keys and values will be stringfor the Store. As we will only read existing config and create/update one.

Finally, it’s time to integrate the solution, Go provides a very powerful function to instantiate init() that runs only once and runs before any other function within the package. We can define multiple init() but all of them will run in order once and once only. We will get the advantage of this feature, as service health checks will be done periodically by the Consul server.

Let’s write just enough code to register the service in the Consul,

If you run the main.goand drop to the console and paste the following —

❯ curl -X GET http://127.0.0.1:8500/v1/catalog/services

It should output something like —

{“consul”:[],”go-service_ms”:[“golang”,”microservice”]}

Let’s clean up the main.go file. We wanted to design the consul functionality as reusable as possible and we managed to do that by using go conventions. But instantiating the services will differ from one to another. For example, one may need to use AWS S3 service, and another one may use AWS Keyspaces. So configuration may vary from service to service. In this case, we need internal a directory to instantiate the consul integrations. Let’s create a file pkg/consul/kv-store.go and paste the following —

Note: The struct ServiceDefinitionshould be populated by viper or any os environment variable parser.

So, our main.go file becomes cleaner and the code base is more manageable. But don’t forget to import the package “_” in front as an anonymous placeholder. It will make sure that the init() function runs once.

_ “github.com/by-sabbir/consul-kv-discovery/internal/consul”

Now, if we run the whole docker compose it should output like this —

❯ docker compose logs -f app
application  | 2022/10/08 21:07:40 service registered:  go-service
application  | 2022/10/08 21:07:40 apigw baseUrl:  apigw.example.com

We have successfully got information from the KV store, let’s check out the new keys —

Get And Put Key Value to Consul

To get the newly created key, go-service make the following request —

❯ curl -X GET http://127.0.0.1:8500/v1/kv/go-service

It should output the query like —

[
   {
      "CreateIndex" : 17,
      "Flags" : 0,
      "Key" : "go-service",
      "LockIndex" : 0,
      "ModifyIndex" : 17,
      "Value" : "MC4wLjAuMDo4MDAw"
   }
]

The value is base64 encoded version of the original message.

We are now able to discover go service from Consul. Also, equipped with the necessary tool to read from and write to Consul KV Store.