Leveraging Consul as Service Discovery, Health Checking, and Key Value (KV) Store in Go Microservices

Before we dig down into our main topics, I want you to know, what Consul is. Consul is a distributed and highly available system. In general, Consul provides four features, they are:

• Service Discovery
• Health checking
• Key Value Store
• Multi data center

In this post I will cover just three main features from that I mentioned above; Service Discovery, Health Checking, and Key Value (KV) Store. Recently, I found that those things can be a useful features who able to solve several cases in system who adopt microservice as architecture. Let’s get started from first and second features.

Service Discovery and Health Checking

Service Discovery talk about how we discover the address of service(s) that we will communicate to. In a modern cloud-base microservice architecture, this is gonna be a serious problem because service instances have dynamically assigned network location. We have to own this feature to automate this. There are two pattern of service discovery, they are:

• Client Side Service Discovery
• Server Side Service Discovery

Client side service discovery, the client determined network location of available service instances and make request across them. First, client get the network location data by querying to service registry, and do load balance request to choose available service to make a request. This following figure below describe how the client side service discovery works:

Client side service discovery pattern

Server side service discovery, the client makes a request via route or a load balancer, and then load balancer will query to service registry and choose the available service to routes the request, this following figure describes how server side discovery pattern works:

Server side discovery pattern

From the two patterns above, we see a part that we never mentioned before, that is Service Registry. It is the most important part in service discovery system, because all the transactions like register and deregister will be stored in it. So let’s do it in Go code.

First, lets assume that our microservices is still at early stage, we only have two core services, they are user service and product service. We start with registering each services to service registry Consul.

register a service to consul agent

Let me explain the three functions one by one. Lets start with port() function. This function simply getting value from environment variable with name “PRODUCT_SERVICE_PORT”, then store the value in p variable. Afterwards, checking the p variable. If its length is 0, in the other hand its value is empty, so we need to set the default value, we hardcoded become “:8100” for a moment. The second one function named hostname(). The objective of this function is to get the host where service running on.

Lets check the interesting one, registerServiceWithConsul(). The objective of this function is to register service to Consul service registry. I’m using library that provided by Hashicorp, you can grab it by running:

github.com/hashicorp/consul/api

From the snippet code that we see above, lets notice line 12 until 34. There are at least seven components that we used to fill to register our service to consul agent, they are registration ID, registration name, host, port, health check URL, interval check, and timeout check. Absolutely there are more variables that we can use, this time lets focus on them. Registration ID and registration name are the ID of your service, they must be unique. I usually using service name to fill them, so that I can easily searching my service in Consul dashboard. Port I simply get from configuration that I put in environment variable. We can see the way to get port from environment configuration in the snippet code above on line 36 until line 42. And now host variable, we can get automatically by getting the host where our service live by using OS standard library from Go. We can see from the snippet code above on line 44 until line 50. Next is health check URL, is the API that will call by Consul worker to check our service is OK or not. It highly associated to interval check and timeout check variables. Interval check determine the interval that Consul use to call the health check URL. While timeout check is the maximum response time from our health check API, if it more than timeout check, Consul will give us information about our service that it is critical, we can see it in Consul dashboard. Here’s simple sample of our health check API:

Here’s the consul dashboard looks like after we register our services:

Consul dashboard to see our service status

Key Value (KV) Store

As its name, there will be key and value that compose the data. This feature usually used to store configuration, because we don’t need to restart our Go service to read the configuration, because it is centralized, and as an API. I usually used JSON object to store the configuration, and yes, Consul support to validate our JSON configuration is a valid JSON object or not, this is awesome.

We can create our KV store or our centralized configuration in Consul dashboard. Here’s the configuration looks like:

Create a KV store as configuration in Consul dashboard

In the image above, we can see that we create a configuration with named product-configuration, and it’s JSON object, and just checked the validate JSON check box, our configuration will validate as JSON object by Consul, and yes it is really cool. And now, let’s call the product configuration API inside product-service. We will make an API called product-configuration who call the product-configuration and return it as a JSON.

Get configuration that we create in consul dashboard

Lets focus on line 7 until 26, especially on line 14. We can notice that we get the configuration by name, here we name it with product-configuration. After checked the value we simply return the value directly. Here the result:

Yes, we did it, successfully get the product-configuration that we create in Consul dashboard. We can change it as we want, and product service will display it as you wish.

May be that’s it, the simple thing that I learned from Consul. You can take a look full resource on my Github. Thank you.

Resources:

Consul by HashiCorp
Service Discovery in a Microservices Architecture — NGINX
Microservices Pattern: Server-side service discovery pattern
Service Discovery in a Microservices Architecture — DZone Microservices