Simple Service Registry

Distributed Config Management for Your Microservices

Here on the Watson Data Platform, we have spent a lot of time over the last six months working on building examples of microservices that demonstrate how developers like you can use our range of database solutions to solve real world problems. So far we have talked about creating a searchable index from a CSV with the Simple Search Service, and showed you how to add autocompletes to your web forms in no time at all thanks to the Simple Autocomplete Service. We’ve also covered logging and caching.

Why Microservices?

Taking a microservices approach means that we break our app down into several smaller, independent applications that handle one specific aspect of the task at hand. This means that your code is more streamlined, services are less dependent on each other, and as such are easier to maintain and re-use.

There are, however, some sticking points. How do we wire all of these services together? What if my app is auto-scaling and nodes are appearing and disappearing at regular intervals? And how do we manage any changes to configuration without experiencing downtime?

Get Configgy Wit It

Na na na na na na na nana …

One way to manage the logistics of a setup like that is to employ some kind of service discovery mechanism. Service discovery is a means of automating the discovery and implementation of distributed services, as well as distributing other configuration information across the whole infrastructure.

The general idea is that each individual service would register itself with a centralised registry, including any information required to connect to and use the service (like hostname, port, etc.). Subsequently, any other service or system that can integrate another can then autonomously discover these services, apply the configuration information to itself, and start making requests — all without the need for human intervention.

Additionally, configuration changes can also be detected and acted upon across a whole cluster of servers, minimising downtime and human error.

Sound good? We thought so too, so decided to set ourselves a challenge…

The Challenge

Extend the Simple Search Service by letting it use our other microservices. We can add autocompletes to the data management pages, cache popular searches, and record searches for further analysis downstream.

To make this a bit more interesting, we decided on a few stipulations:

• The Simple Search Service must auto-detect the presence of the other services
• It should be possible to enable or disable these services without restarting the Simple Search Service
• The Simple Search Service should not rely on any external services to perform its primary task of searching (i.e. the use of external services is NOT required)

The Tools

Let’s take a look at the technology we chose to help us complete this challenge.

Our Existing Microservices

• Simple Search Service
• Simple Autocomplete Service
• Simple Cache Service
• Simple Logging Service

Each of these four microservices has its own HTTP API. None of these services know anything about the other—they are completely stand-alone. Individually, they provide one useful feature each. We plan to make them work together to be even more useful.

etcd

To build a registry for our services to use, we chose etcd. From the etcd homepage:

etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines. It’s open-source and available on GitHub. etcd gracefully handles leader elections during network partitions and will tolerate machine failure, including the leader.

Your applications can read and write data into etcd. A simple use-case is to store database connection details or feature flags in etcd as key value pairs. These values can be watched, allowing your app to reconfigure itself when they change.

Our services register themselves against a known key in etcd, and we can then “watch” these keys for changes in the future to act upon them.

We created a Simple Service Registry module for Node.JS to help with the registration/discovery of services and use this to talk to etcd.

Each of the above services requires a ETCD_URL environment variable defined to enable the registration/discovery processes, which should point to an instance of etcd. For example:

export ETCD_URL='https://username:password@etcd.hostname.com'

It is really simple to spin up an etcd instance with Compose on a 30-day free trial, which is what we used. However as etcd is open-source you can also grab it from GitHub or via one of the popular package managers for your platform of choice.

Socket.IO

Detecting and reacting to configuration changes using etcd is only half of the battle. We also need to visualise the effects of this configuration for users via the UI, and the UI also needs to reflect any changes within etcd. To handle this job, we use Socket.IO to help us implement WebSockets, which can transmit any app configuration changes to the browser.

Putting It All together

We must follow several steps to succeed with this challenge:

1. We modify the Simple Search Service to integrate the additional microservices. (Service discovery is a powerful tool; however, it is almost useless if once the services have been discovered, we don’t know how to use them.)
2. We make all of our services register themselves with etcd.
3. We alter the Simple Search Service so it can discover these services and seamlessly start using them at a user’s request.

Service Integration

We aim to add the following services to the Simple Search Service:

• Autocompletes in the data editor
• Caching of searches
• Logging of searches

Each service requires storage of some configuration information—most importantly a hostname. We are using the Express app.locals object to store this information for each service individually.

Caching and logging of searches requires us to modify the existing search to optionally use a caching and/or logging module, depending on availability. We don’t want the search to break if these services aren’t available, and any switchover should be transparent to the user.

We create two new modules within the Simple Search Service to make HTTP requests to the cache and logging services at appropriate points within the search process. At the point of making these requests, we determine if the necessary configuration information is available. If it is not, then we simply bypass this step.

Adding autocompletes to the data editor is a bit more involved. In order to provide autocomplete suggestions, we need to create an index on the Simple Autocomplete Service. The Simple Autocomplete Service has APIs available that let us specify a remote text file to create our index. This approach is perfect for our use case. We just need to provide the data. When importing data into the Simple Search Service, you can define which fields you want to create facets on, and we can use these facets to help generate our list of terms.

We created a new API endpoint on the Simple Search Service that provides a list of available terms for a supplied facet. Try it on your own Simple Search Service:

curl -X GET http://your.simplesearch.com/autocompletes/<facet_name>

This means that when we enable the autocomplete service within the Simple Search Service, we ping the Simple Autocomplete Service with all of our autocomplete lists so that we have indexes ready and waiting.

We also pass our configuration information to the front end so that we can point the jQuery autocomplete plugin to the correct API. We use our Socket.IO implementation for that.

Service Registration

The next step is to register our services with etcd using the .register() method of our Simple Service Registry module.

The code to do that is quite simple. (We do something similar for all of our services.):

At this point, we have all of our services registered in etcd, along with the URL required to start making requests to their respective APIs. However, before we can do that, we need to let the Simple Search Service discover the other services.

Service Discovery

In the Simple Search Service UI we created a new menu item called Services, which lets you:

• see if service discovery is enabled
• see what services you can discover (Autocomplete, Caching, Logging)
• see which services are discovered
• enable or disable any discovered services

To enable the Simple Search Service to discover any available services, we call the Simple Service Registry module’s service() method, which returns a Node.JS EventEmitter that has events for set, expire and delete. We can then use these events to manage the configuration information for each service.

Here’s a simplified version of what we are doing:

Lets take a closer look at what is happening in that code:

• We attempt to discover the autocomplete-service from within the search namespace.
• When this key is set, we update our autocomplete configuration information.
• When this key expires, we reset the autocomplete configuration information.
• For each event we send a reload-config event to the front end using WebSockets. This triggers an Ajax request to GET /config, pulling in the latest configuration information to the browser.

Enabling Services

At this point we have all of our microservices successfully registering themselves with our service registry. We also modified the Simple Search Service to discover our other services and store the configuration information locally.

The final step is to let users enable/disable one of these discovered services via the UI. On our Services page we have an enable/disable button associated with each microservice. This button pings an endpoint on the Simple Search Service to enable or disable a specific service—something like:

POST /service/enable/autocomplete

It uses the setEnv() method on the Simple Service Registry module to mark the Simple Autocomplete Service as enabled in etcd. We can then use the getEnv() method to detect any changes to this environment parameter on ALL of the Simple Search Service servers in the cluster.

If the autocomplete_enable flag expires, is deleted, or is not set to true, we mark the autocomplete microservice as disabled in our configuration information. However if this flag is set to true, then we mark it as enabled and start to show autocompletes within the Simple Search Service. We can repeat this step for all of the other microservices.

The important thing here is that we are managing our config in a distributed way. As long as we are always referencing the data stored in etcd, we can change our application config in real time across a large cluster of servers.

Try It Yourself

The Simple Service Registry module is designed to let anyone perform service discovery and configuration distribution for any set of apps or services. Why not try and implement it yourself?

Conclusion

A microservices-based architecture has many benefits for both development and operations teams when creating, managing, and maintaining a distributed system. However, a forever-changing infrastructure brings its own headaches and presents several different challenges. A service discovery approach helps mitigate these challenges at the cost of a little extra work up front.

This post shows how your suite of microservices can adhere to the philosophy of standalone, independent microservices while you extend and expand them in a scalable and predictable way using service discovery tools—no matter the size of your infrastructure.