Building Light-weight Microservices using Redis

This post is based on my RedisConf18 talk by the same name. The goal here is to provide greater detail than was possible during the time allotted during my earlier presentation.

Hydra is a lightweight NodeJS library for building distributed computing applications such as Microservices. Our definition of lightweight is: light on external complexity and infrastructure dependencies — not light as in limited. Hydra’s claim to being light on infrastructure dependencies is due to its only external dependency being Redis.

In the extremely unlikely event that you’re reading this and have never heard of Redis — then stop right here and visit redis.io. You can return to this post at a later time.

Hydra leverages Redis’s rich data structures to implement features required by non-trivial microservices. Features such as presence, service discovery, load balancing, messaging, queuing and more.

Hydra is also light in the weight sense. In fact, light enough to run on $5 Raspberry Pi Zeros. More about that here.

Besides being light, Hydra is also one of the easiest ways to build microservices. While this post won’t be an introduction to Hydra, as there are plenty of good introductory resources here, it will offer a deep dive into how Hydra leverages Redis to enable the construction of lightweight microservices.

A quick disclaimer is in order. The first is that the approach I’ll share in this post may not be a good fit for your projects. You may need heavier microservices. And that’s ok. The approach I’ll share here has been proven at Flywheel Sports and a host of other companies. I wrote an earlier post detailing why we built Hydra and the role it played in the building of our nation-wide live video streaming service.

What exactly is Hydra?

Hydra is a NodeJS module which can be imported into JavaScript Node apps to empower them with microservice features. Hydra does this by leveraging Redis.

Here we see three microservices — each with a Hydra module which connects to Redis. In this model, most services don’t communicate with Redis directly. Rather the underlying Hydra module proxies to Redis.

Another point about this diagram is that Hydra is just another imported module — like the ones shown in green. Hydra is only shown in blue at the bottom to illustrate its presence and connection to Redis.

The Hydra module exposes a JS class interface with a total of 36 member functions.

This snapshot provides a sense of the ease of our abstractions. Member functions such as findService and sendMessage are pretty straightforward.

How Hydra leverages Redis

This slide from my RedisConf presentation shows a number of important microservice concerns. Each, is required by non-trivial microservices. We’ll examine, in detail, how Hydra uses Redis to implement each of these features.

Keep in mind that the goal here is to show how it’s possible to do this — not to say that each approach is how you should implement the feature in your own services. A case in point is that while you can store your microservice configuration data in Redis or use Redis as a logger — it doesn’t mean that you should. At least not unless you know exactly what you’re doing and what the trade-offs are.

Also, keep in mind that you don’t need Hydra. Each of these features is made possible by Redis and you can certainly do this in your own apps.

A key point in what I’ll show you is that some of these features are only made possible when combined. For example, request and message routing depend on presence, health, service discovery and load balancing.

As you know, each of these features can be addressed using various infrastructure tools. However, a key goal with Hydra is to simplify building microservices while minimizing external infrastructure requirements. As you build your production-ready services you get to decide which Hydra features you need and which features you’ll get from other tools. It’s not an either-or proposition — but rather a matter of what you’re trying to achieve and how quickly you can get started.

That said, it is interesting to see how all of these features can be implemented just using Redis and your favorite programming language.

Key space organization

The first step in understanding how Hydra leverages Redis is to examine how it organizes the use of the Redis key space.

Hydra’s use of keys — consists of names with 2 to 4 segment labels separated by a colon character. The segments labels are named: Prefix, Service name, Instance ID, and type.

The Prefix segment allows for filtering Hydra vs non-Hydra keys. So if you make heavy use of Redis then being able to filter for specific keys is vital.

The Service Name segment aids in filtering keys of a particular service type. Examples would be an authorization, user or image processing service types.

The Instance ID segment allows filtering keys for a unique service instance. When running microservices you typically want multiple instances of a service type running. Each service instance is assigned a unique ID and being able to differentiate between them is useful.

Lastly, there’s the Type segment which is used to classify the purpose of a key. Not all of the segments are present in each key. For example, Service name and instance ID are not required in some keys.

Here’s an example of a key for the user service. We see the prefix which is hydra:service followed by the service name, in this case, “user-svcs”. Next, we see the unique instance ID. And lastly, we see that the type of this key is presence. So we say that presence information is stored at this key address.

A confusing point in our earlier description is that keys consists of names with 2 to 4 segment labels separated by a colon character. Yet, here we see hydra:service also being separated by a colon character. The thought there was that there might be another hydra:other types where service is only one of them. There is another inconsistency involving messaging which we’ll examine later. We’re planning to address these issues in Hydra 2.0 which will be a breaking change but cleaner internally.

We can enter the redis-cli and type Redis commands to view various keys. We’ll see examples of this through the remainder of this presentation.

A quick note about the redis-cli examples we’ll use: You’ll see the use of the keys command — this is shown for ease — internally Hydra uses the Redis scan command. For more about this see point #4 here.

So to recap — Hydra’s use of keys is organized by segments which makes them easier to query. Furthermore, a consistent organization makes them easier to extend and maintain. As we continue we’ll see the role that keys play in the organization of each microservice feature. Let’s begin by examining presence.

Presence

In the world of microservices, the ability to discover services and to know whether a service is healthy and can be routed to is of paramount importance. Those features depend on knowing that a particular service instance is actually present and available for use. This is also required for features like service discovery, routing, and load balancing.

Once a second, Hydra updates the (TTL) time to live of its service key. Failure to so within a three second period will result in the key expiring and the host app being perceived as unavailable.

We can see here that the Redis commands used are “get” and “setex” which sets a key and an expiration.

We can query presence keys by using the “keys” command with a pattern match. Notice that there are three keys present. This tells us that there are three instances of the “asset-svcs” running.

If we attempt to retrieve the contents of one of these keys we see that it contains the instance ID.

And using the TTL command against the key shows us that it has 2 remaining seconds before it expires.

So to recap. Microservice presence can be managed using keys which auto-expire. Updating the key is done automatically by Hydra on behalf of the host service. Meaning it’s not something the developer does. Failure to update the key within 3 seconds results in the service being perceived as unavailable. That probably means the service isn’t healthy.

Which brings us to our next topic…

Health

Being able to monitor the health of your microservices is another important feature. Hydra gathers and writes a health information snapshot every 5 seconds.

You can check the snapshot for a quick view of an individual service instance’s health. And, the snapshots can be used by monitoring tools such as the HydraRouter dashboard.

So here’s what the health key looks like. Notice that the only new bit is the “type” segment identifying the key as being about health.

When we view the contents of the key we see that it contains a stringified JSON object. In this case, it’s for the “project-svcs”.

Unstringifying the JSON makes it easier to see what’s stored. It contains lots of useful information.

So health information can be stored per service instance. It’s managed using a string key that contains stringified JSON text. And that information can be used by monitoring apps.

Service Discovery

Next, let’s consider service discovery, which is another must-have feature for any microservice architecture.

The ability to discover the IP and PORT location of a service by name greatly simplify communication. Other bonus points include not having to manage DNS entries or create fixed routing rules.

Service discovery information is stored in a Redis Hash with a type of “nodes” The use of a Hash enables blazing fast lookups. We use the Redis “hget”, “hset” and “hgetall” commands to work with the nodes hash.

The following Redis operations can be used to implement service discovery. The first is a lookup for a particular service type. The second is a lookup for available instances. The third lookup, allows Hydra to retrieve information about a specific service instance.

We can see useful information such as the version of the service, the instanceID, IP address and Port and finally the host name. In this example, the hostname also happens to be the Docker container id.

We can retrieve information about all available instances using the Redis “hgetall” command. This is how Hydra Router retrieves a list of services to show on its dashboard.

Let’s recap. Hydra queries using the servicename key segment in order to discover various bits of information about a service. Service details
can be managed using a Redis Hash which offers blazing fast service discovery

Next, let’s consider routes.

Routes

Routing both HTTP and messages such as Web Socket or PubSub — requires that routes be validated. Microservices can publish their routes to Redis. And as an example, HydraRouter uses the published routes to implement dynamic service aware routing.

Each service publishes its routes in a key of type “service:routes”. Here we see the key for the “asset-svcs” routes

Service routes are stored in a Set structure. A good fit because you don’t want duplicate route entries. The SADD and SMEMBERS commands are used.

As an aside, Redis’s rich collection of data structures is one of the reasons that what I’m sharing with you is even possible.

Returning back to our routes. We can pull a list of routes using a key pattern. Here we see routes for a number of services.

We can use the “smembers” command to view the contents of a specific route set. BTW, the bracketed [get], [post] and [put] bits represent HTTP REST endpoints. For other messaging transports the use of the bracket method may be omitted.

So let’s recap. Each service publishes its routes to a Redis Set. Accessing an individual route reveals a collection of route entries for that service.

Routes are stored in Redis using a Set data structure which avoids duplicate routes. The published routes can be used to implement dynamic service aware routing. Next, let’s consider Load Balancing.

Load Balancing

As your application grows you’ll need to load balance requests among available service instances. This is accomplished with Redis using the service presence and routing features we’ve seen. At an application level, using Hydra, this is as simple as using the “makeAPIRequest” or the “sendMessage” call. Load balancing takes place inside those calls as Hydra uses routing and presence information to choose among available target instances.

A nice benefit is that during routing if a request fails on a particular instance, Hydra is able to retry other available instances before ending in an HTTP 503 server unavailable error.

As you can see here, load balancing relies on other features such as presence, service discovery and routes.

To recap, load balancing requests among services can be accomplished using the Presence, Service Discovery and the Routing features we’ve seen. Redis Strings, Hashes and Sets make this possible. The whole is greater than the sum of its parts.

Messaging

Distributed services are forced to communicate with one another over an underlying network. HTTP Rest calls are probably most common, but socket messaging can be much more efficient. Messaging in Hydra is accomplished using Redis Pub/Sub channels and Redis implements Pub/Sub over socket connections.

Here’s an example key. Hydra uses the Redis “subscribe”, “unsubscribe” and “publish” commands.

As an aside, the Hydra router is able to accept messages over HTTP and WebSockets and convert them to pub/sub messages.

To understand how this works consider two services, the “asset-svcs” and the “project-svcs”. Each service creates two keys, one using its service name and another using its service name and its instance ID. Each service listens to both channels.

In most cases, you don’t care which instance of a service handles a request. In those cases, the channel without a specific instance ID is used.

Now, when you need to send a message to a specific instance the channel with an instance ID can be used. It’s important to note that hydra converts requests to a service name to one with a specific instance ID when it load balances. That ensures that only one instance handles a given message or request.

We can see a list of channel keys using the Redis pub/sub channels command. Notice that we have four keys here. The first key is the name of “asset-svcs” — shared by all instances of the asset service. Next, we see three additional keys with unique instance IDs. One for each of the three service instances.

Continuing with our focus on messaging. In order to ensure interoperability between microservices, it’s essential to standardize on a shared format of communication. The universal message format is a documented JSON-based format which includes support for messaging, routing and queuing. These messages are stored in Redis as JSON stringified text.

Here’s an example UMF message.

The “to”, “frm” and “bdy” fields are required and services are free to include their own custom fields in the “bdy” object.

Let’s see how this is used in practice.

On the left, the “client-svcs” sends a message to the “project-svcs”. Note that this only requires a UMF creation call and a send message call, shown here in yellow.

On the right — the “project-svcs” listens for messages and processes them as necessary. That’s accomplished using an event message listener.

Note that Hydra abstracts away the service discovery, load balancing, routing and pub/sub specifics. Sending and receiving messages only involves three member functions. It’s worth pausing here for a brief moment. Take a few seconds to consider what this example would look like using your favorite stack.

Let’s take a closer look. Send message works by parsing the ”to” field in a message to determine the destination service name. With the service name in-hand the next step is to check for available instances. With a target instance in-hand, the message is then stringifed and sent via the Redis “publish” command.

Again we can list all the Pub/Sub channels in Redis. Messages can be sent via these channels and retrieved by listeners. So with a bit of programming code, we’re able to use Redis to route messages using a well-organized collection of channels.

In summary, it’s worth noting that messaging is eventually necessary because services are physically distributed. Redis enables messaging using its pub/sub features.

Standardizing communication enables interoperability between services. We also saw how easy communication can be at an application level when we abstract away the underlying service discover, load balancing, routing, and pub/sub specifics.

Next, let’s consider messaging queuing.

Queuing

Job and message queues are yet another important part of many non-trivial applications. Hydra uses Redis to maintain dynamic queues for each service type.

Service instances can then read their queues and process items.

The content of a queue message is a UMF message which follows the same format used for messaging. Again, interoperability is king!

Hydra automatically creates three queues per service type.

* A “received” queue
* An “inprogress” queue
* And an “incomplete” queue.

Because these are Lists we use the Redis “lpush”, “rpush”, “rpoplpush” and “lrem” commands.

Here’s a diagram showing the message flow between queues. The movement of items between queues is an atomic operation in Redis. So it’s safe regardless of how many microservices you have.

In this next example on the left, queuing a message is as simple as creating a UMF message and calling “queueMessage” to send it. The code on the lower right shows the image processing service dequeuing a message by calling “getQueuedMessage” and later calling “markQueueMessage” once it’s processed the message. How easy is that?

So to recap, sometimes, it isn’t feasible to expect an immediate response. In those cases, we just need to queue work for later processing. The Redis List data structure can be used as a message queue. Commands like “lpush” and “rpoplpush” with atomic operations make this feasible. Here again, we saw how easy basic queuing can be using higher-level abstractions.

Logging

Distributed logging is another vital feature of any microservice architecture. However, if you know Redis you might be appalled at the thought of using it as a distributed logger. And you’d probably be rightfully concerned. However, you could use it as a flight recorder. Where you only store the most serious errors and you limit the number of entries using “lpush” and “ltrim”. Then at least you would have a quick way of checking what might have gone wrong with your microservice.

Here is what the key looks like. Notice the key type is health:log

Here we see that the health:log key type is actually a “List” data structure. So we can use the Redis “lrange” command to view the flight recorder log
for the “imageproc-svcs”.

Recapping: having logs on dozens, or worse, hundreds of machines isn’t feasible with microservices. Distributed logging is definitely the way to go. Using Redis you can build a light-weight logger to use as a flight recorder. Using the Redis List data structure and the handy “lpush” and “ltrim” commands make this possible.

And lastly, let’s consider configuration management.

Configuration management

Managing the configuration files for distributed microservices can be challenging. However, you can even use Redis to store configuration files for your services. We did this and it seemed like a good idea at the time. However, we’re starting to move away from it. As the core disadvantage is that storing configs in Redis makes Redis stateful and that’s less than ideal. But this is possible, so I’d like to share this with you.

Let’s see how this works. There is a configs key type which is a hash. The hash has a key consisting of the service version with a value set to the configuration data for that version.

Here’s a sample configuration. In our case, we used a command line tool called “hydra-cli”, which allowed us to push configuration files to specific service versions. All this did was create a hash entry with a key consisting of the service name and version with the file contents as its stringified value. Keep in mind that you can also use a shell script to drive the redis cli.

We can pull a specific version using the “hget” command and the version of the config.

Let’s quickly recap, we saw how Redis can be used to store application configuration files. The Redis Hash data structure allows us to store configs for each service type. Each config entry is indexed by the service version label and the contents simply point to a stringified JSON config.

Summary

Wow, we’ve covered a lot. But in summary, what I’ve shared here is an approach that very heavily leverages Redis using JavaScript and NodeJS. However, there’s nothing preventing anyone from doing the same with other languages. And because Hydra is open sourced it can serve as a reference platform for such efforts.

I’m pleased to announce that we have a Hydra-inspired Golang version in development, which we hope to open source soon. We’re also considering a Java version.

So that’s how Hydra leverages Redis. And how we were able to both build
and launch our FlyAnywhere platform.

—

Thanks for reading! If you like what you read, hold the clap button below so that others may find this. You can also follow me on Twitter.