Multi-Region Messaging with Nameko
Distributed systems are often hosted in geographically dispersed data centers. This post demonstrates how messaging between services in different regions can be accomplished with the help of Nameko framework.
Overview
Sending messages between data centers in different regions is something most globally distributed systems have to do sooner or later. Since Nameko has a built-in support for AMQP messaging via RabbitMQ, this task is fairly straightforward to accomplish once federation between RabbitMQ clusters is configured.
What is Nameko? The headline of Nameko’s documentations says it all:
A microservices framework for Python that lets service developers concentrate on application logic and encourages testability.
If you’re new to Nameko I highly recommend that you to get familiar with its documentation, source code and sample application to get a feel for what it is and how you could use it in your system.
Use Cases
Three different use cases will show us how messages can be distributed between three different regions.
Events: One of the more common scenarios is a requirement to dispatch events by a service in one region and handle them by services in all regions. This could be useful when a change to an entity has to be propagated to all systems holding information about it e.g. cache.
Master service message processing: In this example I will show how a message sent from a service in any region can be processed by service in only one region. A common requirement for systems relying on database replication where master databases, and services with write access to them, are hosted in one region only.
Asynchronous two-way messaging: In a less common scenarios there is a requirement to send a message to specific region and receive a reply back in the originating region. This could be useful when we have to communicate with a service that is constraint to reside in one region only, perhaps one that has connection to legacy systems located in headquarters datacenter.
Setup
Before delving into the use cases let’s talk through the required setup.
To simulate multi-region deployment, we will use Docker Machine to create three hosts: Europe, Asia and America. Each one will be running Docker engine, with RabbitMQ and example Nameko services packaged in Docker containers.
In production this configuration will most likely be bit more complex. For example I would recommend creating RabbitMQ cluster with HA queues and redundant Nameko services that are deployed on separate hosts. But that kind of a setup is beyond the scope of this post so let’s continue with something much simpler.
Most of the commands below are encapsulated in Makefile found in the root of example repository: https://github.com/kooba/nameko-multi-region-example
Prerequisites
Ensure Docker, Docker Machine, Docker Compose and VirtualBox are installed and running:
Clone example repository
Create three VM machines
This will take a while. You might want to take a coffee break. ☕️
Once it’s done verify your machines are up and running:
Deploy RabbitMQ and Nameko sample service to each of the hosts. This step will also setup three-way federation and policies between Rabbit nodes.
In this step all required Docker images will be downloaded from Docker Hub. You can create your own local images by calling $ make build-all. You will have to change docker-compose file to use them.
Federation
With the step above we’ve enabled threeway federation between all RabbitMQ hosts. This means all of our nodes are connected and messages can be distributed between them based on the policies we’ve set.
To verify federation is working correctly take a look at federation upstreams, policies and federation status in Rabbit’s Management Console.
Upstreams
View of Federation Upstreams from “Europe” node. Each node will point upstream to the other two regions, America and Asia in this case:
Policies
For our example we’re applying one policy to exchanges and one to queues.
Our exchange policy says that any exchange ending with .events should be federated. Nameko will name exchanges responsible for dispatching events this way by default.
Our queue policy says that any queue starting with a prefix fed. should be automatically federated. As a developer you are in control of queue names when publishing messages. Two of our messaging examples will use this naming convention.
Status
Based on our upstreams and policies we can verify what links Rabbit has created for us:
Messages vs. Events
In Nameko nomenclature there is a distinction between Events and Messages. Events represent something that happened within the domain that could be of interest to one or more services that would want to act on it. Service dispatching events don’t know or care which services are the subscribers or what they would do once the events arrive there. Messages on the other hand are published with the intent to be processed by a specific service that will perform a specific action upon receiving them.
Use Case: Events
To demonstrate events propagation our products service will dispatch the product_added event in one region and the indexer services in all regions will handle this event to add information about the product to a cache.
Example above shows two Nameko services Products and Indexer. Upon calling add_product http endpoint product_added event will be dispatched and handled by Indexer services’ handle_product_added in all regions.
Let’s play with the example:
Use Case: Master region message processing
In our second example we will use concept of entrypoint blacklisting to ensure desired messages are processed only in one region.
To accomplish this we have to create custom implementation of Nameko's ServiceContainer:
Right before Nameko starts our container, we will remove any entrypoints which have been listed in ENTRYPOINT_BLACKLIST section of the service’s config file.
To enable custom ServiceContainer implementation we have to tell Nameko about its location by setting SERVICE_CONTAINER_CLS config value to the location of our custom class: SERVICE_CONTAINER_CLS: src.container.ServiceContainer
In our example when placing an order for the product we will publish a message to federated queue fed.order_product which, regardless of which region it was published from, will be processed only by consume_order entrypoint in Europe region.
Let’s play with the example:
This is a handy capability for any system that relies on only subset of services having write access to databases. Normally service in one region will write the data to a master database and services in all other regions will rely on read only replicated view of the datastore.
Use Case: Asynchronous two-way messaging
With a little bit more orchestration we can achieve two-way cross-region messaging.
This setup will allow us to send a message from any region to any other region and receive reply back to designated consumer.
Let’s play with the example:
The limitation of this approach is that we handle response coming from remote region completely separately from where we initiated a request. Although when deciding to send messages between the regions we should always expect delays in processing and handle any intermittent network outages with grace. This is where RabbitMQ shines.
Summary
Nameko comes in with simple but powerful defaults to handle HTTP and AMQP needs of your distributed system. With a little bit of customization you can take it just far enough to cover most of your needs when distributing it geographically.
