Sushi Tonight: SymmetricDS as a Microservices Datapump

When I first heard about microservices they sounded intriguing. There’s the promise of clean, orderly separation and happy independent teams. But then reality steps in and we have to recognize that these teams are not completely independent. These services must collaborate at some level and how this collaboration unfolds is often the difference between well architected system and a mess. In other words, how we handle dependencies could make the difference between a big bowl of spaghetti of a system versus an elegant sushi roll (that sounds good for dinner tonight). (Nothing against spaghetti, but when it comes to software design, I want mine to be more like sushi — awesome, orderly and healthy!).

I was informed the sushi looks like eyeballs. Well, it’s supposed to be sushi. Anyway.

So in our quest for the perfect sushi roll of software, let’s consider the (somewhat prototypical) example of an order system. We’ll have a “item” (or “product”) service which holds product details, a “stock” service which is responsible for knowing which products are where, and an “order” service which is responsible for managing the ordering process.

Each of these services has data that only it is concerned about or it is the natural “owner” of. The Item service has product details such as product images, specifications, etc. while the stock service has stock level details and the order service maintains details about the status of an order and its history. So we end up with data that is local to each microservice. So far so good.

But what about aspects of this data which are naturally shared between these services? Let’s take a simple example of the product identifier and the product description. Naturally the “Item” service will own the product ID and description. But the stock levels and order services also need to know that basic information about a product as well. So how do we handle this?

One simplistic solution would be to allow the stock service to call the item service for information. This could work in some cases. But it would be easy to imagine a stock service working on products en masse, where it would need to know the description of 1000’s of items at once. In these kind of cases, cross service calls might not be feasible.

We could also talk about another sort of microservice: reporting services.

Often, reporting services need a wide array of data from lots of sources. It would not make sense or be practical for a reporting services to make 1000’s or more external service calls to generate a report.

Another approach to solving this problem is to simply share the database. If all microservices need product info, then why not allow them all to query the product table for the info they need? And the reports will be easy because we can just join across lots of tables. There are obvious advantageous to this such as simplicity and good performance at least in the short term. But to me, this should be one of the last options we look to as it introduces a strong but often hidden coupling between the microservices involved. Microservice teams should be free to design their persistence platforms as needed to manage those operationally without thinking about other processes contending for those same resources.

I think a better option for achieving this shared data is the idea of a “data pump”. The idea of the data pump is that changes to the shared data are captured and “pumped” to the other services at the datasource level. The pros to this approach are that you get decoupled microservices, and if the data pump is robust, you can keep this shared data in step, near real time, have an audit trail, etc.

The downside to this approach is that it is difficult to implement from scratch. You have to figure out how to reliably capture changes to your database, how to publish them (some like to use Kafka) and develop code to write those changes to the target datasources. And those target datasources could be from a different vendor of even a different type all together (e.g. maybe the product microservice uses MongoDB and the stock level service uses MySQL). And the data pump needs to be resilient, handling all manner of errors and offline scenarios, and should be fully auditable.

This is where SymmetricDS comes in.

SymmetricDS is an open source, multi-platform data capture and replication engine which could be a drop in solution for a microservices data pump. Plus professional support is available from jumpmind.com. It’s a solid option to consider when designing your microservices architecture. Get started with it today with the open source quick-start tutorial or get a trial of the professional version.

It might just help you trade in that spaghetti bowl for some delicious sushi tonight.

いただきます.