Integrating External Systems In A Microservice Architecture

Lately I had some discussions in a project that we built with a client regarding how to do integration of several external third party system with the Microservice based architecture we use.

While it was clear for me that the core of the application’s domain shouldn’t change and that we want to use message queues for sending out events to the potentially-more-than-one external systems, and receive events from them, we had a lot of discussions regarding where the needed transformation should happen.

Strangely enough, the first reactions from the client’s team was to put all these things not only into our domain, but also into the data model of our services. This was somehow shocking for me. It would mean that for every single external system we would have an impact in our domain model, and work to do on each Microservice involved.

To find a conclusion, I draw three possible solutions to this common problem.

Sender Defines Model, Receiver Needs To Transform

The assumption in this case that the sender defines the data model for the message in the queue, usually based on its own domain model. Transformations have to happen on each receivers’ end, so if there are two receivers, each of them needs to transform the message into the format of their own domain.

Benefits

• Sender doesn’t care about the data model of the receivers
• Receivers can be added without impacting the sender

Drawbacks

• Assuming the messages can go into each direction, every system needs to deal with the data model from the other systems

When To Use

Most likely you should never use this. Reason is, the drawbacks can be huge: If system A is a rather big system compared to the others, they need to adapt on the smaller systems’ data models when receiving information.

And, in all cases, this means that EVERY team involved needs to deal with at least a portion of the other’s team data model!

One Systems Defines Data Model In And Out, Transformation Is Up To The Receivers

In this case, one system is the master, and defines all in and out messages. Each other system needs to transform receiving messages, and transform messages they want to publish.

Benefits

• One system defines it and no longer cares about it
• Interface is well defined when new system needs to be integrated into existing

Drawbacks

• All teams of the other systems need to be good in integration
• They need to fully understand the data model of the master system

When To Use

If you know that the other systems resp. teams can handle the integration, go for it. The master system’s interface must be well documented to not have extra work on each integration.

One System Defines Data Model In And Out, Transformation Is Done For Receivers

Very close to the last one, but the transformation is done by either a team very close to the master system, or the team responsible for the master system. Even though this means that this team needs to deal with every integration of another system, you have the integration knowledge in one place.

Benefits

• Knowledge for the integration is within one team
• Easier to operate as the knowledge of the transformation is in the team
• Smaller system’s team doesn’t need to deal with integration

Drawbacks

• You need to have dedicated integration knowledge in your team

When To Use

I prefer this one because if you have a more dominant system that connects to other, smaller systems, it is most efficient to have a dedicated integration team that knows the domain of the master system very well. Very often, the systems to be integrated come from smaller teams, or teams less experienced with integration. And you need to explain your data model on top of that.

Where To Store Transformation Specific Data

In almost any cases, the transformation won’t be straight forward — you will need to map values or keys from your system to the others, and most likely you will need to adapt to the processing logic of the other interfaces. The typical use case would be a system that can be queried for changes, but the event doesn’t contain the changed data itself but some reference, for instance a URL to a service call. So your system needs to query for the data and put that into the message.

Second typical scenario is when you need to poll an interface to detect changes, or to detect the result on an asynchronous call to an interface.

Rules I Recommend To Follow

• Never store any mapping in your domain model
• The same rules as for microservices apply for the transformation — e.g. do not query data from a table that is not part of the transformation’s domain
• If you have a mapping that is used more than once, provide a service for the mapping
• If not needed, put the mapping into the datastore for this transformation service
• Never store the state of processing logic in the domain model of your core domain, instead store it only in the domain of the service dedicated to the transformation