3 lessons from the frontline of microservices
We’ve been using microservices for around 4 years now and we’ve come a long way in terms of understanding how to get the most out of the pattern. A number of hard lessons had to be learned to get us to this position and I’ll summarize these below. This is not meant to warn people off microservices. Our experience with them has been overwhelmingly positive and I’d hate to think about where we’d be today if our application was built as a massive monolith!
Bring in experienced people
Almost all of the lessons below will be familiar to anyone who has spent time working in microservices. When we began, this experience was rare and we found ourselves in a position where the engineering staff were all microservice beginners. Previously, we had all been working on monoliths and we tried to use this experience to build them. Big mistake! The thought process for building microservices is in many ways antithetical to building monoliths. It’s difficult to remove yourself from that mindset, especially if you’ve been working on monoliths for many years.
I strongly suggest that if you decide to switch to microservices, you should bring in lots of people that have experience in that domain. Switching is not for the feint-hearted!
Artificial boundaries are useful
It’s amazing the difference in how developers approach a task when there is an artificial boundary in the way.
An early decision we took was to separate the Customer microservice from the KYC (Know your customer) microservice. Once this boundary was in place, the developers needed to design an interface between the two. They used REST to initiate the check from Customer to KYC and then messaging to inform Customer about the results. A few years later, another part of the company, with a different model of a Customer needed to conduct KYC checks. With some small alterations, the KYC service could support this other Customer model. The original KYC service had taken several months to complete and so this duplication of work was saved.
If the application was a monolith, there would be little chance that the KYC feature could be reused. Most likely, the code boundary between the model of the Customer and the ability to conduct KYC checks would become very blurred. Trying to pull out the KYC part would have been nigh on impossible for someone without an intimate knowledge of the codebase.
Another boundary that we found useful was the code repository. In the early days, all the code was stored inside one repository and built as a single unit. This was several microservices. When one microservice’s build started failing, the entire build would fail. This stopped developers from working on the other microservices until the build was fixed. We were missing out on one of the key benefits of microservices — the ability to work on a single microservice while not caring what is happening in the others.
After a while, we separated everything out into different code repos and allowed them to have separate builds. While there was some extra overhead in maintaining all of these builds, we saved a lot of time and anger in enforcing this boundary.
Adding new microservices should be easy
Imagine it takes a month for a new microservice to move to production in your stack. What kind of behavior are you pushing developers towards? I would argue that developers are more likely to add code to an existing microservice than produce a new one even if it makes sense more to add one. No one wants to plan out a task yet have to wait for a month to start coding it.
If you keep adding code to existing microservices, you eventually reach a position where you have multiple monoliths instead of microservices. You lose all of the benefits of microservices, yet also have the disadvantages of them as well! A microservice should be small enough that it could be rewritten in a few weeks and understood in a few days.
You should aim for the time between a developer deciding a new microservice is required to the point that it is available in production to be as short as possible.
Roll your own at your peril
It’s easy to fall into the trap of thinking that the small size of microservices means that they are simple. If anything, more care must be taken in coding them than the equivalent monolith.
As we created our first few microservices, we built on top of the Jersey web server. It all worked swimmingly to start with but after a while, we encountered problems.
One in particular was the problem of debugging. Simply following a request in the logs from start to finish become exponentially more difficult when you have several microservices involved. The answer to this problem is available and well understood — the correlation id (also known as distributed tracing). The id is generated when a request first enters the system and then is propagated through the calls to other microservices. It is part of every log line, and so makes it easy to track a single request.
So, we did exactly this. Using SL4J’s MDC feature and generated UUIDs, we added the feature to every microservice via a library. And it worked fairly well. We had some issues with repeated IDs and other things but essentially it worked. We fixed the bugs, and got ready to release the second version. Now, to get that into production we had to release *every single microservice*. A job that would have been a simple job to fix in a monolith turned into a big one with microservices.
This is repeated every time you find a bug in any of your base libraries. Worse still, if you decide to use a different language for a new microservice, you will need to reimplement the libraries in the new language. For these reasons, I suggest using well known open source libraries for these kinds of things. Hopefully, most of the kinks will have already been worked out in this software and you will need to update them less often.
The microservice ecosystem is flourishing at the moment, and there are plenty of choices for something like correlation ids. For example, Zipkin (https://zipkin.io) presents a language agnostic way of doing this with libraries available for every popular language.
