7 principles of microservices made developing easy
Discover how microservices allow you to align your system design with your organization’s goals
The coffee machine is, to me, undoubtedly the single most important place in our office. Look at what we hung next to it:
Yep, those are the 7 Principles of Microservices by Sam Newman (Building Microservices, O’Reilly, 2015). When I started working for yourapi’s predecessor, one of my first assignments was: read this book. Thoroughly.
That’s what I did, and it was a very interesting read. I come from a background in bioinformatics and for me, the focus was much more on streamlining data analysis workflows than developing and deploying applications. So yes, very interesting read, but reading the theory, its advantages and pitfalls feels very theoretical. As we were developing a system to automate human administrative processes, and as yourapi was born from that, the theory started to make more and more sense to me. The way the theory works out in practice for us has made developing easy. I’ll tell you how, around the 7 principles that I get to look at every cup of coffee.
Model around business concepts
I’m one of those people who likes to put everything in boxes. During my work in bioinformatics I got to do that a lot, because in a data analysis pipeline it’s very useful to focus on in- and output and regard a step in a workflow as a black box, hiding the internal implementation. You could view an application as a workflow, since there’s in- and output, transformations on that data, events triggered by that data, and so on. Start at the input and follow the flow of data through the application, right?
At the time, we were serving customers with a number of pretty complex processes that we automated using software robots. The application had been developed with microservices in mind. I quickly learnt that the system wasn’t divided into workflowy boxes, but into business concepts. A central web server to handle all communication between services and to the outside world and to persist state, a dispatcher service to assign workers and return their response, a flow controller to orchestrate non-atomic tasks and robot clients that perform programmed business tasks.
As we set out to design a much more complex robot platform (from which yourapi emerged), we got to rethink the service boundaries. As a developer new to microservices this has been an outstanding opportunity to learn about domains and models and boundaries in practice. But also about not wanting to split out too eagerly and too early. As a team, we grew into this, which led to a very natural way of deciding if something should be a feature on an existing service or a new service on its own.
Adopt a culture of automation
For us, this means that we’ll always look for a uniform way of accomplishing something, but that we won’t automate manual steps prematurely. An example is the fail-over for our software robot system database. We took the time to set up replication and to work out fail-over. Once we were confident the manual steps were correct, we… left it at that. That’s 2 years ago now, and we have had to use the manual steps only once. We could also have spent a lot of time setting up fully automated fail-over, but doing it manually these couple of times actually took a lot less time.
In other cases, after we fine-tuned steps to accomplish some task, we actually did automate, at least to some extent. An example is the steps to deploy our current services using Kubernetes, where we let a script take over tedious and repeated manual steps.
So our culture of automation is taking manual steps which are never automated prematurely. This pragmatic approach (and that we all do it this way) helps in fine-tuning the steps to take, making the process less error prone and less time consuming.
Hide internal implementation details
My classic view on a black box in a workflow was ‘here’s the data, I expect you to do this analysis, I couldn’t care less how you do it, as long as I get the results, fast.’ I would basically ask the black box how I should hand in the data and what the output would look like. The business logic might be hidden, but the implementation details mostly weren’t. Calling a Perl script doesn’t exactly hide the programming language, to use an obvious example.
Erda Estremera on Unsplash.
At yourapi, we have adopted the REST principles to integrate our services. This hides implementation details at different levels. For one, thinking in terms of resources completely decouples the interface from the underlying persistance layer. The fact that we have a resource called ‘customers’ doesn’t necessarily mean we have a SQL database with a table called ‘customers’. Also, identifying items by a uuid hides any details on how and where that item is stored. Is it a MongoDB object? A Google Datastore blob? A PostgreSQL record in some table?
On top of that, all communication with any of our services is done using HTTP. We can use any client we like and never know that the service was written in Python.
This decoupling of interface and underlying technology tremendously helps developing new services, because integration is uniform across all services, no matter how they’re implemented.
Decentralize all the things
To maximise autonomy of a service the first thing is to use a choreographed approach. We let our services handle their own job, instead of having a single orchestrator that holds all business logic in one place. Our services subscribe to relevant events by knowing what action to take on a certain incoming call, and they pass on the process by making relevant calls to other services based on their own business logic. This decouples our services, making things a lot easier when there’s a change in some service.
To further maximize service autonomy, each service should be completely self-contained. Sam Newman addresses a number of pitfalls in his book, such as the perils of code reuse and how to efficiently decentralize security or monitoring. He also discusses strategies for persisting data in a decentralized way, both with respect to design and to deployment and management. Still, a team developing and managing a service needs to be full stack and the same choices of basic functions have to be repeated for each new service.
We realised that this naive approach wasn’t going to help us developing our services. We would have to spend a lot of time on service plumbing, time that we’d rather spend on interface design, business logic and, well, the next service. We solved this by using a standard container providing all plumbing (serving the REST-interface, data persistance, logging, etc), and then delivering the functionality of the microservice as a plugin. The container is identical for each service, can be deployed in as many sites as needed and is designed such that updating it does not impact the plugins.
We can now fully focus on the functionality of a service and have one team focus on the full stack development of the container, while still ending up with decentralised microservices.
Independently deployable
Perhaps you’ve noticed that we don’t like to do things prematurely. We don’t like to split out new microservice boundaries too soon. We don’t like to automate steps when the manual steps suffice for the time being. We take the same approach with deployment.
The use of the above container approach has the benefit that services can be deployed independently, but that they don’t have to. We can now focus on the best ways to deploy, manage and scale a container with plugins without having to repeat every small change across many deployments. Then, when needed, we can split out this very coarse-grained deployment.
Isolate failure
This, too, boils down to uniformity and avoiding premature optimization. Because we use REST over HTTP, we have a uniform way of catching, communicating and acting on problems. This starts at the service’s code base (both in a plugin as in the container), where all exceptions are caught and handed down to the very core of the container. They are all translated into the relevant HTTP error codes, so a service will always communicate an exception in a way another service (or any client for that matter) understands.
On top of that, by careful adoption of REST principles, communication between services is safe and idempotent where needed within an orchestration. This means that no caller can inadvertedly break another service simply by making a call that was supposed to be safe.
A phased development process also greatly helps. We’ve adopted DTAP (Development, Testing, Acceptance, Production on separate environments). We can safely fail massively at our development environment without affecting production at all. We’ve also implemented this approach for our customers, you can define and deploy your APIs in a development stage, verify their functionality in the test stage, verify all expectations are met in the acceptance stage and then finally safely move to prodution. Anything that happens in one stage won’t affect the other stages.
There are other obvious ways of creating bulkheads, such as careful splitting of services, decent load balancing and connection pooling, independent deployment, perhaps even load shedding. We have created a framework where we can implement any such stategy in a uniform way, across the relevant services, at the time that we feel we need it, without much hassle.
Highly observable
We’ve had some hands on experience trying to monitor microservices. We ended up with a number of microservices: a central monitoring server, an aggregator, an analyser, an alerter and a dashboard. The idea was that each microservice pushed relevant data to the central server, data was aggregated and then analysed, and results shown in the dashboard. The alerter sent SMS or mails when something was afoot. Still, we struggled to get more information than basic system metrics, and we also struggled to get information from all of our services. The main cause of this failure was the lack of (there it is again) uniformity across services.
With the plugin container approach yourapi was built with, we now have a standard way of logging things that happen in each microservice. Each log entry is automatically accompanied by several labels that correlate the entry to its reporter, all the way from which service to which pod it was running on.
Actually, serverless cloud deployment using the Google Cloud Platform and Kubernetes has helped tremendously in structuring the way we monitor services. We can now track all our logs and metrics in one place without having to rely on a custom aggregator and dashboard.
Conclusion
We have adopted Sam Newmans 7 Principles of Microservices and we have carefully weighed the pitfalls. This has lead to building a platform true to microservices, that makes developing very easy for us. And for our customers for that matter!
