Baker : A microservice orchestration library
For this first post, I am glad to be able to present to you a library we developed internally: Baker is an open-source Scala library that reduces the effort to develop orchestration logic in APIs to one third compared to traditional approaches.
Reducing the orchestration logic
Baker was built for a specific use case: individuals becoming customers of ING (customer on-boarding) via the mobile app of ING. The supporting API had to meet the following requirements:
- Implement a process flow of more than twenty steps;
- Steps spanning over an incredibly complex system landscape, evolved over tens of years and twelve separate systems of record;
- An event-driven process with business events occurring in a timeframe from two minutes to six hours;
- Re-use functionality already built in other business processes.
We anticipated that a direct implementation in Java would be a recipe for disaster because it would be difficult to reason about the logic once implemented. So we could do one of the following:
- write once in Java and rewrite the majority of it in case the process changes
- or come up with a suitable abstraction to compose the API from.
The following is a visualization of what happens under the hood when a user becomes a customer using our Mobile application. Would you feel comfortable building this easily without orchestrator? Neither did we!
Because of all those various requirements, we knew that a direct implementation was a recipe for failure : We needed something to make the whole workflow manageable, future proof and debuggable!
Principles
Baker is written in Scala, and as such is compatible with Java and Kotlin and all other JVM based languages.
Baker provides a domain-specific language (DSL) which supports developers during the design and development of the orchestration logic. It validates if a recipe is correct by making sure no loose ends in a process exist. Baker is also a run-time engine that can be plugged in an API that executes the process flow.
A recipe is made out of:
- Interactions (calls to underlying systems of record)
- Ingredients (containers for data)
- Events (what happens and is relevant for the process)
A recipe can also be visualized.
Interactions
An interaction is analogous to a function. It requires input (ingredients) and provides output (events). Within this contract it may do anything, for example:
- Call an external system
- Put a message on a bus
- or a pure function with no side-effects that generates a document or transforms from one ingredient to another;
When executed, an interaction provides an event as its output.
Ingredients
Ingredients are containers for data. They can be primitive Java types or POJOs. For example:
- an IBAN (unique identifier for a bank account)
- a track and trace code
- a list of phone numbers
- a customer information object with name, email, etc.
Events
Events signify that something of interest happens for a process instance. They may also provide ingredients required to continue the process.
Example use case
Coming back to our mobile onboarding example, this is what the real life recipe looks like :
Let’s be honest, this is not as simple as one might hope. But hey, we’re talking about the whole process needed to make any user a customer from ING! Additionally, we can see a few interesting things :
- The recipe defines all interactions and ingredients, but not their relationships with each other. This is normal. Baker figures this out by itself! Each interaction simply defines its own requirements, and the workflow is created based on this only.
- Each element is very simple by itself, and everything semantically makes sense.
- It is easy to see the ‘moving parts’ of the workflow (amount of retries, types of inputs, …)
- Everything is defined in a single location, which makes it easy to search for more information and without knowing anything about implementation artifacts.
When to use Baker
Baker is for you if you:
- Have a JVM based stack;
- Have business processes with many steps;
- Need a feedback loop with your business experts;
- Have a large number of teams that can re-use functionality from each other’s processes.
Additionally, Baker benefits from:
- A permissive MIT open-source license;
- The advantages of Petri-nets;
- The distributed computing capabilities of AKKA.
Baker is not recommended for:
- Low-latency, high throughput use-cases such as payments processing;
- Simple use-cases with two or three steps;
- Re-use doesn’t provide any added value.
Other equivalent libraries
Other big companies have the same problems as us, and they also created their own solution. We will name a few, and try to highlight the main differences. You definitely should look into them too and make up your own mind!
To see the complete comparison, please have a look at Baker’s documentation.
A word about reuse
Next to its benefits in terms of Orchestration, Baker also has very interesting properties in terms of reuse. Even though Banking is composed of a lot of various accounts, systems and processes, it is actual very frequent that those different products contain a lot of similar steps.
As a concrete example, we are verifying the identity of a person that wants to become customer and check that he/she is at legal age. This is something that would also happen if you wanted to open a youth account, or a savings account. In the same way, the generation of new bank account numbers is something that can be found in a variety of use cases.
Because of the very loose coupling between ingredients and interactions in Baker, it becomes possible to make those generic and create a ‘catalogue’ of ready to use elements. The huge benefit is that it becomes possible for teams to basically ‘assemble’ those interactions and create new recipes (meaning new capabilities) as high speed.
And in the same way as npm or maven dependencies, reusing the same basic pieces over and over again also contributes to obtaining higher quality ingredients!
What are we working on now?
We’re working on a serverless implementation of Baker, so that developers can focus on writing recipes and deploying them in a shared runtime environment. This will further reduce the time to market of new functionality for our customers.
Closing words
This is a very short introduction to Baker. Originally developed to solve our own pain points, it has now been picked up by numerous other teams at ING in the Netherlands and in Europe. Because of the way recipes are structured, teams are now creating repositories for interactions and share those snippets with each other, in the same way node modules can be reused. Very powerful for large companies with a lot of teams having some processes in common.
The project has now over 100 stars on GitHub and has been forked almost 30 times and growing. It starts being used outside of ING, and has a very permissive license. What are you waiting for to start using it as well?
Additional resources
If you want extra information, feel free to open issues on the Github project, or contact me (Nikola Kasev). Thanks to all the reviewers that helped writing this post, including Tim Linschoten and Julien Lengrand-Lambert.
