Being a startup CTO in Stuttgart shaping the monolith
An integration pattern supporting a non-monolithical mindset to shape a currently monolith app for future flexibility and scalability
In this article I will outline a new pattern we plan to incorporate at LASERHUB to avoid building a monolithical application in the long-run.
In my opinion, there are two main reasons why companies build up a monolith and find themselves trapped after some time, missing the point of no return. In the last 12 month I had many of those situations where I felt like: “Wow, I have to think about that NOW to avoid being stuck in technical dept in not-to-near future”. There are too many of them and you can’t always act immediately. I try to balance my cost/value choices wisely, especially with those “reduce technical dept” moment.
One of the reasons I see is the ongoing battle of speed now vs. speed long-term. Solutions, which are now faster to implement win the battle in startups as speed is everything. But speed is relative. Many decisions are made based on current code and design patterns and more important — current experience or know-how. Code does not evolve out of its own or because someone read a nice article. Only if we adopt something, we learned in actucal practice, this might break with current way of thinking. For the monolith problems, this means that we will usually start with a one-repo, one-app strategy to avoid any complexy in deployment, etc. — we bootstrap. But as we go on, we incorporate this way of coding into our coding practices — this can be risky.
Some engineers will come with ideas how to break out, but then we will fall back and say: “hey, i think it’s not the time or this feature will stay small or it will not be that hard to break things apart”. Might be true for now, but every new file added will increase that migration complexity.
At LASERHUB, we try to avoid having a spaghetti like code base without any guidance by having a domain-like thinking about repo structure, but that’s only a start and it only works if every engineer incorporates the idea.
This brings me to my second main reason why we build monoliths — the engineering team’s mindset. Having a non-monolith mindset (like microservices, serverless, event-sourcing architectures, etc.) and having experienced tightly coupled software in the past, will always make you cautious. But what if your team is not as experienced or sees too many barriers or is overwhelmed by tasks anyway? How can you grow this mindset without loosing speed in current deployment, without increasing deployment complexity right away — but start building a non-monolith mindset starting TODAY?
Here is what I came up with, with credits to Holger Reinhardt.
We introduce a new patter in our architecture which fulfills the following requirements
- It adds value short-term
- It helps us thinking like a non-monolith team long-term
- We start using new technolgies supporting a non-monolith mindset and learn at a very limited scope
Now, here is our challenge and the according pattern, we will introduce at LASERHUB.
We need various endpoints for three different third party SaaS providers and their respective webhooks. We will use webhooks mainly for receiving update requests. I will not talk about details — just that. All three use cases differ completely. Two systems serve us with tracking information while the other one is part of our core functionality — real time pricing and offer for individual sheet metal parts.
The pattern is fairly simple
- Using the serverless framework, we build REST endpoints on AWS’ API Gateway
- Lambda functions run at each endpoint and simply push individual, service specific update tasks to a central message queue
- Workers in our main app consume these messages and run updates
In my opinion, the main benefits are the following:
- We learn how to build infrastructure as code (serverless framework)
- We start using Lambda functions as a new way of serving a REST API
- We keep our overall business logic in one repo for now to avoid ad-hoc migrations costs (move all parts of a “microservice” to a stand alone repo, use requests to de-couple business logic in the main app and make your current REST-API ready for server-to-server communication)
- We can make use of any additional services provided by AWS in the realm of API Gateway (i.e. auth strategies to name one) and Lambda functions
- Using a message queue as integration point should futher provide the flexibility to decide on who consumes the messages and is reponsbile for handling the upates. From there on, consumers are indepentend and can be build and ran in whatever scenario you can think of.
By entering the API Gateway world, I assume we further get scalability, flexibility, security and many other things “out of the box” — to be validated…
At the same time, it seems that the overhead is little, as we “only” have to learn how to setup the REST endpoints (which seems to be fairly simple using serverless framework) and writing update workers which would anyway be required. We already have a message queue in place, to setting up the workers in the current structure is easy.
While there are so many possiblities out there to implement simple webhooks, this seems to be one where can reach our goals short-term, learn a lot and start a spark for a non-monolith mindset. Worth the effort in my humble opinion.
What do you think about that pattern? Please let me know, I most probably miss another interesting perspective on this. Happy to receive your feedback.