Changing the rails
This is part two of our series on how Catawiki started.
When an online business takes off, there is always an imminent desire to be at the forefront of technology. The need to feel confident of being on the right track and a fear of falling behind competitors can help push a company to succeed — or instead, hold it back. As we previously explained, Catawiki made several technological choices early on that contributed significantly to our success.
In 2008, there wasn’t a more prominent web-development framework than Ruby on Rails. Even though Ruby was never the most popular programming language, the shift of paradigm in web-development it introduced drove the framework to success. In this context — like many other emerging Web 2.0 businesses — using Ruby on Rails was the obvious choice to help us streamline our development flow and focus on things that matter: continuous innovation.
Challenges faced
Six years after Catawiki was founded, the time was right for a larger technological leap to support expansive growth facilitated by the global recognition the company was gaining¹. By the end of 2014, we faced the inevitable question: “How can we best support our business growth with our current technology, while keeping the promises made to our customers?”.
This question became even more complicated in the context of the challenges we had already been facing:
- Regular platform outages and ‘Fridays on fire’ caused by the end of week influx of visitors and bidders;
- On-premises infrastructure with a high degree of manual system management;
- Lack of development and staging automation, which constrained engineering team growth;
- Having a monolithic service that practically combined two different businesses: a platform for experts to review items and create auctions, plus a platform for collectors to find special objects and bid on them.
Tackling the issues
Although there was a lot of uncertainty at this point, one thing we knew for sure, was that tripling the engineering team to keep up with the increasing amounts of issues and business objectives would not be a sustainable solution. Instead, we had to rethink the way we were building our system for the future.
Of all the issues mentioned above, the one that had the most impact and hindered our progress on other problems was having a monolith system. Monoliths can be successful and often are the best way forward in the early stages of business development. During this period, businesses often need to prioritise product innovation over engineering quality. However, this creates a large amount of technical debt, which a successful company should pay off sooner or later when the business takes off.
Moving away from our monolith
At this stage, migrating to a microservice architecture seemed like the obvious solution, but we had little clarity on what the process or the end result should look like.
Starting such a large transformation poses significant risks. Data input and output, processing, error handling, and the UI — usually written once and reused across the entire monolith — represent a challenge for microservice design.
Plenty has been said about the advantages and disadvantages of microservices, but in our case the advantages outweighed the risks. As our team grew, we needed to split the application to make teams more agile and reduce the learning curve for new engineers. We also needed to run more instances of certain parts of the application, to tackle the performance issues mentioned above. Moreover, a microservice architecture would enable us to leverage new technologies without being forced to adopt an all-or-nothing approach, for example by rolling out these changes per service and prioritising the services where these changes would be most beneficial.
Focus on foundations
Organisation-wide standards are vital to the success of the microservices journey, so the first step was laying these foundations for our new architecture. To achieve this, we created a repository with an app skeleton, automation, and guidelines to enable teams to consistently deliver new services.
We also built libraries with essential tools to streamline integration. It is best to start early with an internal SDK, OpenAPI specification, and infrastructure as code. In these efforts, some monolith code can be reused or used as a blueprint for a common library that each app may depend on. However, due to their different nature, in some cases, it will be necessary to implement new solutions.
Focus on data
Another challenge of migrating to a microservice architecture is handling data. While an application layer is usually scaled by simply multiplying compute workload, the data layer eventually becomes a bottleneck.
Microservices can cause significant increases in costs if the data design is not done properly. Replacing in-process code execution with cross-host API calls costs user time, and is more expensive for businesses. The efficiency of any service depends on how data is stored and exchanged among services.
In fact, inefficient data structures and data-querying algorithms are the biggest problems to handle when scaling an application. Frederick Brooks, author of The Mythical Man-Month, says:
“The programmer’s primary weapon in the never-ending battle against slow systems is to change the intramodular structure. Our first response should be to reorganise the modules’ data structures.”
Therefore, it is of primary importance to categorise data and their usage, establish data exchange rules, and apply the right tools for specific use cases. It’s always difficult to change data structures at a later stage, so it’s best to ensure storage scales x10-x100 times to prevent future issues.
Getting down to work
Once the end result is clear, it is time to focus on how to get there. For an established business, a rapid rebuild from scratch is usually not an option, as it requires stopping feature development in the old system to avoid a never-ending cycle of catching up with updates. This means we have to build a new system that works alongside the old one, exchanging and duplicating data and features, to provide clients with a gradual transition away from the original system.
An effort like this can cost any company a huge amount of time and resources. Nevertheless, it should be seen as an investment since the organisational and technical benefits of a modern Service Oriented Architecture (and Microservices specifically) will easily outweigh any benefits of a monolith in the long run.
An organisation with hundreds of people is a complex structure. To make the transition successful and get buy-in from everyone involved, it is crucial to make plans with clear rationales and an understanding of the goals. Because of this, we built a company-wide Engineering Roadmap reflecting all the steps of the migration, accounting for the people we had and our planned hires for the year.
At the time of writing (August 2022), we are happy to share that our journey to microservices is almost complete. When our monolith is finally decommissioned in the upcoming weeks, we will be ready to face new challenges and, of course, celebrate.
Originally posted by Vladimir Bobrov on dev.to on 30 Nov 2020, edited on 26 Oct 2022.
