Migrating our content moderation platform to Kubernetes
Part 1: We decided to migrate from Mesos to Kubernetes! Find out why and how we started
By Sébastien Georget, Architect Lead
Serenity is the solution provided by Adevinta’s Trust & Transactions tribe to help marketplaces automatically moderate user generated content. It’s used by 14 marketplaces within Adevinta.
tl;dr: If you have hard deadlines, reengineering your application during migration may not be the best approach…
This article will introduce you to the early stages of our journey:
- The beginning: why we decided to move to Kubernetes
- The proof of concept: to get some ideas about potential challenges (Q3 2019)
- The re-engineering: to increase the value of the migration (Q4 2019)
The beginning
Two years ago the team started the discussion about leaving Mesos. The cluster was stable, our code too, the application was scaling pretty smoothly and we had almost no oncall triggered for infrastructure issues.
However… it had a cost. Not a direct one, since the incident rate was almost 0, but a hidden one: nobody wanted to touch the Mesos framework. It was almost 50,000 lines of C code and even if it was well coded and documented, the entry cost to change something was super-high. As a result, a bigger and bigger technical debt was starting to slow down our projects.
That’s why we decided to look at Kubernetes. We believed it would have several benefits for the team:
- We could use the Kubernetes orchestrator instead of maintaining ours (a big part of the Mesos framework was dedicated to managing and scaling deployments)
- We would reduce the learning curve needed to onboard new software engineers in the Serenity team
- We would benefit from the Kubernetes namespaces to easily create isolated environments, especially for developers (in Mesos we only had pre-production and production environments)
- Overall: we would reduce the time spent on the infrastructure which would give us more time to focus on the features we wanted to deliver!
So last year, when the Container Runtime Environment team told us that they were planning to stop their Mesos cluster, we decided to start our journey towards Kubernetes.
The proof of concept (July 2019)
- Our configuration is set through environment variables
The first step of our journey was to check that we were able to deploy and run Serenity in Kubernetes. It turned out to be pretty fast; in two weeks we were able to run our modules (Serenity components). It was possible due to the fact that all the components were already containerised to run in Mesos and were following some of the 12 factors that make migration easier:
- Our modules are stateless
- Our services are exposed through ports
- Our logs are sent to STDERR and STDOUT
As the modules part was easy, we just had to change some environment variables and start them in Kubernetes. We have really seen the benefit of containerisation: we were able to run our application in different environments without any changes thanks to docker!
We had another major component in Mesos: the router, which is in charge of calling the different moderation modules for a given content. We decided to port it (in C) to Python (which has many of our moderation modules). For the Proof of Concept (POC), it was pretty fast. However when we went to production we had to rework it to support some edge cases that we missed at the beginning and to improve its observability.
For the Kubernetes part itself, we started by writing Kubernetes manifests (resource configuration files), but we quickly saw that to deploy our stack, we would need more than that.
We decided to go with helm, which defines itself as “the package manager for Kubernetes”.
Our first approach was to create a “chart” for the whole Serenity stack and to use this chart for each tenant integration.
The whole thing was working pretty well: we were able to deploy our 14 integrations in Kubernetes (~ 800 pods), get logs and metrics, and run some basic tests.
This was the definition of “done” for our POC. We had reached our first milestone!
The re-engineering (October 2019)
(We had to pause our work on Kubernetes for some time due to operational issues and started again in October).
After the POC we decided to see what we could optimise during the Kubernetes migration. The idea was to generate more value during the migration than just changing the infrastructure.
Since Serenity is multi-tenant, we have a lot of configuration parameters to tune for each tenant. This allows us to fine tune the behavior of our solution for each marketplace. But when looking into the details, we found that the configuration between each marketplace was really close. So we decided to refactor our configuration management to switch from open parameters to feature flags (= just enabling some behaviors with booleans).
One side effect of this was that we started to create dedicated helm charts per module. Remember that in the POC we had one chart for everything? This approach was simpler: super readable configuration, ad-hoc deployment per component… But it was also quite time costly to implement, and the deadline was getting closer and closer (end of Q1 2020)…
Find out whether or not we made the deadline in Part Two!
