How we build microservices locally, scaling docker-compose

On my first day at Qonto, I was told to install an internal tool we called qonto-env. It installed all the tools and cloned all the git repositories I needed for work. It felt like magic.

But, with Qonto expanding growth, going from a dozen developers to ten times more, what was magic became a nightmare. With an ever-growing number of services, developers struggled to maintain qonto-env.

We decided to take the bull by the horns and change how developers worked in their local environment.

How qonto-env came to be

What’s one of the fastest ways to get developers onboarded on a project? The obvious answer is docker-compose: it lets you start a whole container-based development environment with a single command. That’s what we started with, and then we built qonto-env in Ruby as a wrapper around docker-compose. It handled the detection of dependencies and the installation of docker-machine, parameterized the compose configuration file using ERB, Ruby’s templating system, and it had built-in commands to manage the services.

. 
├── bin 
│   └── qonto 
├── lib 
│   ├── qonto 
│   │   ├── docker 
│   │   │   ├── docker-compose-backend.yml.erb 
│   │   │   ├── docker-compose-front.yml.erb 
│   │   │   ├── docker-compose-kafka.yml.erb 
│   │   │   ├── docker-compose.yml.erb 
│   │   │   └── environments 
│   │   └── load.rb 
│   └── qonto.rb 
└── qonto-env.gemspec

This approach worked well for a team of 10 engineers, handling around 10 applications, all written in Ruby. Then, we started building our CBS (Core Banking System), written in Golang, and onboarding more and more new developers. New joiners struggled to maintain and debug qonto-env, which became a real black box to them. Adding new services to the docker-compose stack became a real pain. Every time something in qonto-env failed, people had to investigate its source code to understand how it interacted with docker and docker-compose under the hood. It didn’t scale with Qonto. We went from onboarding within a day to within weeks.

Finding the right balance

We went back to the drawing board. What did we want from this system? An easy setup for developers to run services locally and a setup that is close enough to the production setup.

In production, we use Kubernetes as a container orchestrator. What if we used Kubernetes locally? There are tools, such as Skaffold or Garden, to develop locally for Kubernetes, but:

• Developers don’t need to be kubectl gurus to develop their applications
• Maintenance of the local cluster would have been a major bummer
• Running all the microservices locally would have burnt the developers’ laptops from the sheer energy consumed.

The only dependencies for the local development environment should be Docker and docker-compose.

That said, we shouldn’t have to hide docker-compose behind a wrapper:

• docker-compose already has everything built-in
• wrappers introduce bugs
• wrappers hide stuff

Asking developers to use docker-compose is not the end of the story. How are going to organize our docker-compose files?

Distributing docker-compose

Here we are again, starting from scratch with a docker-compose file as a starting point. But this time, unlike my first day at Qonto, the file contained a hundred services. Maintaining such a file raised a lot of questions: which team would be in charge, how to introduce a change… We needed a way to split this huge file into smaller ones and spread it across multiple repositories, without relying on a wrapper or a templating system, of course.

Fortunately, docker-compose provides the ability to use multiple configuration files:

docker-compose \
  -f docker-compose.svc1.yml -f docker-compose.svc2.yml up -d

Now, it’s only a matter of organization!

So, we can put each compose file in each application’s repository. This way, each team is responsible for maintaining its application compose file, and it’s now easy to structure the layout of each repository with a common set of files:

qonto-env-compose.yml
dev.env
Dockerfile

But entering the path to each docker-compose file each time you had to enter a docker-compose command would be an awful developer experience. Thankfully, docker-compose offers an environment variable: $COMPOSE_FILE, where you can specify the list of all the compose files you want to use for your project.

$ echo $COMPOSE_FILE
/Users/amine/Qonto/secret-banking-project/qonto-env-compose.yml:/Users/amine/Devops/even-more-secret-project/qonto-env-compose.yml

But different developers may have cloned a different set of repositories on their computer. Or the team responsible for customer registration may not need the core banking repositories.

So, to make it easier for developers, we can simply use the find command to aggregate the compose files, and then export $COMPOSE_FILE :

$ find . -maxdepth 4 -type f \( -iname qonto-env-compose.yml -o -iname qonto-env-compose.yaml \)
/Users/amine/Qonto/service1/qonto-env-compose.yml
/Users/amine/Qonto/service2/qonto-env-compose.yml
...

Thus, we prepared a very straightforward shell script that does that for everyone:

$ devenv_activate
-> Looking for qonto-env-compose.yml in /Users/amine/Qonto
-> Looking for qonto-env-compose.yml in /Users/amine/Devops
-> Looking for qonto-env-compose.yml in /Users/amine/Data
[dev@qonto] $ docker-compose up -d secret-banking-project
Creating network "qonto_default" with the default driver
Creating volume "qonto_data_pg" with default driver
Creating volume "qonto_data_es" with default driver
Creating volume "qonto_data_redis" with default driver
[dev@qonto] $ docker-compose exec postgres psql postgresql://AzureDiamond:hunter2@postgres/secret-banking-project

From there, everyone can use the docker-compose commands they already know: no learning necessary and no wrappers with errors that are hard to investigate.

One last thing: the dependencies to common infrastructure services (Traefik, PostgreSQL, Redis, ElasticSearch, …) are bundled with devenv_activate shell script and we included all the configuration files that set them up for every developer :

├─ common-compose.yml
├─ traefik
│  ├── dynamic_conf.toml
│  └── traefik.toml
├─ elasticsearch
│  └── elasticsearch.env
└─ postgres
   └── postgres.env

This guarantees that developers have the main dependencies running, with the same configurations and versions, when launching the services they need to work on.

It brings us to a no-brainer development environment where everyone can quickly get up and running, working on improving Qonto.

Conclusion

Changing is one thing, but actually improving requires:

• communicating in order to ensure a smooth transition to the new system
• analyzing the effects and measuring the improvements.

For the first part, we organized team workshops and training to ensure everyone understood the new system and spread its knowledge.

During the training, we measured how much time it took for a new joiner to have a service up and running from scratch: it took less than an hour… when it previously took three hours with qonto-env!

Too much abstraction on something people already know defeats the purpose of simplifying its use.

Qonto is a finance solution designed for SMEs and freelancers founded in 2016 by Steve Anavi and Alexandre Prot. Since our launch in July 2017, Qonto has made business financing easy for more than 200,000 companies.

Business owners save time thanks to Qonto’s streamlined account set-up, an intuitive day-to-day user experience with unlimited transaction history, accounting exports, and a practical expense management feature.

They have more control, whilst being able to give their teams autonomy via real-time notifications and a user-rights management system.

They have improved visibility on cash-flows through tools such as smart dashboards, transaction auto-tagging, and cash-flow monitoring.

They also enjoy stellar customer support at a fair and transparent price.

Interested in joining a challenging and game-changing company? Consult our job offers!