Modular and Re-usable Docker Environments with Carnotzet
We created a tool integrating Maven and Docker to help hundreds of our engineers manage complex development environments and work with hundreds of services with minimal effort. This is the story of how a crazy idea became reality. This is the story of Carnotzet.
It all started about 5 years ago, Swissquote was growing fast. We had about 70 engineers working on large java applications that required everyone between 1 to 2 days of configuration to run on their machines. We hate to lose time on repetitive tasks ! So we decided to improve this process by using Vagrant and Chef to automate our development environment configuration. This was our first Sandbox project.
We like to share lightweight, reproducible, isolated and portable development and testing environments
For some time, it was good. Developers could simply checkout a project, run “vagrant up” and work on their projects. We used this technique for about two years and were pretty happy with it.
But then those big applications started to become difficult to work with due to the increasing complexity of our business (white-labeling of our trading platform). So we decided to do what most organizations in 2014 would have done : split the business in microservices.
This went well and by 2016 we had about 150 services in production. But the Chef code used to provision VMs had grown out of control. The development environments were now more complex with some application depending on about 30 services. We also discovered that knowledge of the Chef framework is not part of the usual skill set of our developers. So a lot of people just hacked it until it worked, sometimes copying bad examples. To avoid breaking the environment of others teams, they created a long lived branch for their own team and stopped sharing their environment definition code.
We had to fix it.
We identified the following weaknesses in our “sandbox” :
- Knowledge of Chef required. For existing employees and newcomers, learning this system automation framework was expensive. It was overkill for automating only development/test environments.
- No easy way to abstract/compose/re-use parts of environment configuration. This meant that people had to understand all the environmental details of their downstream dependencies, usually maintained by other teams. This lead to friction and unnecessary cross-team interactions.
To fix the first point, we decided to switch to Docker. It has a softer learning curve and also enables other advantages for our organization such as standardization, PaaS support and simpler promotion between environments.
We also looked at docker-compose. It looked very promising but despite it’s name, it lacked the composition, abstraction and re-usability aspects that we wanted. Our dream was to be able to take a full environment, add one service and optionally override some configuration. We wanted to do that without having to copy-paste the environment definitions or having to understand the details of transitive dependencies and their configuration.
This is when we had this wild idea : What if we took Maven, the dependency management system that we use to build our java applications, and use it to also build our environment definitions. This would allow us to abstract transitive dependencies, package configuration in a versioned and re-usable manner. Also we would not have to learn a new dependency management system.
After some iterations, we ended up with the following solution :
- Each maven artifact represents a fully functional environment that can be imported as a dependency in other environments. (a minimal environment is a single service such as a postgres DB for example)
- JAR files contain configuration. Environment variables, application’s config files, the name of the docker image to use, etc...
- This configuration can be overridden by upstream dependencies if needed, but is re-used by default.
- A java library can spin up a full environment (using docker-compose internally) from a single maven coordinate (GAV or pom.xml)
Maven allows for the distribution and versioning of those “environment artifacts” and we can easily split their source code between teams. Things became easier to manage and developers started using the modules from other teams and maintaining their own.
As a bonus, all tooling integrated with Maven can be used, for example the dependency graphs generated by IntelliJ IDEA now give us free auto-generated architecture diagrams :)
While there were some barriers for adoption such as teaching all teams to package their applications with Docker and package their configuration in Maven modules. The general feeling is that it became easier to import applications from other teams and maintain complex environments.
We started working on this about 1 year ago and we now have about 200 of those environment modules that are used for development and tests. We discovered that the library is also great to manage short-lived environments for end-to-end tests in our CI platform.
We are investigating how we can further re-use this technique to manage some of our UAT / Integration environments by spinning them up on Kubernetes instead of docker-compose to enable rolling-updates and computing elasticity in those bigger environments.
You’ll be happy to know that the project has been open-sourced on the Apache 2.0 License and is available on Github here: https://github.com/swissquote/carnotzet
Enjoy :)
