leboncoin commit’s life
By Guillaume Chenuet (Infrastructure Lead Engineer)
As you may have noticed by reading our posts, leboncoin is growing fast, more than 250 developers spread across Feature Teams (e.g. search, ads list, identity). But having a lot of developers also means having a reliable workflow to let them develop and deploy as fast as possible. Hopefully, a team is dedicated to these subjects, the Engineering Productivity team, working on workflows and CI/CD pipelines.
To explain this, let’s take an example of a backend commit in our Golang mono-repository. Currently, 60+ developers are working on more than 200 microservices in this repository, which means a specific way of working together.
Let’s start with Julie, she’s working in the Ads feature team who’s in charge of the ads deposit on leboncoin.fr. and today she’s working on the opening of the Animals category for professionals.
After writing some code, Julie opened a new review on Gerrit with her changes.
Review
Gerrit is used at leboncoin as Git server and Code Review, aiming to provide an interface between developers and continuous integration (CI).
Gerrit is a commit review tool. Developers amend individual commits in response to feedback; it tracks each version of each commit, allowing developers to view the differences between versions, and to rebase individual commits while keeping shared history in the review interface.
This patch-based workflow is different than the pull request workflow used by several git platforms such as GitHub and GitLab. In the pull request workflow complete branches are reviewed and merged instead of Gerrit which all commits are reviewed and merged.
Once the review opened, Julie’s colleagues are going to be able to review its changes and also trigger the CI.
Pushing a new review to Gerrit create a review number ( 82502 for our example) and add a change-id (Ic176f7963c771534458cc6efee1849fd11b524af6)into the commit.
The code review process includes that reviewers formally express their opinion about a change by voting on different labels. There are three main labels at leboncoin:
- Code-Review: This is the human label and it can be translated into ‘I read the code and it seems reasonably correct’.
- Quality: The CI tool is expected to use this label in order to vote on a change after running quality (e.g. linter) jobs.
- Integration: The CI tool is expected to use this label in order to vote on a change after running integration (e.g. unit tests or builds) jobs.
For a change to be submittable, the latest patch set must have a +2 in the Code-Review label and no -2. Thus -2 on any patch set can block a submit, while +2 on the latest patch set can enable it. Also, the CI should also approve this patch set by voting a +1 on the Quality and Integration labels.
So, back to our review.
Hopefully after some patch sets, the CI has agreed to vote +1on the Quality and Integration labels. Then, Christopher, a backend developer in Julie’s FT votes +2 to this review. Everything is ready and review can be merged!
But wait a second… How does the CI work ?
We already wrote an article last year about our CI but many things have changed and it may be useful to do a quick reminder!
Continuous Integration
Let’s start with a quick introduction:
As you will know, we used Zuul as the CI tool which is a pipeline oriented project gating and automation system.
Zuul watches events in Gerrit (or Github) and matches those events to pipelines. If a match is found, it adds the changes to the pipeline and starts running related jobs.
The gate pipeline uses speculative execution to improve throughput. Changes are tested in parallel under the assumption that changes ahead in the queue will merge. If they do not, Zuul will abort and restart tests without the affected changes. This means that many changes may be tested in parallel while continuing to ensure that each commit is correctly tested.
Zuul uses Ansible as a testing language, which means, each tests is an Ansible Play or Role that you can run locally on your computer, making it easy for developers to write & debug tests.
These tests are running in Virtual Machines (Openstack, AWS, GCE), Containers (Kubernetes, Openshift) or Baremetal (Linux, MacOS ,Windows). Meaning you can run your iOS tests on Mac Mini for example!
When Julie opened her review, two pipelines were triggered by Zuul: check-quality and check-integration .
For each review, multiple tests can run in parallel (e.g. when developers modify a common golang library, 200+ unit tests are triggered).
If the associated jobs all report success, the pipeline reports back to Gerrit on specific labels the vote of +1, or if at least one of them fails, a -1 .
Back to our example.
Julie’s teammates agreed with the change and voted +2 on the Code-Review label. In the same time, Zuul has performed tests without errors and voted +1 on Quality and Integration labels. The commit is ready to be merged!
Julie submitted the commit, Gerrit merged the commit into the master branch of the repository, which triggered the Zuul Post pipeline. This pipeline is in charge of building the code and uploading new Docker images to the registry.
And they lived happily ever after.
Wait…The commit is merged but we need to deploy our code now.
Continuous Delivery
To deploy our micro services into the multiple stacks, we are using Concourse CI.
Concourse was dreamt up and built by Pivotal while working on the Cloud Foundry project and experiencing the interesting CI and CD challenges that it has. Concourse limits itself to three core concepts: tasks, resources, and the jobs that compose them.
Concourse uses a configuration-as-code approach that makes it much easier and more reliable to set up, replicate, and repave than other products which require clicking through a settings UI to configure.
In addition, your pipeline configuration is visualized in the web UI, taking only one click to get from a red (failed) box to see why it failed. The visualization also provides a useful “gut check” feedback loop — if it looks wrong in the UI, it probably is wrong. Platform engineers and app developers can manage state and trigger builds using a well documented CLI or a browser-based dashboard interface, depending on their preferences and experience levels.
To have a concrete look on Concourse and pipelines, you can browse Concourse or RabbitMQ CI (with huge pipelines).
Hopefully at leboncoin, our pipelines are smaller and simpler than RabbitMQ and we chose to have one pipeline per micro-service, meaning around 200 pipelines in Concourse.
As Julie had her change merged, Concourse will trigger all pipelines related to this commit.
Let’s take the example of the adsubmit-api micro-service, which is one the micro-services involved in the Julie’s commit:
For this pipeline, we have two jobs: one triggered by zuul-publish resource when a commit related to adsubmit-api is merged and another one triggered by zuul-release resource when backend team will create a new tag on adsubmit-api .
As the commit is now merged by Gerrit, zuul-publish resource has detected a new version of adsubmit-api and will launch jobs associated to: QA-ALL-EQX and QA-ALL-ILD. Long story short, these jobs deploys the new Docker image on Kubernetes clusters into different QA namespaces for testing.
Once deployed, QA analysts will perform non-regression tests (e.g. add a new ad) and validate the change.
If everything is working as expected, Julie will create a new tag in the git repository with her change.
Zuul will be triggered by this new tag and launch all related jobs to build new Docker images and add them to the registry thanks to the release Zuul pipeline. (See Zuul dashboard screenshot above).
Once the new Docker image is uploaded into registry, the Concourse zuul-release resource will be triggered , starting the K8S-PREPROD job and deploying the new version of adsubmit-api micro-service into our staging environment.
Last but not least, if everything is working as expected on staging env, Julie will be able to launch the last step; deploy into production environment!
We are currently working on automated tests allowing automatic production deployment.
Congratulations Julie, the animals category is now live and available for professionals!
Our workflow may appear to you heavy compared to what smaller companies use or an all-in-one tool like GitLab. But it ensures a lot of flexibility and we can adapt it or fork it for specific usages, making it easy for us to enable new companies who are joining us to profit from what we built on, and it’s very robust too! As we say in France: “ne pas mettre tous ses oeufs dans le même panier” may be a good philosophy.
To give you an overview, each month leboncoin merges 13K+ commits, performs 60K+ CI tests and releases more than 1000 changes made with ❤️ by 250 developers. Not so bad :)
We hope this commit’s life story will give you a better perspective of how leboncoin-tech works and maybe some ideas for your own company.
