leboncoin commit’s life

leboncoin
leboncoin
May 14, 2020 · 7 min read

By Guillaume Chenuet (Infrastructure Lead Engineer)

As you may have noticed by reading our posts, leboncoin is growing fast, more than 250 developers spread across Feature Teams (e.g. search, ads list, identity). But having a lot of developers also means having a reliable workflow to let them develop and deploy as fast as possible. Hopefully, a team is dedicated to these subjects, the Engineering Productivity team, working on workflows and CI/CD pipelines.

To explain this, let’s take an example of a backend commit in our Golang mono-repository. Currently, 60+ developers are working on more than 200 microservices in this repository, which means a specific way of working together.

Let’s start with Julie, she’s working in the Ads feature team who’s in charge of the ads deposit on leboncoin.fr. and today she’s working on the opening of the Animals category for professionals.

After writing some code, Julie opened a new review on Gerrit with her changes.

Review

Gerrit is used at leboncoin as Git server and Code Review, aiming to provide an interface between developers and continuous integration (CI).

Gerrit is a commit review tool. Developers amend individual commits in response to feedback; it tracks each version of each commit, allowing developers to view the differences between versions, and to rebase individual commits while keeping shared history in the review interface.

This patch-based workflow is different than the pull request workflow used by several git platforms such as GitHub and GitLab. In the pull request workflow complete branches are reviewed and merged instead of Gerrit which all commits are reviewed and merged.

Once the review opened, Julie’s colleagues are going to be able to review its changes and also trigger the CI.

Pushing a new review to Gerrit create a review number ( for our example) and add a change-id into the commit.

The code review process includes that reviewers formally express their opinion about a change by voting on different labels. There are three main labels at leboncoin:

  • Code-Review: This is the human label and it can be translated into ‘I read the code and it seems reasonably correct’.
  • Quality: The CI tool is expected to use this label in order to vote on a change after running quality (e.g. linter) jobs.
  • Integration: The CI tool is expected to use this label in order to vote on a change after running integration (e.g. unit tests or builds) jobs.

For a change to be submittable, the latest patch set must have a in the Code-Review label and no . Thus on any patch set can block a submit, while on the latest patch set can enable it. Also, the CI should also approve this patch set by voting a on the Quality and Integration labels.

So, back to our review.

Hopefully after some patch sets, the CI has agreed to vote on the Quality and Integration labels. Then, Christopher, a backend developer in Julie’s FT votes to this review. Everything is ready and review can be merged!

For each patchset, users can view the logs and coverage results.

But wait a second… How does the CI work ?
We already wrote an article last year about our CI but many things have changed and it may be useful to do a quick reminder!

Continuous Integration

Let’s start with a quick introduction:

As you will know, we used Zuul as the CI tool which is a pipeline oriented project gating and automation system.

Zuul watches events in Gerrit (or Github) and matches those events to pipelines. If a match is found, it adds the changes to the pipeline and starts running related jobs.

The gate pipeline uses speculative execution to improve throughput. Changes are tested in parallel under the assumption that changes ahead in the queue will merge. If they do not, Zuul will abort and restart tests without the affected changes. This means that many changes may be tested in parallel while continuing to ensure that each commit is correctly tested.

Zuul uses Ansible as a testing language, which means, each tests is an Ansible Play or Role that you can run locally on your computer, making it easy for developers to write & debug tests.

These tests are running in Virtual Machines (Openstack, AWS, GCE), Containers (Kubernetes, Openshift) or Baremetal (Linux, MacOS ,Windows). Meaning you can run your iOS tests on Mac Mini for example!

When Julie opened her review, two pipelines were triggered by Zuul: and .

Tests in the check-quality pipeline is done and check-integration tests are in-progress for review 82502.

For each review, multiple tests can run in parallel (e.g. when developers modify a common golang library, 200+ unit tests are triggered).

If the associated jobs all report success, the pipeline reports back to Gerrit on specific labels the vote of , or if at least one of them fails, a .

Back to our example.

Julie’s teammates agreed with the change and voted on the label. In the same time, Zuul has performed tests without errors and voted on and labels. The commit is ready to be merged!

Julie submitted the commit, Gerrit merged the commit into the master branch of the repository, which triggered the Zuul pipeline. This pipeline is in charge of building the code and uploading new Docker images to the registry.

And they lived happily ever after.

Wait…The commit is merged but we need to deploy our code now.

Continuous Delivery

To deploy our micro services into the multiple stacks, we are using Concourse CI.

Concourse was dreamt up and built by Pivotal while working on the Cloud Foundry project and experiencing the interesting CI and CD challenges that it has. Concourse limits itself to three core concepts: tasks, resources, and the jobs that compose them.

Concourse uses a configuration-as-code approach that makes it much easier and more reliable to set up, replicate, and repave than other products which require clicking through a settings UI to configure.

In addition, your pipeline configuration is visualized in the web UI, taking only one click to get from a red (failed) box to see why it failed. The visualization also provides a useful “gut check” feedback loop — if it looks wrong in the UI, it probably is wrong. Platform engineers and app developers can manage state and trigger builds using a well documented CLI or a browser-based dashboard interface, depending on their preferences and experience levels.

To have a concrete look on Concourse and pipelines, you can browse Concourse or RabbitMQ CI (with huge pipelines).

Hopefully at leboncoin, our pipelines are smaller and simpler than RabbitMQ and we chose to have one pipeline per micro-service, meaning around 200 pipelines in Concourse.

As Julie had her change merged, Concourse will trigger all pipelines related to this commit.

Let’s take the example of the micro-service, which is one the micro-services involved in the Julie’s commit:

adsubmit-api micro-service deployment in Concourse CI

For this pipeline, we have two jobs: one triggered by resource when a commit related to is merged and another one triggered by resource when backend team will create a new tag on .

As the commit is now merged by Gerrit, resource has detected a new version of and will launch jobs associated to: and . Long story short, these jobs deploys the new Docker image on Kubernetes clusters into different QA namespaces for testing.

adsubmit-api image deployment on Kubernetes cluster into different QA namespaces for testing

Once deployed, QA analysts will perform non-regression tests (e.g. add a new ad) and validate the change.

If everything is working as expected, Julie will create a new tag in the git repository with her change.

Zuul will be triggered by this new tag and launch all related jobs to build new Docker images and add them to the registry thanks to the Zuul pipeline. (See Zuul dashboard screenshot above).

Once the new Docker image is uploaded into registry, the Concourse resource will be triggered , starting the job and deploying the new version of micro-service into our staging environment.

Last but not least, if everything is working as expected on staging env, Julie will be able to launch the last step; deploy into production environment!

We are currently working on automated tests allowing automatic production deployment.

Welcome to the animal category!

Congratulations Julie, the animals category is now live and available for professionals!

Our workflow may appear to you heavy compared to what smaller companies use or an all-in-one tool like GitLab. But it ensures a lot of flexibility and we can adapt it or fork it for specific usages, making it easy for us to enable new companies who are joining us to profit from what we built on, and it’s very robust too! As we say in France: “ne pas mettre tous ses oeufs dans le même panier” may be a good philosophy.

To give you an overview, each month leboncoin merges 13K+ commits, performs 60K+ CI tests and releases more than 1000 changes made with ❤️ by 250 developers. Not so bad :)

We hope this commit’s life story will give you a better perspective of how leboncoin-tech works and maybe some ideas for your own company.

leboncoin Engineering Blog

Learn more about creative engineers & data scientists building a French virtual Flea Market

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store