We lost 10% of leboncoin traffic… and that’s good news!
By Aude Février, Product owner & Renaud Godefroy, Web developer & Océane Staron, Back-end engineer
At leboncoin, the Promote team is responsible for delivering the right communication in the most efficient place to the right user at the right time. The website and mobile applications use a pop-in, a header banner, and several other placements to communicate messages to users in real time, such as service shutdowns or discounts on delivery.
To display the right content for the connected user, a call to the backend is necessary for each page and location. Messages are then filtered: A discount for the package delivery company Mondial Relay, for instance, will be displayed only if the concerned ad is eligible.
However, we noticed that 85% of API calls ended in 404 errors since the backend didn’t have anything to display (especially on the header banner, which appears on every page of the website but has little content available).
So as part of the second edition of the Sustainable Digital Challenge held by Apidays in 2022, we decided to work on a proof of concept (POC) to place a feature flag on the header banner placement. Although it was a good start, the solution turned out not to be perfect in a scaling context.
In this piece, we will discuss the technical details of the final solution we chose and how it improved our team’s performance.
Feature flag POC
Our efforts during 2022’s Sustainable Digital Challenge paid off as we were able to save 30% of our API calls each week (out of a total of 522 million) by using a feature flag on the header banner placement. With this feature flag, we could turn off placements that were not in use at a particular time.
Although it was quite a good performance, we realized that the feature flag was not fluid and responsive enough for our needs. From time to time, we need to activate the placement quickly, for instance when communicating about an issue with one of our services, like messaging.
So we began searching for better solutions using our experience in production.
An urgent need for scalability
Our growth ambitions, resulting in a significant technical transformation project, prompted us to accelerate our solution.
As part of the project, our Feature team completed a scalability assessment to help us determine if our infrastructure would be able to handle the traffic increase. We realized our current design would lead to high latencies if left unchanged (which would be particularly frustrating since we are responsible for notifying users of website issues) that simply pouring more money into won’t resolve, as infrastructure performance doesn’t scale linearly with cost.
We had to take action!
The chosen solution: Call optimization
Technical details
It took one sprint to develop this call optimization feature, so two weeks (plus integration time for the mobile applications and the frontend). It was put into production on December 7, 2023.
Call optimization consists of a new endpoint returning a list of locations with content available to users. Frontend and mobile applications were already sending us user information so the backend could determine if content was available. Now, before doing that, the frontend and mobile applications call the endpoint to get a list of locations containing available content for them.
This solution does not require a cache of the content, which allows us to manage more easily any legal and product constraints that sometimes force us to quickly deactivate campaigns (for example, if there is an issue or if the campaign has finished). The only cache that exists is a 15-minute one used on the frontend for the locations list. This way, frontend and mobile applications only make a call when the location is in this list.
One of our services experienced a decrease in requests the very same day we deployed the solution.
The result was both satisfying and frightening at the same time. Our next step was to look at the global impact it had.
Impacts
Impact on microservices
The number of 404 calls to our services reduced by 71.93%. While this proved the effectiveness of the solution, it was the total number of API calls that was the real indicator. In the week following the deployment, we saved 3.62 billion calls (from frontend to backend, and from backend to backend).
The number of replicas we use is another interesting indicator. Replicas are clones created by the production instance of the service when there are too many requests to handle. Before the deployment, we sometimes had 120 replicas, which is quite a lot. Now, with the same replicas’ configuration, we never exceed 40.
Finally, we were able to save 2.7 billion API calls frontend to backend, which represents 10% of leboncoin’s global traffic.
Impact on the database
To handle API calls, your microservice needs to connect to the database to find out whether some content is available. Therefore, the number of database connections is a good indicator of infrastructure improvement.
Within a week, we had reduced the number of connections to the database by 172,450, or 25%.
But how about the CPU utilization of the database? In the past, we sometimes used up to 60% of the CPU when handling requests, but now we don’t exceed 20% without modifying the database at all. As a result, our database has become too large, so we may have to scale it down (a nice victory for us).
Impact on costs
Reduced infrastructure also means lower costs. In more concrete terms, we reduced our Kubernetes costs to a third of what we were paying before.
As a result, the Promote team is no longer included in the top 10 most expensive departments at leboncoin, which we are very glad about.
Impact on sustainability
As a reminder, part of the Sustainable Digital Challenge was about coming up with an eco-responsible approach to solutions, and we were able to achieve that with this project as we ended up using fewer machines, thereby reducing our carbon footprint.
A month after the deployment of the chosen solution, we noticed that 49.66 cores had been saved. The CPU reduction we achieved was equivalent to turning off one of the AWS instances of type c6g.12xlarge (based on the technical specifications).
In order to estimate the carbon footprint of this workload, we used the carbon footprint estimator created by Teads Engineering. By doing this, we worked out that 24 kg of CO2 had been saved, the equivalent of 14 tons per year.
Main achievements
Implementing the call optimization solution allowed us to:
- take advantage of our experience to rethink our software solution: While our previous software solution worked, it wasn’t a perfect fit when it came to production
- meet the scalability-assessment requirements of our internal project
- reduce the size and cost of our infrastructure (and we could go even further by reducing the size of the database)
- make our solution more sustainable: Whenever we deploy new features now, we try to consider the ecological impacts and financial costs
The best part is that it was all done without causing any damage to the product! Feature-wise, our teams get the same benefits, but under the hood it runs much more efficiently. The impact on the user is not obvious but still there: Their mobile phone will not heat up as much, we will save bandwidth, and the website will load faster. Furthermore, we have a more maintainable and clean code that will last longer.
Prioritizing highly technical projects can be very difficult. It’s tricky because they are critical to the health of your code and being guaranteed a reliable and durable product. Is that something you’ve experienced too? Why not comment below and we can talk more about it!
