Leveraging A/B Testing to “soft disable” unused features and reduce unnecessary calls

Sharing our user-centric approach to reducing emissions through informed decisions.

As digital technology powers an increasing share of day-to-day human activities and commerce, its carbon footprint represents 3 to 4% of global emissions in 2019, with a 6% annual growth rate (source: The Shift Project).

With 10 billion page views per month and 8 million daily users (as of March 2023), leboncoin, a popular online marketplace, tech leader in France and part of Adevinta, has a growing responsibility to minimise its digital emissions.

After winning the 2022 edition, leboncoin participated in the 2023 cross-company “Sustainable Challenge” organised by APIdays to promote eco-design best practices. During the three-month challenge, each company participating should lead an internal digital project applying eco-design principles.

Our target: reduce unnecessary API calls without impacting operations

The cross-functional team participating in the challenge had identified some features with low user interaction yet high API consumption. This was due to where these features were implemented and the high traffic we have in leboncoin.

By disabling rarely-used features, we could cut down API calls and the associated carbon emissions. The question was: since some users are using the feature, how could we disable them without compromising our operations?

Soft-Disabling features

Instead of permanently removing features and potentially impacting some of our users, we adopted a user-centric and data-driven strategy. We implemented a “soft disabling” mechanism that deactivates features based on user interaction. This is accompanied by a transparent explanation of our approach and the opportunity for users to reactivate the feature if required.

The approach consists of:

1. Incrementing a counter when users see but do not interact with the target feature.
2. Disabling the feature once counters breach predefined thresholds.
3. Letting users know we deactivated the feature along with a block explaining why, in place of the feature.
4. Allowing users to easily reactivate the disabled feature with a toggle.

We believed this was a good balance between maintaining user experience and business outcomes while improving API call efficiency. But would it work in practice?

Putting our soft-disabling mechanism to the test

To accurately evaluate this idea, we turned to Adevinta’s in-house experimentation platform, Houston. Houston’s statistical engine and integration capabilities to leboncoin’s codebase and data platform had been improved recently, allowing for a rapid setup of an A/B test on Android.

We tested the soft-disabling concept on a feature showing nearby ads at the bottom of the Android homepage. This feature generated 2 million API calls per day with a 2% conversion rate.

Our experiment examined four variants:

1. Control group: Feature always enabled
2. Variant B: Disable after 30 views without clicks
3. Variant C: Disable after 50 views without clicks
4. Variant D: Disable after 100 views without clicks

Through Houston, we collected and monitored the below KPIs:

• The average number of API calls for this feature per user: This was the success KPI of our experiment to validate our hypothesis.
• The % of users who reactivated the feature: A learning KPI to measure user demand for the feature.
• Some business KPIs as health KPIs to measure the potential negative impact of removing the feature.

Houston leverages existing data pipelines to collect business KPIs, therefore adding them as health KPIs to our experiment came with little to no additional cost.

The outcomes

Once we reached the required sample size, we easily analysed the results in Houston UI and the results were clear. Compared to the control group, the number of API calls for this feature dropped significantly by 40–75% depending on the chosen threshold. Despite this, reactivation rates remained low (below the 2% identified usage) and we didn’t see negative impacts on our business KPIs.

Given the A/B test results, it was decided to roll out soft disabling on the tested feature. The mechanism will now likely be tested on the other platform and low interaction features to enable informed, data-driven decisions.

At the Sustainable Challenge’s conclusion, leboncoin was awarded the “Sobriety Prize” for this initiative. But more than an award, this project demonstrated that we can balance user-centric approach, data driven decisions and increased tech sobriety.

And by open-sourcing our approach through talks and articles like this, we hope to see the concept adopted and adapted at other companies — we need everyone onboard to reduce global emissions!