A/B Experiments: Successful Showcase from a Solo Developer
Recently, we tasked our developer community with conducting A/B experiments by using flags, and the results have been illuminating. Our team unanimously selected a project titled “Wall to Wall” as the most intriguing experiment. We congratulate Max Born from JustPlay on this remarkable achievement and aim to dive deeper into his distinctive experiment through this article.
- What hypothesis was tested?
The developer aimed to assess the impact of altering game balance on game metrics, specifically focusing on the effect of increased rewards on playtime.
- What kind of game?
This is a sandbox-style game that pits two teams against each other and allows player to deploy troops, vehicles, and objects strategically. The objective is to defeat the opposing team and earn rewards to acquire new troops within the game.
- What flags were implemented?
The developer used two parameters of game balance into flags: enhancing rewards for level completion and increasing incentives for watching rewarded ads.
All players, totaling 100%, participated in the experiment and were divided into three key groups:
- Control Variant: The original game balance with no alterations to rewards.
- Medium Rewards: Level completion rewards increased by 1.5 times, and rewards for watching ads raised from 20 to 50 coins.
- Abundance: Level completion rewards increased by 2 times, and players received 100 coins for watching rewarded ads.
Results
Playtime: Players in both groups that received more rewards spent significantly more time playing, increasing by over 17% compared to the original group. Interestingly, the group with medium rewards showed an even stronger effect of 21% than the group with the highest rewards.
Interstitial Ads: Instances of interstitial ads, typically displayed between levels, also surged by over 15%. Once again, the Medium Rewards group displayed a bigger increase in ad views.
Rewarded Ads: In the Abundance group, where rewards were significantly bigger, rewarded ad views dropped by over 30%, indicating a reduced need among players to replenish in-game currency through ads. Conversely, the Medium Rewards group maintained stable ad views, underscoring the importance of balanced rewards.
Conclusion
The Medium rewards group experienced a notable increase in average playtime per player, resulting in a higher frequency of interstitial ad views while maintaining rewarded ad views.
Conversely, the “Abundance” group saw a similar increase in playtime but witnessed a significant drop in rewarded ad views, likely due to the surplus of in-game currency, reducing the necessity for additional rewards.
It’s essential to analyze results based on significant changes, as shown by color-coded values in the Games Console interface.
In this case, the developer successfully found a game balance that increased playtime by 20% and boosted full-screen ad views while keeping rewarded ad views stable.
Simply raising values doesn’t always lead to the best outcomes. A/B testing and experimentation, similar to what top game developers do, are crucial for finding metrics that improve certain game aspects without harming others.
We encourage you to apply these insights and tools in your projects, avoiding blind decisions and wishing you success in your game development journey.
For more information on A/B experiments, check our documentation: https://yandex.ru/dev/games/doc/en/console/vq-test
