The art of staging a rollout
On leveraging one of the greatest tools for publishing Android apps
No matter if you’re an independent developer, if you work for a startup or if you are employed in a huge international company: every user counts. It is among our ultimate goals to provide a valid service to users, but at the same time to keep them engaged and to gain their trust.
Therefore, we need to treat our users with extreme care — a broken experience could scare them away forever.
This poses a difficulty that, compared to other platforms, is particularly harsh on Android, due to the countless combination of OS versions, manufacturers, and hardware. We need to make sure that our product works on every device (or at least, on a vast majority of them), and given that we cannot possibly have access to every Android device in the market, we need to test the product in production. But how do we test in production without the risk of endangering our entire user base with the threat of malfunctions and crashes?
Curtains up. Enter the staged rollout.
What is a staged rollout
To explain the concept using Google’s own words, a staged rollout is a way for your update to reach only a percentage of your users, which you can increase over time.
This lets us release an update only to a small subset of the user base, monitor its stability or any other metric of interest, and decide whether to roll out the update to more users, to wait for more data to come in, or to halt the rollout altogether.
In general, a staged rollout follows the following rules:
- From your perspective, users who take part in the staged rollout (i.e. that receive the update) are chosen randomly; there is no way for you to target users based on some property
- From a user’s perspective, there is no way of telling if they received a regular update or an update from a staged rollout
- Every new staged rollout uses a potentially different user group (again, users are selected in a random fashion)
- Staged rollouts can only be used in production (no alpha or beta channel)
- Staged rollouts can only be used when updating an existing app
- There is almost no minimum percentage, you can target as low as 0.01%
If everything goes well, you will start a staged rollout, monitor the metrics and keep increasing the percentage until you reach 100%.
… but what if something goes wrong?
Halting a staged rollout
Say that one of the metrics you’re monitoring is off, or you check the user reviews and you see some complaints on a specific topic. In order to prevent more users from potentially experiencing wrong behaviors, you can halt (as in, pause) the rollout. Users in the staged rollout that haven’t installed the new version will no longer be able to receive the update and will only have access to the last fully published version. This lets you analyze the situation without the fear of everything burning around you!
At this point, several scenarios might unfold.
You might find out that the metric wasn’t that off, or that those crashes were related to a third party, so you can decide to resume the rollout and to even increase the percentage.
Or, as Murphy would have it, your app is severely broken and requires some proper fixing that takes quite some time. You will then start all over again with another rollout and see from there.
There’s a third scenario, perhaps the most interesting and lesser known. Assume that you introduced a problem that is now compromising 30% of your user base. You are not sure about the solution, so you need to release an update. Ideally, you would want that update to be released to the same set of users, but I mentioned before that every staged rollout targets a different random set of users. Or does it?
Playing with percentages
Until some time ago, the Google Play Console wouldn’t allow you to update a staged rollout with a percentage lower than the current one. However, a couple of weeks ago I’ve noticed this:
When you halt and then resume the rollout of your release, you’ll be affecting the same set of users. [1]
and shortly after this:
When you do a staged rollout of a new release before completing the rollout of the previous release, the new release will use the same group of users as the previous release (depending on the percentage of the rollout). [1]
Back to our example, where we were stuck with a bad staged rollout at 30%, we could release an update to 5%, but those users would still be part of the previous “30%” group! This is great, because it allows for testing the fix only on users that were actually affected by the problem.
However, it is unclear what happens if you try to reduce the percentage even further. Do you target the “5%” group or some other users within the “30%” group? That “depending on the percentage of the rollout” is not very specific.
Bottom line
Staged rollouts are incredibly powerful and beneficial. I can’t think of a valid reason why you wouldn’t be using them almost every single time, given the fact that they come with a lot of advantages and literally zero disadvantages. At Clue we use them extensively, and we recently had a situation where we successfully reduced the percentage of the rollout to better understand the situation. Moreover, the console is extremely well organized and gives you a lot of information and context around present and past staged rollouts.
App is stable. Users are happy. Curtains down.
