Release Strategy — a twisted tale in apps
Four years back, on a beautiful late afternoon, one of our leads and a product owner were debating about when that game-changing feature would get released on AppStore. The next day, another product owner came to the same lead and had the same debate. I am sure, many of you must have encountered (or still encountering) these scenarios. But today we don’t see that show anymore. Now you would wonder, how? It was not something that we hit with one slingshot. We had our own learning curve to find a state where we are now.
Our Challenges
Beforehand, we didn’t really care about our release schedule. The release could happen monthly or even once every two months. Indeed this “random release date” makes a lot of problems.
The Waiting Game
We work with many teams which are domain-specific. And every team has its own objectives which turn into features, bug fixes and improvements, and yes they have their own schedule too. We understand that and quickly get the job done. But whether you are ok with it or not, the feature can’t be released even though the development is done. Why? Because there is always a feature almost done, and somehow we need to wait for it.
Customer Satisfaction
The higher our app rating, the more we get the trust for our apps. That is why ratings and reviews are always crucial, it’s like the brand value of our apps. And you know customers always expect their problems getting resolved ASAP. Customer satisfaction is directly proportional to the speed at which we solve the bug that converts into the higher app ratings. This is also one of the core values, our DNAs
FOCUS ON CONSUMER.
Question of the Day
Imagine you are coding happily and a product owner comes to your desk, questioning — When is the app release date? You know when that concerned feature would ok-tested-done. But still, you don’t have the answer to that question. So you rush to your leads, shooting the same question and they make a conditionally definite statement —Maybe on <Some Date> if <Some Dependency>
Trust me, even they don’t have the exact answer for that. And this was every scenario back then.
Solving the Maze
Now starts our path to an ever-evolving solution to this everyday problem.
The First Iteration
We came up with the Cut-Off System. Basically, the concept was to make a strict schedule for our release date. One week of development, followed by five days of code review, and three days of regression. So three weeks in total (Of course we don’t count the weekend). This scheduled release surely helped us to make better planning and prioritization of our tasks. But, in the company this large, and the market this competitive, the changes needed to be faster.
Why Walk if We Can Run
Then we decided to make our release schedule biweekly. Aside from only shortening our schedule, we moved from fixed development to parallel development. While the QA team is making sure we are up to the mark in the current release, the developers can work on the features for the next release. So the time we need to develop new feature still the same as before. Yes, this is the first time we started using a release train strategy. 🎉
“Speed will follow when the mechanism of the movements is more assured.” ― Rafael Sabatini, Scaramouche
Can We Run Faster?
Of course, we can. When we need to go somewhere by train, if the train comes hourly, we will try to catch the train at any cost which has the least waiting time. What if the train comes every five minutes? Yes, frequency matters. We can just relax as we can always take the next train.
Weekly Release Train Strategy
You see the picture above that’s how we can move to weekly releases from biweekly releases. To make it possible, we needed the entire mobile apps team to collaborate. The developers would always be coding and reviewing code from others. The Test Engineers will have their own share of responsibilities, and so on. Also, we have special days with defined scopes for all the roles.
Monday is the Code Freeze day so merge before it or otherwise wait for the next train. In parallel, we release the previous version to 1% of the users — Phased Release. The % of users will increase every day.
Tuesday is the Bug Bash day! Generally, we spend an hour to focus on selective areas of the product and discover as many bugs as possible. This event is hosted by QA Team and the participants are — all the related Product Owners. And how about the developers? They focus on the next feature.
Wednesday — The developers should fix the bugs which were found in the Bug Bash. If any feature has too many bugs, we must revert it and move it to the next train.
Thursday, the regression day. We perform regression every Thursday. Along with manual regression, we also run all the UI tests.
Friday — TGIF. Finally, we can submit apps to the App Store. If you wonder why we submit the app on Friday — to makes Apple review on Saturday 😈. Just Kidding!!
Support the Speed
“It’s always an advantage to move quickly, but we’re not going to sacrifice quality for speed.” ― Dwayne Nielson
Once the app is live and it is in users’ hands, we can’t revert it. So to minimize the risk, we implement Feature Toggling using feature flags using Firebase Remote Config.
The invalid test case and bug need to be discovered earlier. Tokopedia app is designed to be a multipurpose app with many features. So, we need to make sure that everyday changes don’t affect previous code or another feature. And hence, we write Unit and UI tests.
We are releasing app updates every week and that needs us to create release builds for submission. There is a checklist that we need to go through before release, update the version, running our unit and UI tests, and since we have two code repositories (native & react-native), we need to synchronize it. We all are engineers and we should spend our time in solving problems creatively. To save our time from doing these trivial things and prevent from possible human errors, we have created a Cron Job to do all that.
We need some tools to enable us to operate at this speed. We use Fastlane to create the builds and running the tests. We use CircleCI for continuous integration and manage the cron job.
Conclusion
As I said in the starting, this path was never straight. We struggled but to find a sustainable solution. Our creativity made us come up with one solution after another. Embrace the process, continuously improve the velocity without sacrificing the quality. With great teamwork and suitable tools, we can make it. It is not just a story, it is a quest. A quest to always find a better solution. As we say at Tokopedia — Kami Baru Mulai (English translation — We are just getting started). Cheers!!🎉
