Why did we launch the Mobile release train?

In Bazaar, we faced few problems related to mobile releases that are:

Problems:

1. Previously, there was not a specific day to release the mobile app. Every time, we have to think about when should we release the build. It was difficult for business and engineering to get aligned for a specific release day.
2. The team didn’t have a clear picture of when to stop development and plan for release. Always had last minutes changes that haven’t properly grilled through the testing life cycle.
3. Most of the time engineers couldn't manage or take responsibility for release because of active sprint development.
4. Feedback and testing sessions couldn’t take place properly that can help a developer to resolve the issues efficiently.
5. Active sprints were also being hurt because of release monitoring and staged rollouts.

To resolve these problems, we decided to adopt the concept of mobile release train.

Mobile release train concept:

The mobile release train concept describes how to deliver software efficiently. It also helps a team to release features on schedule and avoid spillovers because of promoting the concept of small releases with a bunch of features or bug fixes. It can be used by small/big distributed teams who work on a single app. The train concept is also a part of the scaled agile framework.

Step by step guide:

Let’s take a look at a mobile release train.

In the following picture, you see a simple development phase, in most cases 2–4 weeks.

Step 1: Locking the release branch

On Monday morning, the release would be locked for any further modifications. Until this time, the teams will continuously review, test, and give feedback on the features that should be part of the train.

At 3 pm, the release captain will pull the code from the master branch to the release-candidate branch, either manual or in an automated way.

Step 2: Testing the release branch

This release branch will get a final internal validation and quality testing phase. In this phase, all team members should check that the new features are working as expected. If there is a problem on the release branch the bug will be fixed on the master branch and later cherry-pick to the release branch.

Step 3: Proceeding with new development cycle

Once we lock the release, a new development cycle will start and all contributing teams can start a new sprint and continue with the development. The great thing about the mobile release train is, that everyone knows the next planned release and can plan their work accordingly.

Problem : How to monitor the previous release ?

We faced a problem where the whole team had to monitor the previous release along with the current development phase so we also introduced the concept of release captains that will only be responsible for the release and other members of the team can continue the current sprint development.

Release captain

There must be one team or person handling the release. Either it’s always the same person/team or it is a shared effort among all developers. In any case, one person must be the responsible “release captain.” This person is responsible for getting all required information for the release (e.g. release notes, new store images, and so on). Furthermore, this person can inform in every standup to the rest of the team about ongoing releases and problems that occur on the current live version.

Let’s continue toward the release train concept. You can look at the extended version of the mobile release train in the following picture.

Step 4: Upload the app to beta channel

Once the app has been tested internally, the release captain uploads the app to the beta channel. In this channel, real customers will receive an update notification on their devices for testing. During the beta testing phase, beta testers can give direct feedback via the play store. Next to the feedback, the crashes can be monitored in monitoring tools

Step 5: Push the app to staged rollout (Last step)

After the beta testing phase, the app will be pushed to the staged rollout. In this stage, the release captain can define how many percentages of the users will get the update. In this case, the steps are 10%, 20%, and 50% rollout. In each stage, the reviews, as well as the crash tools, must be monitored to check for possible problems.

If a problem gets visible in the 20% stage, the team has the chance to react to the problem and use the mobile bug matrix to decide if the problem is worth a hotfix. If this is the case, the train should not reach the next rollout step of 50%. Instead, the problem must be fixed for 20% of the users. Once the problem is fixed and verified the train can move on to 50%.

The concept of feature flags:

I also want to draw your attention towards one more excellent concept of feature flags that helped us very much to rollouts seamlessly. It empowers you to enable/disable any feature that is creating bugs in systems, percentage rollouts, and enable testing on production.

A feature flag is a software development process used to enable or disable functionality remotely without deploying code. New features can be deployed without making them visible to users. Feature flags help decouple deployment from release letting you manage the full lifecycle of a feature.

Fruit of this activity:

As a result:

1. We built the track in our system so our train can run smoothly on track and reach the stations on schedule.
2. Now business and the engineering team already know the schedule of release so they align their tasks accordingly.
3. Only release captain(s) have to look out for the previous release and all other team members can continue current sprint development
4. Now, we all have a clear picture of when our feature cycle can be completed from development to customer handover.

How to establish your Mobile Release Train ?

Before you start to implement the mobile release train in your company, you must define the time slots for each stage.

For example 2 weeks (10 working days) for development. But then, you need to define how much time you want to spend doing the integration testing, beta testing, and rollout stages.

The train can be broken like:

1. 10 days of development & feedbacks
2. 2 days of integration testing
3. 2 days beta testing (It can also be 5%, real users)
4. 2 days 10% of the users
5. 2 days 20% of the users
6. 2 days 50% of the users

In total, it takes 20 days until a new app version is live for 100% of your customers.

8 important aspects to follow:

Establishing and using a mobile release train sounds great, but keeping the process up and running can be tough. Here are some aspects you should follow:

1. One team/person must handle the release coordination and the release itself.
2. Always use feature flags for big features so our release can’t be delayed due to any reason and we have the power to keep it disabled in release and after a fix, we can enable it again.
3. Have a weekly/bi-weekly release standup to exchange release status between the contributing teams.
4. Only accept reviewed and tested code.
5. Only accept code that has unit tests and end-2-end test automation in place (if needed).
6. Don’t accept late passengers. Don’t merge code to the release branch which was still in development on release lock day. This code hasn’t been reviewed and tested enough.
7. Use stage-rollout.
8. Monitoring the released app version is crucial.

In case you missed:

In one of my last blog posts, I wrote about Avoiding multiple concurrent requests to rotate Refresh Token and how a big system can be hurt due to overusing concurrency and discussed one big issue that can be faced in results and how can we fix it through synchronisation code blocks.

— Shahab Rauf