Automating the release process

Hi there! My name is Daniel Kheyfets and I am a QA Lead for mobile at hh.ru. In this article I will tell you how we automated the flow of mobile application release, how we divided it into stages, what tools we used and what problems we faced.

This is a text transcript of the video on our channel, so if you prefer to watch it instead of reading it, welcome to our Youtube channel.

In previous episodes, I’ve talked about how we’ve improved our mobile app release processes, what practices we’ve introduced, what problems we’ve encountered, and what we’ve come up with in the end.

In the previous episode

We switched to GitHub flow, wrote autotests and ran them on CI, launched Release Train, and introduced the practice of Duty. Thus, we achieved stable, regular, weekly releases. But there were still a lot of routines we wanted to get rid of.

Routine

All the routine for mobile app release can be divided into two parts: technical and managerial.

Technical routine:

• creating a release branch
• version increment
• upload to the app stores
• autotest run
• build launch

Managerial routine:

• release description
• what to write about in the “What’s New” section of the Stores
• setting a task for marketing to write “What’s New” texts

Although all these operations are not complicated, they require human participation. Accordingly, the most common human mistakes are not excluded: someone forgot something or didn’t do something, and so on. Of course, we had checklists and instructions for such situations. However, even this did not save us from small but unpleasant problems.

To eliminate the human factor, we decided to automate the process, and we found the task interesting.

Work planning

Before automating, we decided to visualize the entire process of our release: how it looks, what statuses it takes, what causes the statuses to change.

The stages that make up our release flow:

• Creating a release
• Running autotests
• Launching the release build
• Uploading to Google Play / App Gallery / Apple Store
• Closing release

There are transitions and events between stages that cause the release statuses to change. This is where relatively harmless events can happen, as well as real trouble. For example, if we find a bug/crash in the production, we need to fix it, run autotests again, rebuild the app, and upload it again. And we can also get rejected at a Store review, and then we have to go through exactly the same cycle of checks and rebuilds again.

Our plan turned out to be universal enough that we were able to describe it at once for all our apps (jobseeker /employer), platforms (Android, iOS) and Stores (Google Play/App Store/Huawei App Gallery).

Work detalization

After describing the release process, we decided to analyze in detail what specific steps would be taken at each stage. We dug around in our checklists, instructions, invited a ̶s̶h̶a̶m̶a̶n̶ ̶t̶o̶ ̶s̶h̶a̶m̶a̶n̶i̶z̶e̶ ̶w̶i̶t̶h̶ ̶a̶ ̶t̶a̶m̶b̶o̶u̶r̶i̶n̶e̶ and described it all in a large and detailed diagram.

Detailed release process

After taking a closer look at the scheme, we realized that at each stage there is a certain set of atomic actions that are, for the most part, independent of each other.

For example, at the stage of creating the release we:

• create a separate branch for a future release
• increment the application version
• look for a list of tasks completed since the last release
• ask marketing to write text for a future release based on the list of completed tasks.

We prioritized each step: something we wanted to automate first, something we could wait on, and something we might like to automate, but we haven’t had the opportunity yet.

Excellent: the release stages are clear, the actions at each stage are clear, it’s time to make a plan for how to implement it all.

We started by explaining to management why we needed this task. If we do this task, it will save us a lot of time and help us to avoid mistakes in manual release management. We’ve clearly outlined deadlines for automating each step. This all helped us prioritize the task for automation, and we finally got the go-ahead.

Let’s get the work done!

Implementation

Data structure

First of all, we decided how we were going to store all the data. We decided that we would describe a simple PostgreSQL database of two tables: the main one storing all release data (release number, branch name, creation date, Jira ticket link, release status, etc.), and an auxiliary one for information about marketing tasks.

A piece if the main table

Okay, we’ve figured out how we’re going to store the data, now we’re going to decide how to interact with it.

Data visualization

A typical Tuesday of the hh team before the release automations:

• “Has the release branch been created?”
• “Did you run the autotests?”
• “What’s the current version, help me!”
• “Where are the texts for the stores?!”

As you can see, the problem we had was that no one ever knew what the status of a release was. Had the apps been uploaded to the store? Has the old release been rolled out? Where are the actual texts? So many questions, so few answers.

Thanks to the database we created, we have a source of truth about the releases, but going to the database every time you want to see what happens there wasn’t comfortable.

So we decided that to visualize this process we would use the tool we usually work with — Jira. We made a board, where each column symbolized a particular stage in our release process. Such as: launching autotests, what autotests passed, uploading to the stores, or rolling out to users.

Jira board with releases

And we also created a special type of task for this board — “release”. In this task we attached all the necessary information that we wanted to know about the release: what its status was, whether it passed autotests, a link to run these autotests, as well as the tasks that were included in it.

A release ticket

In addition, the task displays the dates of each stage. So we can easily see on what date and at what time each stage was passed.

Thus, we managed to visualize all the most important information about our release and we don’t need to make manual queries to the database to do it. Anyone can now simply open the board in Jira and see what the status of a particular stage is.

How do you manage a release?

But displaying the database status in Jira is just half of the task. You also have to modify it, and to do that you have to come up with a scheme to manage it all.

We decided to manage our releases from the same Jira. By moving the ticket there, we can start one or another stage of our automation. When the status of the release task changes, we’ll start a special plan on CI, which will start doing all the necessary work.

Since our CI server is Bamboo from Atlassian, it was pretty easy to make it all work.

There was only one question: what to write automation scripts in?

We investigated existing tools and came to the conclusion that the most convenient one for us would be fastlane. There were several reasons for this choice.

First, fastlane is probably the only convenient way to release and automate anything related to iOS apps. In addition, the iOS team has used this tool in their existing automations.

Secondly, fastlane has many implemented plugins, actions that we need for our tasks, which means we don’t have to write everything from scratch.

Third, fastlane can work with both iOS and Android applications. This will allow you not to write automation for each platform, but just use a single tool, without duplicating code and basically make everything much faster. You can see our effing story about using fastlane for Android.

What’s under the hood of the release train

In our automation system we got four main parts: the PostgreSQL database, the Bamboo CI server, fastlane itself, and Jira.

As I wrote above, thanks to the easy integration of Jira and Bamboo, we made it so that when we move a Jira release ticket from one status to another, we run a special plan on Bamboo. And it is triggered by a special lane fastlane.

Each stage of release in the code looks like a separate lane fastlane, which defines everything that needs to be done at this release stage. The script goes to the database and determines what status the last existing release is in. There are only two statuses: EXIST or RELEASED. If the status of the most recent release we have is RELEASED, then there is no release, and we need to create a new one. If the status of the most recent release is EXIST, then there is a release in progress right now. And we need to go to Jira to clarify the stage.

The actions description for each release step in the code looks like a separate lane fastlane.

All of the actions during the release can be broken down into four large groups:

• Work with DB;
• Work with Jira;
• Work with GitHub;
• Some kind of auxiliary utilities.

For each task type, we have described Ruby wrapper classes in the code. We use the Jira-ruby library to work with Jira and the standard Postgres library to work with the database. We also wrote a small client to send queries to the database and handle errors. We also described the necessary commands for the git and other utilities. For example, the code for sending notifications to Slack.

We got such a universal set of functions, which we put in a common repository, and use both in iOS and Android. We created such a plugin to be connected that we use when developing in the iOS repository or in Android to avoid code duplication.

What’s the bottom line

We did quite an extensive job of automating our release, and we got a lot done.

1. We eliminated all the manual steps to create a release branch, increase the version number, run autotests, build and upload to the stores.
2. We got rid of the routine of describing the release composition: our scripts automatically analyze merge commits to the development branch and generate a list of tasks that make it into the release. All necessary tasks in Jira related to the work of other departments are also created automatically.
3. We have added many notifications of changes in release status for all concerned: for managers, we send notifications of releases rolled out in the production, for testers, about the readiness of builds for regression, about successful (and not really) runs of autotests.
4. And, most importantly, we’ve visualized our entire release process. Jira is now a single source of truth about the release, now no one has any questions about what’s going on with the release.

The solution, of course, is far from perfect — somewhere there are technical errors, but step by step we get rid of the human factor in important stages of our releases.

Has it gotten better?

Has it gotten better since we implemented our automations? Definitely yes: it’s much easier and more convenient for us to drop releases now. But there’s always room for growth.

We’ve built a foundation on which we can continue to thread many other automations that may appear in the future. We already have several tasks waiting to be done:

1. We want to learn how to change the status of different product tasks, depending on the release status. Now we occasionally forget to put different labels to tasks in Jira, or move them to the final workflow status.
2. We want to automatically roll out apps not only to App Distribution, but also to the stores themselves, including Huawei App Gallery for Android. At the same time, we want to roll out apps to internal, beta, and release tracks as well.
3. It would be great to learn how to automatically start Jira tasks for new crashes and bugs after they appear in production, as well as some mass user complaints.

That’s it. Automate the routine, comment, share your findings. Bye!