Bringing sanity to mobile release responsibilities
The ideas of mobile app updates and releases are still relatively new and unclear to a lot of people. While pushing to the web has become a well understood process, the advent of the App Store dictates different needs and requirements. Without some guidelines and boundaries, mobile releases can suck up a lot of time and resources, and worse yet, they can cause real problems for your users. At Runkeeper, we’ve developed a number of practices to manage releases and stay productive. In this post, we’ll discuss those practices, and hopefully give you ideas for how you can do the same.
This isn’t for everyone…
It’s worth noting before we go any further that this post won’t be for everyone. It is 100% true that many larger mobile companies (Facebook, Twitter, etc) have built powerful feature gates and release systems that allow them to distribute functionality to their users outside of an app update. This is great and fine, but if you’re like most mobile teams, you don’t have the time / resources / luxury of being able to build a system like this for everything in your app. At the opposite end of the spectrum, there are teams small enough (i.e. less than 5 people) where some of the practices below may solve problems they simply do not have. Ultimately, these processes may not make sense (yet!) and be too much overhead for what you’re trying to achieve. If you fit into either of these two categories, feel free to skip the rest of this article and simply flame these obviously wrong and harmful processes in the comments. :) For everyone else in between the giant company and the small team, read on.
Why do we need a mobile release process at all?
Let’s reset on why having a process for managing mobile releases is so unique (and important!). In a web development world, each feature / bug / whatever can be pushed out to the world discretely and on its own. Team A can push their work to users completely independently of Team B, make sure no problems happen, and then Team B can push hours or sometimes even minutes later. Strong web development teams will even boast about how many 100’s of times they push to production per day.
This is simply harder on mobile platforms, because mobile releases have much more bagga — err, let’s call them supporting artifacts. Release Notes. Screen Shots. Localized versions of both. Crash support files. And all this needs to be bundled together for submission to your app store of choice, where it will need to wait for review and propogation times. Of course, once your app update is finally in the store, remember it still has to get downloaded on to your users’ devices. These and other artifacts ultimately take time, making multiple releases a day incredibly difficult. At Runkeeper, that generally means we ship once or twice a month. And because of this frequency, each release tends to contain work from multiple teams across the company.
So with muptiple teams responsible for disparate features that need to be bundled with a variety of artifacts, who ends up responsible for all this?
Enter: Release Leads!
At Runkeeper, we find it’s helpful to identify one of our 5 mobile teams as being primarily responsible for the health and success of a release. Deciding on the responsible team is more art than science; we’ll look at what features are being delivered, key partnership deadlines, and domain expertise, and attempt to choose the team most equipped to create a successful release.
Within this team, we’ll identify a Dev Lead (typically one of the more senior engineers on the team), a QA Lead (typically the more senior QA on the team), and call out the Product Manager on the team as being the PM Lead. Each lead has a number of responsibilities consistent to every release identified on an internal doc. If you don’t have a similar doc… you should! There was a time as a smaller team we were able to simply remember everything that needed to be done in our heads. That time has long since passed! This post will go in depth on some of the more nuanced responsibilities of each lead, but we’ll also include our entire list at the bottom if you’re looking for a starting point.
One of the cornerstones of this system working is that everyone can push a release. This may feel counterintuitive to some who believe in a growing company, you need to start locking down permissions and only have the Grand Master of Releases be allowed to push things live. This is an anti-pattern. Empower everyone on your team to be responsible for the success and quality of a release. Not only will it improve your team’s happiness (who isn’t excited the first time they get to “push the button?”), but it will also save the more senior people on your team (maybe you, intrepid reader?) from spending all their time pushing out releases.
Dev Lead Responsibilities
Simply put, a Dev Release Lead is responsible for the most technical actions necessary to support a release: managing git, building and uploading releases to the appropriate app store, uploading DSYM files to Crashlytics, etc. Typically speaking, engineering is the only group with domain knowledge to complete these tasks. However, like most of the tasks in this article, we’re working to automate as many of these processes as possible. We’ve had a lot of luck with Fast Lane, but that’s a whole separate blog post (seriously; we wrote one). As an example, using a Slack command to run a build on a dedicated build machine and upload directly to the store saves us a ton of time.
One of the more nuanced responsibilities of the Dev Lead is what we affectionately refer to as “traffic copping”***. Essentially, as we go through the final stages of preparing for a release, a number of technical tasks are going to come up: fix this bug that cropped up in final regression, change the copy in these labels, etc. We take the opinion that the most effective and efficient engineer to fix these final issues is the engineer responsible for the original code. There are plenty of reasons for this. If John broke XYZ feature, John is more equipped to fix XYZ feature while keeping whatever he built for the release intact. If copy needs to be changed in new labels Jane created, she knows where similar text may reside on another screen and how to handle both scenarios. The Dev Lead’s responsibility in all this is to be the main point person for all incoming technical tasks gating the release, and playing traffic cop to distribute them out to the responsible engineer across all our mobile teams. Yes, this means that every team can potentially be pulled in to an upcoming release. However, assuming the work we’ve done leading up to the release is high quality to begin with, we generally find the release work managable.
QA Lead Responsibilities
As an established, mature app with 10 years of development and millions of users, the final testing of Runkeeper is incredibly important. Simply put, if you break things, you lose users. We have a variety of tools at our disposal to vet the quality of a given release including automated integration tests, manual regression, crowdsourced testing, and beta testing. With this range of tools, you can probably imagine bugs flying in from all of them. It’s the QA Release Lead’s responsibility to vet bugs as they come in and help determine what to do about them. Is this bug a regression? What’s the priority of this bug? Under what conditions does it happen?
Similarly, as these bugs come in, they effect the timelines of the release, as well as how much time the QA and other teams need to spend on the release. All this information has to be communicated to product and engineering, which the QA Lead is responsible for.
PM Lead Responsibilities
Finally, the Product or PM Release Lead is responsible for communicating and coordinating with everyone outside of this release process. Believe it or not, there are people outside of Product and Engineering who are interested in a release: Marketing, Support, Business Development, Management, etc. The reality is that these groups also play an integral part in the release process as well, including creating release notes, getting press for new features, managing new customer feedback, etc. A big part of PM’s role is coordinating with these groups, keeping them up to date on the release progress, and making sure that all the less technical artifacts (release notes, screenshots, etc) are created.
Release ownership doesn’t end when the build goes live
The final point we’ll leave you with is that regardless of what type of Release Lead you are, your job does not end as soon as the build is live. Release Lead responsibilities continue after the release is live to make sure that the build is accepted and adopted well by the user base. These are some of the most critical responsibilities of release leads: monitoring crashes, understanding user feedback and reviews, watching key analytics metrics, managing a patch release if necessary, etc. If you don’t finish the job by ensuring the release is in a good state before moving on, you’ll regret it in the long run.
Hopefully, this post has given you some ideas of how you can better corale your mobile releases. As promised, here’s a rough list of Release Lead responsibilities at Runkeeper:
Dev Release Lead:
- Branch the release in git (x.x-branch)
- Update the relevant Slack room topic to communicate the new branch name
- Change the release version (Info.plist or manifest.xml)
- Play traffic cop for regression bugs
- Merge all changes from release branch back to master
- Keep PM Lead appraised of dates
- When ready, create release build (both beta and app store)
- Tag release in git (“x.x-AppStore”)
- Merge branch to master
- Submit beta release
- When ready, submit app store release
- Upload DSYM to Crashlytics for iOS release
- Update internal release list documentation
- Watch crashlytics post release, traffic cop new crashes
- Make assessment on whether or not dot release is necessary
- If necessary, own dot release.
QA Release Lead:
- Upload build to crowdsourced testing platform
- Assess crowdsourced bugs, bring in to regression story if necessary
- Keep PM Lead appraised of dates
- Own regression cycle, traffic cop with other QA
- Coordinate beta feedback prerelease / support feedback post release
- Pulling in PM regularly as new bugs come in to keep bugs triaged & priorities clear
PM Release Lead:
- Finalize release notes 5 business days prior to app submission and submit to for translation.
- Coordinate with other PMs to make sure all stories are tagged with the appropriate fix version
- Create new version in iTunes Connect (if not already created)
- Finalize and upload release notes (including translations)
- Review release with support to flag potential risk areas or things to watch out for
- Finalize release comms with marketing (if applicable: blog post, PR outreach, screenshots, email announcement, social media)
- Communicate release status to team, especially if dates are changing
- Monitor app store reviews for bugs, user frustrations, or other feedback
*** Yeah, maybe we should just call this “directing traffic” instead of making a noun into a verb and butchering the English language. I’ll be sure to take those learnings into account next time we retro.