Best Practices in iOS App Release

Making mistake is an unavoidable part of software development. In Agile, which encourages small and fast delivery, we tend to commit even more mistakes during delivery. And app release issues are something that no Software Engineer will ever want to commit, but this doesn’t mean we should be over-protecting and aiming for no production issues at all time, this will just cost more testing time instead. In this article, I will share some simple practices for iOS app release management, which help to avoid big mistakes, reduce big impact, act quickly to fatal issues and make release process less painful.

Why iOS app release is painful?

As an iOS app developer, we know that there are a few steps to release a new app version:

• Pack the whole app bundle into a binary build and make sure code signing is working correctly
• Submit the build to App Store Connect and wait for review process which may take 1 day
• The build must comply to App Store Review Guidelines, we will get app rejection for violations and must submit a new build to fix them
• Once the build gets approval, the new version is published through App Store and it may takes >3 days to get 80% traffic from this new version

As we can see, it takes up to 4–5 days to get our app available to all users. Imagine we have a critical issue in this new version, it will be too late to wait for users’ complaints and start working on a hot fix (the hot fix release will also carry the same risk). The broken version is already stuck in users’ devices and we have no ways to roll back that, except waiting for them to update to hot fix version.

To cope with this painful process, our release techniques should follow a few principles:

• Always use Phased Release
• Have built-in rollback capability for big initiatives
• Allow enough time for internal/beta testing
• Combine changes as pack and avoid minor release
• Monitor new release before increasing traffic

Release changes as pack

Release Train

Release train is a fixed schedule for submitting new app versions, we should have weekly schedule or bi-weekly schedule for it. Some sample rules for a Release Train:

• Code freeze by end of Thursday
• Reserve Friday for internal/beta testing new build
• Allow committing only bug fixes after code freeze
• Submit to App Store Review by end of Friday
• Should get approval during weekend
• Release with phrased release on Monday morning
• Monitor new release 3 days then release to all users
• All teams must align on the same release schedule and we should plan feature releases that match with app releases cycle

Benefits from Release Train:

• By locking changes with code freeze, we avoid unexpected issues of random changes
• We have Friday for internal testing and weekend for beta testing, any critical issues can easily be pointed out during this period
• App rejection may happen and we still have time to resolve it during weekend
• By releasing on Monday, we have full team support if critical issues found. This is very important, the root cause of an issue should be found quickly in order to prepare for a hot fix asap. Sometime, an issue can easily be fixed with back-end deployment so we also need back-end team be available
• We have 3 days to monitor the new release, this is long enough to discover abnormal behavior from logs or analytics
• After releasing to all users, we still have 4 days to get all users updated the app before driving a new version on the next release cycle
• With a fixed schedule, we save development resources and alignment time since every team will share the same release timeline

Release Manager

Release Train is a complicated process, we should have a dedicated engineer to handle it and avoid misalignment. That doesn’t mean this engineer will work fully on managing release (otherwise he will resign soon), we should have a rotation between all engineers to take role as Release Manager.

Release Manager responsibilities:

• Prepare stable builds for testing and submitting
• Communicate and coordinate to resolve release issues
• Submit app for review
• Solve Apple rejection and feedback (if any)
• Release 1% and monitor new release
• Release 100% if no critical issues found
• Chase for fix and create a new release if any critical issue found

CI/CD

CI/CD is an important part for a smooth release, we should never use an individual laptop to archive builds (this may cause slow performance or mis-configuration). A good CI/CD setup should have these criteria:

• Should have 1 UI test that cover app happy flow
• New codes should have at least 80% code coverage by unit tests
• Archiving should take less than 60m

Archiving time is another pain point when deliver new iOS builds, it takes quite sometime (around 20–30m) to pack app bundle into a binary build. If our project settings enable bitcode, TestFlight takes even more to process the build (sometime the build just hang there), we may end up waiting for 1–2h to get the build ready for install.

A few techniques to deliver testing builds quickly and reduce idle time:

• Reserve adhoc builds for internal testing. Adhoc builds are very fast to make and installation is easily done via link sharing. However, we must whitelist the testing devices with a limit of 100 devices, for that reason we should reserve these devices for QA, dev, designer, stakeholder to help them quickly check the new release
• Beta testing builds should be handled via TestFlight since it allows more testing devices, also TestFlight build is the closest one to AppStore release
• TestFlight builds should be scheduled at night to reduce idle time of build processing
• Bug fixes should be verified using adhoc builds if they have minor changes

Monitoring

After so much effort to develop, archive, verify, adjust, submit build, we come to the final step: RELEASE!!! This is obviously the most stressful step, despite how careful we prepare for the release, production issues are still unavoidable. Therefore, we must prepare ourselves some powerful tools to quickly detect, act, response to reduce production issues impact as much as possible

Some key metrics to monitor new release:

• Crash free rate
• Conversion rate
• Error rate
• New app rating
• Performance metrics (ex: app launch time, API load time)

How to react to incident:

• Pause phased release to prevent more updates to broken version (new installs are still with this latest version though)
• Investigate and trace back the root cause of issue, prepare for hot fix asap
• Look for feature toggle if the issue comes from a new feature. All new features should have feature toggle since they have a high chance of causing new issues. Feature toggles also help to decouple app release with feature release
• Look for quick fixes from back-end. Some issues may come from false response from API or can be fixed by a small change in API response, a quick deployment is the best option to resolve those issues

A few tools that help to investigate production issues faster:

• Changelog: every release should come with a changelog that list down all new commits from this release. Also, tasks should be tagged with the release (ex: v4.39.0-b3939)
• Release notes: which is very helpful to explain spikes
• Analytics: a crash log or error log should come with analytics logs ahead of it, which is very useful to reproduce issue

Example of analytics logs

Hot fix

In most cases, we should avoid random release in the middle of release train, because it will cause a rush for testing & rollout and can bring even more serious issues. However, in some cases, a new release is mandatory to get rid of the fatal issue.

For that reason, we must develop a hot fix trigger base on monitoring metrics. For example, I can build my hot fix trigger like this:

• Crash free rate drops below 99%
• Conversion rate decreases by 5%
• Error rate increases by 5%
• New 5 rating with 1 star
• App launch time increases by 30%
• API load time increases by 30%

We should also make sure the hot fix is saved for release by using these techniques:

• It should be built on top of current release
• It should add only the changes to fix fatal issues
• It should have a built-in rollback option (either by feature toggle or API response)

Trunk Based Development

A source-control branching model, where developers collaborate on code in a single branch called ‘trunk’, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell, do not break the build, and live happily ever after. Reference: https://trunkbaseddevelopment.com/

Trunk Based Development is a very powerful branching model which also help to early detect obvious issues during development. Some benefits of Trunk Based Development:

• Smaller PRs to review
• Easy tasks breakdown
• Drive more testing resource (from other developers), obvious issues can sometime be found by teammates
• Continuously release small chunk of code to production hence reduce the risk of big feature release