The present and future of app release at Skyscanner

7 min readAug 23, 2016

Posted on August 23, 2016 by Tamas Chrenoczy-Nagy

Since we’ve just released our 3-in-1 update for the Skyscanner app, it’s a great time to share some info with you about how we release iOS and Android apps at Skyscanner. In this post we would like to give you a brief overview of our app release process, how we adopted the de facto industry standards and extended them to our needs. Beside this we also would like to give you a couple of ideas where we are planning to improve in the future.

We’ll cover:

Separating code drops from feature releases
Release trains
Feature flags
Faster release cycles
Future ideas and improvements

Decoupling releasing features from releasing the binary

Imagine a company where multiple teams are working on the same app. The teams own different functions, or different screens of the app. They would like to work and deliver new features independently. So if team A has some kind of delay in releasing a new feature, it shouldn’t block team B to release their own feature on time. The company should be able to release a new version of the app even if not all new features are ready for it.

It is also very important to be confident that the new feature is delivering value to our users. Firstly we release only to a small number of users, and depending on their reaction we continue the rollout to all.

To achieve these goals we had to decouple releasing a feature from releasing the app binary. We put the new features behind a feature flag and only turn them on if the feature is ready to be released in production. Meanwhile we release the new binaries in a fixed schedule, so teams don’t need to synchronize their feature delivery, the schedule is available and they can plan ahead easily.

Release train — shipping the new binary on schedule

Releasing the binary follows a fixed schedule, let’s call it a release train. So if you finished your new feature on time, you can release it with the upcoming release train. If you are not ready on time and you missed the train and then you will be able to release the feature with a next train.

On both iOS and Android we have a 2-week release cycle, which mean we ship a new binary bi-weekly. The releases on the two platforms are held on alternate weeks, so each week either a new iOS or Android release train starts.

Feature flags — the tool for actually releasing features

By using feature flags, we are able to specify which features the users should see when they run the application. The feature flags can be modified remotely, so we are able to turn on and off features without releasing a new binary.

Feature flags provide a lot of value at four different stages of development:

Feature is under development: If the feature is still under development and it is not ready to be released to any users, the flag is only turned on for developers — so they can iterate on the feature and commit changes without causing any user disruption.
Rolling out a new feature: If the feature is ready to be released, first we usually turn it on for only a small amount of users (for instance 1%), so we can measure the reaction of the users and the quality of the feature (by checking analytics data — key business metrics, errors, crashes). If everything seems to be ok, we increase the rollout, sometimes in multiple steps, for instance 10%, 100%.
A/B testing: This scenario is similar to point 2 but with variations of what we release. We would like to experiment with a new feature before releasing it to all of our users — so sometimes we can ship multiple variants of the feature and measure which one performs the best.
Kill switch: If we have a feature which has already gone live and something goes wrong (i.e. the app starts crashing because of it), we can turn it off remotely.

To release the 3-in-1 update we also used feature flags. Early versions of the feature have been presented in the app binary since last December. At its early stages it was only turned on only in our development builds, then in our test builds. After we finished the development we released to 1%, 5%, 20%, 50% and finally to 100% of our users using the feature flags. Meanwhile we continuously monitored the performance of the feature. After the 100% rollout we also kept the feature flag as a kill switch for a while, so if something really bad had happened we could have still rolled it back.

Releasing a new binary frequently

Increasing the frequency of releases is a key part of keeping the pace in delivery. Frequent releases help us to experiment and iterate on our features faster. We can get valuable data about user behaviour faster so we can react on them faster. As of now we have a 2-week release cycle on both android and iOS and we are planning to further increase the frequency.

Each Wednesday morning we have a code freeze on either iOS or Android. After the code freeze we start a ~1 week long stability period. During this period our main goal is to get confident in the stability of our new release, so we are continuously testing the app and working on fixing critical/major bugs.

After we tested and fixed all critical bugs in the new release, we still cannot be confident that the app will work well for all of our users. There can be some issues which can be detected only in production. That’s the reason why on Android at least, we use the staged rollout feature of Google Play. During staged rollout the new release is rolled out to the users gradually (for only 1% first, then 10%, 100%). We are continuously checking key business metrics (like search rate, conversion rate), and also error rate, crash rate, etc. In case the metrics look good we increase the rollout, but in case we detect any major issue, we try to disable the feature using feature flags and ship an update with the next train. If it isn’t possible we release a hotfix to fix the problem. The staged rollout usually takes about one week.

After we validated the new version in production as well (using staged rollout) we release the app globally. But the story doesn’t end here. We keep monitoring the app metrics, user feedback/reviews, crashes and react if necessary.

What’s next? — There is a lot what we can improve

Shipping a new app version to our users in every two weeks is a good thing. It is good because we can quickly iterate and release new fetaures. But is it good enough? Let’s do some maths on how long it takes for one new feature to get to the users.

Let’s say development of this feature takes two weeks. If you sum up the length of the release process , you can see it takes an extra two weeks to get that feature shipped to our users. Beside this — if the feature isn’t finished on time and it misses the release train, that adds an extra 2 weeks again.

So shipping a new feature can take from 4 to 6 weeks to get to 100% rollout. If we need to iterate multiple times on the feature and make several experiments or adjustments, it is even longer. This is way too much compared to a web environment with a continuous delivery flow, where you can release the feature almost immediately. Can we achieve the same for apps? We believe so, in time.

Beside the development areas in our processes, we are facing several constraints coming from the nature of apps. One of these was the review process in App Store, which in the past took at least one full week. Fortunately Apple has worked hard to bring this down — as little as 1–2 days now, so it should not act as a blocker.

Another constraint is that apps are shipped to the users in big packages and app updates are still a big thing — and even more so if we ever had to perform a rollback of something that couldn’t quickly be hidden with feature flags.

However we can see some promising improvements in the industry which might result in making app releases a less heavy thing. Google’s Instant Apps feature is the best example and worth checking out.

So applying full continuous delivery for apps is not something which is possible for apps at this moment in time. But we can still apply best practices from web and use them to increase our release frequency to get that bit closer.

Hopefully the industry will also move in a direction which supports our goals, so if our processes and tooling are sound, we should be able to capitalise on any changes without major lifting.

Tamas Chrenoczy-Nagy is a product engineer working on improving release and development processes, tools for apps.

This blogpost is part of our apps series on Codevoyagers. Check out our previous posts:

The present and future of app release at Skyscanner

Cooking up an alternative: making a ‘knocking gesture’ controlled app

How we improved the quality of app analytics data by 10x

Designing Mobile First Service APIs

How We Migrated Our Objective C Projects to Swift — Step By Step

Analytics and Data Driven Development in Apps

Transitioning From Objective C to Swift in 4 Steps — Without Rewriting The Existing Code