Cadence, Choo Choo!!

Over the last four years the mobile team at Strava has shipped more than fifty releases across Android and iOS. Our release cycle has varied from over three months for huge monolithic features to less than a week for hotfixes of issues discovered shortly after releasing. Historically app updates and new feature releases have been tightly coupled. As demands on each release escalated we saw release dates slip and quality jump around due to making concessions trying to push a release out the door. After going through a particularly ambitious release in early Spring 2015 we needed to reassess how and when we release.

“The Lumens Express” by Nick Fisher is under CC BY-ND 2.0

At the end of Summer 2015 we shipped a landmark update. The difference between this release and the scores of prior releases was not a groundbreaking feature or an important bug fix, the difference was that this release went out on its own schedule: it was not dictated by the need to release a certain feature and it was not a response to an emergency. It marks a move to a constant four-week long release cycle and an important shift in how mobile engineering and product development is done at Strava. We call it Cadence.

As Strava grew the number of projects we tackled simultaneously and the number of stakeholders in those projects grew as well. A few difficult releases illustrated how hard it is to align goals and deadlines across a growing number of teams. With enough stakeholders feature-based releases inevitably slip. As the deadline for a release approaches everyone lobbies to jam in their feature. Scope gets cut, bugs are introduced, unplanned hotfixes have to happen, and it all snowballs into a large, complex, stressful and often indeterminate release cycle.

Over the past few months we’ve adopted and defined a set of practices to help development and product releases stay agile. By decoupling feature releases from app updates we have intentionally attempted to reduce the stress of having to have a feature completed on a hard date, and eliminated the anxiety and ill-effects that happen when such deadlines are inevitably missed. We’ve also found that we’ve developed a rhythm for prioritizing and fixing bugs as well as releasing an app of continually increasing quality. Strava has always been quality focused and we’ve undertaken an effort to ensure that focus is built into the testing and release process itself.

We’re currently on our sixth iteration of the Cadence release cycle and instead of having one large team stressing out about working on one large feature we have a few more nimble teams, concurrently working on several features, and most of the stress has dissolved.

How it Works

Cadence is based on a four-week release cycle and a set of practices that determine what goes into each release. We’ve adopted some of the metaphors graciously loaned to us by friends at other companies. At the start of each four-week release cycle, or train, a train branch is cut from our main development branch, master. We work on the assumption that master is stable and always in a shippable state. To do this we ensure that all features that are currently in development get hidden by feature switches which can be controlled easily in code or remotely from a server driven API.

There are two types of work that get released: bug fixes and new features. Work on bugs can be targeted at a given train branch and will go out to the public when that train leaves the station after four weeks. This process is how we prioritize bug fixes. A bug gets fixed on the current train by targeting the train and then gets cherry-picked back into master. Opportunistic work on future bugs can always go in master, but we try to encourage developers to fix bugs on the current train and we continually go through a triage process to ensure that we’re addressing bugs in a timely and efficient manner.

Work on features continues on master in parallel. Features can span multiple train releases because they are introduced with their feature switch off by default. This allows developers and internal testers to evaluate the feature along the way. Once a feature is completed we turn the feature switch on for everyone. When the next train branch is cut the new feature will be available to the public at the end of the release cycle

Throughout a train release cycle we send out betas based on the train branch to the entire company and a curated set of external beta testers. We also build internal nightlies off of master for developers, designers, and product managers who are working directly on a feature in development. This means that bug fixes that go into the train branch are tested for up to a month before they ship and that features are tested for at least a month, if not longer, before they go out to the public.

Cadence is led by train conductors, one for each mobile platform (Android and iPhone). The conductors change each release. They have a handbook that defines a set of practices and responsibilities including ensuring consistency between the train branch and master and ensuring that beta and nightly builds are going smoothly. A conductor will remind (nag) engineers about bugs and work left to do on their branch, as well as coordinate with team leads to ensure work is on track. Attention to bugs can be a challenge as engineers have often moved on to new feature work in master. A conductor will work closely with QA to triage, assign, prioritize and punt issues. Finally, one of their most important duties is to drive the process itself — they own the release. We’ve learned something new each train and with that has come gains in documentation, tooling, and a shared appreciation for what it takes to make a release go smoothly.

What’s Been Working For Us

Before we adopted Cadence we were gradually losing our agility and our product development process was becoming more “waterfall”. We spent too much time planning a release up front which was eating into actual development time. A poorly defined feedback cycle, where feedback came late in a release, led to more planning and development which wasn’t captured in our initial estimates. We’d often encounter a rush at the end of a release to finish or cut scope. We’d file bugs to make up for work that didn’t happen but new feature work took priority over defects in following releases. This led to a snowball of a issues that would only get looked at toward the end of a release. The available time to work on bugs would conflict with the need to slip-in forgotten features, instrumentation and analytics, the ever popular “just one more thing”.

Since adopting Cadence the stress about meeting a specific date and the associated ill-effects of rushing toward that date have largely disappeared. Features ship when they are ready. We’ve refocused on craftsmanship, not just for the product itself but for our working habits, processes and testing practices.

Getting Started

One of the hardest things we encountered with Cadence was simply getting started. It’s all too easy to play the engineering, design, product, marketing finger pointing game in any start-up: marketing relies on engineering to ship marketable updates but pressure from marketing can lead to risks when shipping. We were a little nervous about being in a good place to start because the initial release required a one release cycle delay and we had features in development hoping to ship as soon as possible. But finally we dove in. And that is what I would like to encourage. Set a firm deadline to start, don’t let any outside stakeholders try to slip one last thing or release in before your first train arrives in the station. Just get started, reassure product and marketing that things will get moving, because once the trains start running you will quickly see the gains in quality and agility.

Summary

Mobile engineering at Strava has undergone a pretty drastic shift in team structure and release process. We’re no longer a monolithic team on each mobile platform all working on the same feature with a deadline looming over us. Instead we’re half a dozen tactical smaller teams working on different features but all working together under Cadence to improve Strava and keep shipping an amazing app.

There have been challenges along the way as we’ve adopted this new process. Ensuring that work is continually accepted and that our velocity is appropriate was difficult at first. As was simply ensuring that everyone had enough to do or knew what they should be working on. Prioritizing and ensuring that bugs got worked on was also an initial challenge. However as each train progresses and after each one ships we’ve taken a look back to see what we could improve. In each train we’ve addressed at least one problem we encountered on the previous train, such as ensuring there was a product manager or a team lead on top of making acceptance meetings happen, or adding new practices and tooling to ensure that bugs get fixed. We’re still working out the kinks, such as having to deal with review times in the Apple App Store causing one train to bleed into the other, only being able to have one release out to beta testers at a time (a limitation of TestFlight), and ensuring consistency between localizations in master and our train branches. The beauty of the process is that it is quick and encourages iterative improvements.

One thing that hasn’t changed is our desire to create an awesome product and always improve ourselves. That’s what Cadence has brought us.

Acknowledgments

We’d like to thank friend of Strava, Jim Puls, for his early guidance when we considered this drastic change to our software development process. His advice and time were invaluable, thanks Jim!

Originally published at labs.strava.com by Jeff Remer.