Candy Crush Soda Saga delivery pipeline
We release a new version of Candy Crush Soda Saga every other week on iOS, Android, and Facebook. It’s a complicated picture — two code bases (C++ for mobile and AS3 for Facebook), as well as dependencies on internal libraries and frameworks that are updated frequently. It’s the kind of juggling act that we at King are constantly working to simplify and improve. Here’s how we’re currently doing it.
Version source control
The Soda team uses the same Mercurial repository for both code bases, and while the game logic is completely separate, the two applications share graphical and UI assets. We use a mixture of trunk-based development, feature branches and feature toggles. Some of our features can take a couple of sprints to be completed, in which case they’re implemented in a feature branch. Some features are developed in default but not enabled, most likely strong candidates for A/B testing. Finally smaller increments of code that can be completed within a day or so are implemented in default right away. As we gear up for a release, we create a Release Candidate (RC) branch. The RC branches are our only official branches.
Jenkins is our continuous integration (CI) tool. With every change to the repository, both code bases are built and unit tests are run. Every successful mobile CI build results in runnable apps, and we run automated system tests on those apps several times a day. We also have a lightweight server layer written in Java, packaged with the AS3 flash app. The server code and flash app are deployed on demand to our test servers. The deploys can be easily done by any member of the team via Jenkins. We also use Jenkins for automating tasks such as creating RC branches or importing data that’s generated externally into our repository.
As mentioned above, we first create a release candidate — at this point, all new features have already been tested by the group that implemented them.
The automated Jenkins tests are then run on our release candidate. Next, the app is tested manually by the whole team. Everybody participates in playing the new levels and checking that the game looks and feels OK. Finally, the build is sent to specialised device-testing teams, who load the app manually and test it on all kinds of devices. The whole process takes a couple of days, and we allow for time to fix any issues if they’re found. Three days after creating the release candidate we deploy the server, Flash app, and Android app, and we submit the Amazon and Apple apps.
We end the entire cycle by holding a release retrospective, where we create action points for the items we’d like to improve during our next release.
Who does all the work?
We all do! Since our release process is not yet 100% automated, it requires a technical person to be involved, so the responsibility of pressing the buttons is rotated among the developers, with two assigned per release. To ensure a continuous transfer of knowledge, and to avoid reinventing the wheel with each release, the rotation requires that one of the assignees was also assigned to the previous release, and the other assignee will be on the following release.
This sounds good, but is this really the way it works? As with all projects and processes this is a rosy picture of the perfect release workflow, when everything goes smoothly. It doesn’t always go smoothly, of course. Sometimes, Jenkins breaks, and the builds don’t go through; sometimes we’re not so great at testing all of our features before we create the release candidate, and spend the next day fixing unexpected bugs; we test too many things manually; every once in a while we even get a live bug that forces us to patch.
That’s the messy reality, but we’re always improving our release process. We focus on shortening the time between creating the Release Candidate and the actual release, while striking a balance between keeping up with the latest releases of the internal dependencies and ensuring that the app is always working. With each release, repetitive manual tasks are automated, and most importantly, we stay aware of our shortcomings and work actively to address them! So our rosy picture isn’t really too far from our everyday reality.