Release Train
A Continuous Deployment-ish process
In my almost 10 years as a developer, I have seen quite a lot of ways to manage changes in product and organize the delivery of a new feature to users. I have seen a lot of teams that spend months without a new deploy, whole squads that have to wait for midnight on weekends to minimize impact and rollouts that last weeks before delivering something to 100% of the users.
In all those cases, what I saw was a huge lack of confidence in the quality and stability of the product that was being launched or in the process of delivering it, but I don’t mean it is not justified. Every deployment in this scenario carries a lot of changes, months of continuous development, executed by a team that has gone through this process just a couple of times or, in case of mobile apps, is not possible to rollback. With this cadence, probably most of the work is manually executed and brings the system down for at least a few minutes.
Quoting David Dryjanski, Product Manager at Lyft, from the event Fast Data Driven Growth on Native Mobile in 2015:
The less you release, the less confident you are as a company that whatever you’re pushing out is not going to crash and you’re not going to be able to fix a major bug that affects all your users.
Along this line of thought and accompanying the agile mindset, a lot of buzz words were created, but I would like to focus on one of them: Continuous Deployment.
In a few words, it can be defined as the greatest dream and, at the same time, the worst nightmare of most DevOps professionals. It establishes that every time your code is considered correct and stable by an automated test suite, it is automatically deployed to production. It depends on a really REALLY mature development team, codebase and process to give the confidence needed to implement this methodology and, sometimes, the technology itself is an impediment. For mobile apps, for example, there is a limitation imposed or strongly recommended by the stores themselves.
What we want to demonstrate is how Release Train can help you through the baby steps needed to start continuously deploying your apps to production or, at least, as close as mobile stores will allow you to.
But first, let’s make a deal
For this article’s purpose and to make this approach as agnostic to your company’s agile methodology of choice as possible, let’s only talk about the actual development process. What I mean is: from the moment the development team starts developing one or more features until it’s finally delivered to the users. You may have only one or many squads working on the product, it doesn’t really matter.
Why did I want to emphasize this? Well, because SAFe (Scalable Agile Framework) has an Agile Release Train definition of its own that establishes how the whole company, team, roles and product conception should be structured and I would like to ignore most of these specific characteristics to make it a bit less complex (in my point of view) and easier to relate.
For Christ sake, get to the point, will you?
Most apps I’ve worked on plan their releases based on features. Which means that only after a feature, or a couple of them, is completed, we start to plan its delivery. In this scenario, questions like these are pretty common:
- Should we wait for that other feature that is almost done?
- Is there any code that will be a problem with this release?
- Who will be on call this weekend for deployment?
It may sound simple enough to answer these questions, but in a big and complex product with multiple squads working on the codebase and different agendas, it is not so simple. It costs a lot of time and effort to bring all this information together and we tend to try to put together as many features as we can, because you never know if the next deployment will be in a week or six months from now.
Release Train’s core characteristics, that we’ll explain below, will help answering these questions and, hopefully, make the releases more frequent, more reliable and a lot less stressful.
Fixed deployment frequency
This is the core concept of Release Train, it adds predictability to your process and lets everyone know what to expect. You should really think of how a real train works: it has a schedule, it won’t depart only if it’s broken and it does not wait nor stops for anyone (maybe if someone lays down on the rails…maybe). So, if:
- There are any validations that need to be done after packaging and before delivering, people can organize their schedules to be prepared for it
- Your feature misses this train, there will be another in no time and you know exactly when it will be.
- There is something on the main line that is not ready for production yet, you will have the time to throw yourself on the rails and shout out “STOP!” to the train driver
Small frequent updates
As Jim Rose, CircleCI CEO, pointed in his article:
Deploy frequency indicates a company’s actual speed… With each “pulse”, or deployment, an organization is delivering value, discovering customer needs and fixing problems.
So, if you want your company to grow at a fast pace, your deployment frequency needs to reflect that. It has to accommodate all the small quick changes that you want to get out there as soon as possible instead of spending months planning and theorizing the results.
It is expected that your company, if it never had a MVP mindset to start with, will think it’s weird to go through the whole deployment process just to release a not so critical bug fix. However, try as hard as you can not to delay the train because of this, it can mess up all the following releases on schedule. Furthermore, it will be beneficial to the company if it can actually change its mindset to embrace small and incremental changes, keeping the product gradually growing and constantly fixed.
But, let’s be realistic and empathetic, if you need to manually run every step of every single deployment and summarize the features that are being released from all squads, you would think twice before increasing frequency. It takes us to our next section.
Make a script do all the work for you
Imagine today is release day. There is no automation script to be seen and you have to execute everything manually, all the 15 deployment’s steps. Maybe, you forgot the 10th and now your build won’t work and you have no idea why. This is entirely possible and more likely to happen the more frequent the releases are and the more complex your product gets. We are just humans after all, prone to mistakes and memory problems. Besides, it takes a lot of time and effort of someone or some people to execute every single step. They can’t just start the process and continue their work, they have to be focused and execute things to get the release done.
With scripts, you can do all this more efficiently and reliably, also creating a documentation of how your deployment works. It helps when you need to share this knowledge with new members, allowing more people to participate in the process, and is less likely to get outdated than a real documentation.
In one of the products I’ve worked on, we’ve created scripts and workflows that:
- Format all the information of completed tasks from six squads based on the commits
- Increment the system version based on that tasks
- Create a tag
- Deploy it
- Notify everyone involved in that release through Slack.
All this with one request to CircleCI and, now, the deployment process takes no more than 12 minutes (mostly because of build time) to release 3 apps at the same time by one person that can continue coding while monitoring the log. We have reduced the chance of human error and the effort needed for every iteration.
Still in the subject about human error, we have the last item of our list
Reliable automated test suite
With multiple squads working and merging their code, how do you know that everything that is in the main line is stable enough for production? How do you know that the code that you have developed will not impact the feature being developed by another team?
You could spend days doing regression tests or hiring an entire team just to manually test your system, but it is not scalable. This solution will not allow your product to grow and evolve in the rhythm that the world demands.
That’s when you need a reliable automated test suite the most. It will allow your team to confidently merge their code into the main line knowing there is something to notify them if they break something or interrupt the release before it gets to the users (stop the train).
Sounds simple enough, but… How often should I deploy?
It doesn’t matter if you try to start with a deployment every hour, every few weeks or every couple of months, it’s important to start somewhere, but it is equally important to try to improve. So, every time the team feels comfortable with the cadence and it is not going to greatly disrupt all the process, try to shorten the time.
Of course, as I said before, there might be some limitations that you have no way to avoid. The easiest to notice are Android and iOS apps. Apple, with its review process that can take days to approve one deploy, places a very big wall with laser guns between us and our dream of delivering anything more frequent than once a week, and Android has a very strong recommendation that can be seen in its API documentation:
Do not publish alpha or beta updates more frequently than once a day. (Production apps should be updated even less frequently than that.)
I have worked on mobile apps that were deployed every one or two weeks. Both worked fine and accommodated very well the number of squads and the rhythm of feature development, but you have to adjust this frequency to what makes sense to your product and team.
Great! We’ve started it! Now… Am I going too slow?
As David Dryjanski said in 2015
One of the biggest indicators is that a release is like a very special occasion.
So, if you need a two hour meeting, talk to all squad leaders, manually run a list of 15 commands and everyone is releasing their holded breath after every deployment, you might be going too slow and need to review the deployment process.
Uh…What were we talking about again?
So, basically, Release Train adds predictability and reliability to your process, making it less painful and more natural through its four main characteristics:
- Fixed deployment frequency
- Small frequent updates
- Scripts doing all the work for you
- Reliable automated test suite
Now, let’s go back to the question we had before and try answering them:
Should we wait for that other feature that is almost done?
Nope, the train stops for no one, but, don’t worry, it will soon be released, because there will be another train in X days.
Is there any code that will be a problem with this release?
If the release finishes successfully, there shouldn’t be, we have a reliable test suite that guarantee this, after all.
Who will be on call this weekend for deployment?
No one, in time, this process will be so painless and more reliable due to the automations, that problems with the deployment are not likely to happen.
Of course, all this still requires some maturity from your development team and process, but it will make it possible for them to gradually become even more mature and eventually adopt the Continuous Deployment process and deploy as often as technology allows them to.
The idea is to turn the deployment process into something less painful and dangerous, integrating it to everyone’s routine and automating most of the process, if not all of it. However it takes some time to get used to it and get all the scripts working, so do not be discouraged if it doesn’t go smoothly in the beginning and always have a fallback. Keep on trying, always improving and delivering the best experience to your users. =D
