Define a valid Rollback strategy
How do you deal with broken builds? Applying methods like Continuous Integration or Continuous Delivery usually make the lower development environments (usually tagged “integration” and “UAT”) unstable. As several teams merge their changes to the software, the end-to-end tests might fail because of unforeseen circumstances: datasets may not be valid for the certain version, the models linked to the database could not reflect the state of a table, or miscommunication between members could generate software that just plainly refuses to start.
It just happens. We have to keep in mind that detecting errors as soon as possible means that we have the time to fix and polish the product. We actually want those environments to be broken as soon as possible.
However, that could also imply downtime for the working teams. Breaking a development environment is no big deal, as long as we have a mechanism in place to reinstall a working version of our programs as soon as possible. We call that “a rollback”. You might have heard the term being applied to databases. After executing a script to change the data or structure, if something goes wrong, this operation returns your information to a working state. The concept is just the same, in a different context.
There are several ways to deal with rollbacks. The most common involves a blue/green strategy: you install your application on a dynamic path, and have an indirect access that can be changed to the new destination. If something goes wrong, your rollback becomes just as easy as changing this symbolic link back.
A repository with tagged artifacts works just as well. Each change in your source code generates a new version that is stored somewhere in your infrastructure. Then, you provide some tool that can request a specific version, so it overwrites the one currently installed on your server.
Talk with your teams to find the most suitable strategy for your business. A quick way to solve issues means that you are making sure that downtime is as short as it can be, so that your talent can be focused on what really matters: your product.
Originally published at Álvaro G. Cachón.