The DevOps Rainbow: Blue, Green, Yellow and Black
In many projects, in order to deploy changes to production environments, it’s necessary to stop every server, install the new software, and restart everything again. This, obviously, cannot be done at any time, as we’re taking the service down and we could make the customers angry. So normally these deployments are made at night, causing a disruption in the personal life of the poor Operations guy who is responsible of everything to go OK.
So, ironically, to have customers pleased we’re messing up with our employees personal time, forcing them to work at an unreasonable schedule.
As happiness is one of the most important things that DevOps is trying to bring to everyone, not only our customers but also our employees, I’m writing this post, to introduce this kind of “DevOps rainbow” with three alternatives to this tedious night deploying.
Blue & Green
The BlueGreen Deployment is an amazing idea that is as simple as powerful. I heard about it at Martin Fowlers’s website, and to bring it to life we only need two servers and a load balancer.
The idea behind this is to deploy our changes into the Green servers, to test and verify them. While everyone is doing this (at office time!!) the service is brought to the customers by the Blue server. When the DevOps team decides the Green server has everything OK, they change the load balancer to point to the Green server, causing the customers to see the new changes without any blackout. If something goes wrong, the load balancer can point to the Blue server again to mitigate the problem.
One of the inconvenients of the BlueGreen Deployment is that sometimes it’s difficult to automate the change of the Blue server to the Green one, as the validation in the new environment might require manual testing.
In order to solve this, there’s another technique called Canary Releasing. This technique takes its name from the ancient mining industry, which used canaries to save miners’ lives when a gas leak was originated, as the animals couldn’t resist as much as humans, and were used as a signal of danger, which indicated the miners that they needed to leave as soon as possible.
In software development, the canaries are a small selection of people that are the only capable of seeing the changes deployed into production, by clustering all the machines into specific target groups.
For example, it’s known that Facebook tends to first activate changes to the machines that serve their own employees, and after everything is OK, they add some little percentage of people to the cluster, increasing continuously the amount of people that sees the changes.
With this technique, we do not depend on any manual step at all (the clustering can be done with a load balancer like Nginx), and we still have control of the new changes made into production environments, because we have our canaries that alert us if something goes wrong.
Both BlueGreen and Yellow are techniques based on infrastructure, balancing the new features between servers or users, respectively. But what about controlling the production deployments within the code itself?
This technique is called Dark launching, and consists of developing modularized features that can be switched on and off, in order to control what the user sees.
This is usually achieved by using conditionals and toggle flags that are modifiable in runtime, and it has also been used in companies such as Facebook.
With this technique, for example, you can implement a critical component in your website, retrieve metrics, calculate the correct infrastructure needed, etc. without the customer even noticing it, because he/she cannot see it as it’s disabled.
When the component is finished, you can switch on its behavior. The customer will see the changes, but the component has been tested already quite time ago and it will have no breakdowns.
There are many techniques to avoid nightly and unpleasant deployments into production:
- BlueGreen Deployment, changing to the serve with the new release with no blackouts.
- Canary Releasing, clustering which users can see the new features.
- Dark Launching, with feature flags to enable or disable behaviors.
So, if your company is forcing someone to keep working at night, let them know there are alternatives out there, not only for improving the employees lives, but also the customers experience.