Deploying to Production

This article is aimed to help you better understand how to deal with deployments in your development workflow and provide some useful practices for deployments. Sometimes a bad production deployment can ruin all the effort you invested in a development process. Having a solid deployment workflow can become one of the greatest advantages of your organization.

The Workflow

Deployments should be treated as part of the development workflow, not as a separate process. If you are developing an application, your workflow usually includes at least three environments: Development, Staging and Production. General workflow might look like this:

Developers fix bugs and create features in the separate branches. All stable and verified code should be merged to the stable development branch (e.g. master for Git, trunk for Subversion or default for Mercurial).
Once features are completed, they are merged into the staging branch and deployed to the Staging environment for quality assurance.
After testing is complete, feature branches are merged into the stable development branch.
The stable development branch is merged into production and then deployed to the Production environment.

Development Environment

Development environment is a set of components which required for the application. Environment could be local (installed on the local machine) or remote (e.g. set of virtual machines with installed application components).

Desirable to set up automatic deployments for development environments on every commit or push. However, testing every change could take a lot of time — because the change must be committed, pushed, deployed, and only after that it can be verified.

Every change should be verified both locally (e.g. using unit-tests) and remote (e.g. using required scope of functional and integration tests), then, once it’s more or less stable, it can be pushed to a Staging environment for proper quality assurance testing.

Staging Environment

Once the features are implemented and considered fairly stable, they get merged into the staging branch and then automatically deployed to the Staging environment. The staging environment is usually a full-copy sandbox, so it is as similar to the production as possible.

This is when quality assurance magic begins: full scope of automated functional tests runs on the staging servers and verify that the application works properly. Limited numbers of checks could be run manually (e.g. if your would like to collect some metrics, but overall idea is a focus on the automation).

There can be situations when you need to revert your remote server to a previous state/commit. Your deployment tools should have a really simple procedure of Rollback. One way is to use “Rollback to” button near on your latest deployment in the list of deployments.

Also it is very useful to have a separate branch called staging for your staging environment. It will allow developers to deploy multiple branches to the same server simultaneously, simply by merging everything that needs to be deployed to the staging branch. This also helps testers understand what sort of code changes is on staging servers at the moment, just by looking at branch’s merges history.

Production Environment

Once the feature is ready (implemented and tested), it can be deployed to production. If the feature should be merged into a stable development branch first.

The next step is to make a diff between the production and development branches to take a quick look at the code that will be deployed to production. This gives you one last chance to spot something that’s not ready or not intended for production (e.g. some valuable parameters are missing). General verification should be auto-complete. But it’s essential that the final verification to be performed by a responsible human being. Why? Because relying only on automation in case of deployments for production can lead to unexpected results.

You should refresh the staging environment during the maintenance window, not before. Full-copy sandboxes can take some time to create or refresh, so it is important to pad your maintenance window to account for this.

Also deploying major releases to production should at a scheduled time. And these action typically happen during a maintenance window when users is not working in your application (off-peak hours) and your team is aware of deployment (someone needs to be around after the deployment for at least a few hours to monitor the application and make sure the deployment went fine).

Deployment tool should send an email to all team members with a summary of changes after each deployment. This helps team members to understand what exactly went live and what features is available for customers now.

Rolling Back

Sometimes deployments don’t go as planned and things break. In that case you have the possibility to rollback. Sometimes a rollback brings more chaos than the issue it was trying to fix. Before performing a rollback, answer the following questions:

Did it break because of the code that I deployed? You can only rollback files that you deployed, so if the source of the issues is something else (e.g. infrastructure issue) a rollback won’t help.
Is it possible to rollback this release? Not all releases can be rolled back. Sometimes a release introduces a new database structure that is incompatible with the previous release.

If the answer to both questions is “yes”, you can rollback safely. After rollback is done, make sure to fix the bug that you discovered and commit it to the bug-fix branch.

Permissions

Every developer should have access to the Staging environment. To be able to both monitor working application and track code changes in staging branch.

Your Production environment should only be accessible to a limited number of experienced engineers. These guys should always be prepared to fix the servers immediately after a deployment went rogue. Instead, give your developers Monitoring tools such as Zabbix, Sumologic, Data Dog, etc. — so they could monitor their applications significantly and without special permissions to production infrastructure.

Conclusion

Creating a deployment flow from a scratch isn’t a simple task. Usually it takes time to get a working solution, sometimes even through broken production servers. But if you’ll focus on result and ready for iteration improvement— someday you’ll notice that you have a incredibly smooth production deployments.

Thank you!