The process of releasing software to users at scale is one that can be particularly confusing for new software engineers, as most don’t have exposure to this part of the software development process until their first internship or job. I found this to be the case during my time at Zendesk as a Data Engineering intern in the PandAI team, especially the terms deploy and release. It’s easy to think these terms carry the same meaning and can be used interchangeably, when in fact they are vastly different concepts.
What’s the Difference?
Release: The process of making a software application, feature or bug fix available to users. This could include all users or a subset of users, like only 5% of the user base.
Deploy: The process of installing a software application, feature or bug fix in a specified environment. This could include production environments where the application is available to users or staging environments where the software can be tested. Deploying code doesn’t have to involve a release, for example deploying to staging. In fact, even deploying to a production environment doesn’t necessarily constitute a release.
Separating deploys and releases is advantageous (more on that later), meaning not every deploy should involve a release, and not every release should involve a deploy. To achieve this, a number of release methods can be employed which allow development teams to decouple deploys and releases. The PandAI team at Zendesk use a number of these release methods to mitigate risk and ensure fast feedback.
Types of Release Methods
There are two types of methods used to control software releases: environment based and application based. These methods encourage easy releases by minimizing risk and facilitating straightforward rollbacks.
Environment Based Releases
An environment based release method that is commonly used at Zendesk is the Canary release strategy. This method involves releasing new code to a small portion of production users, before releasing to the rest of the user base. For example, if a development team wanted to release a newly developed feature, they would first deploy to the Canary production environment. This server may only contain a subset of users (only 5% of the user base) but can be used to determine if the change is working as expected. The users contained in the Canary production environment can be chosen in a number of different ways, they could be employees, a random sample of users or users from a particular region.
The main advantage of using this method is the ability to test software changes with real users, and gather valuable metrics which can help development teams safely deploy to the rest of the user base. Automatic rollbacks are also an option with Canary releases, meaning if changes fail to meet certain technical or user metrics, it can be rolled back to a previous version.
At Zendesk an open source tool developed in-house called Samson is used to manage deployments. This is done using a web based GUI that allows teams to specify the version of software to deploy and which environment to deploy to. Samson enables Canary releases as development teams can deploy changes to the Canary environment only.
The Canary release strategy can be extended by implementing phased rollouts, where software is released to users progressively. For example, software is released first to a server containing a small subset of users, and then to a server containing more users and finally a server containing the rest of the user base. Phased rollouts are achieved at Zendesk by first deploying to the smaller EU user base before deploying to the much larger US user base. After each deploy, the production environments are monitored to ensure the software is working as intended. Monitoring includes technical metrics like performance and reliability and if relevant, user metrics pertaining to how users are responding to the changes.
Application Based Releases
Application based releases involve implementing controls in the application code to allow a development team to progressively expose new features to users. Feature toggles are a popular way to implement application based releases, and allow development teams to enable or disable features for certain users. This means teams can deploy to production without actually releasing any changes.
Zendesk engineering has developed a library called Arturo to implement feature sliders for Ruby and Scala applications. Sliders are controlled in an internal account management tool, which allows Zendesk employees to enable and disable features for all accounts or a selection of accounts. The PandAI team are currently developing a product called Content Cues, which is only available as part of an early access program (EAP). Feature sliders are an essential part of the development process as Content Cues can be enabled for accounts that are part of the EAP, while being hidden for other accounts.
Feature toggles are great for testing, as new features can be enabled on developer accounts while remaining disabled for the rest of the user base. Feature toggles are not only useful for releases, but can be used to disable non-essential features during times of high load, or when a feature is failing.
In addition, feature toggles are also an important part of continuous deployment, which is the practice of deploying code to production autonomously. Changes that pass through the deployment pipeline, will automatically be deployed. Not every code change that passes tests will be ready for user consumption, therefore feature toggles must be used to hide changes until they are ready.
Advantages of Deploy and Release Separation
The aforementioned release strategies play an important role in allowing deploys and releases to be separated. When deploys and releases are separated, it minimizes technical risk associated with code changes, while allowing for faster feedback from users.
Using the Canary release strategy and feature flags allow code changes to be tested with production traffic. These releases can be on a small scale and heavily monitored with teams ready to rollback changes if problems arise. Rollbacks are less complex as it is much easier to revert a version in a Canary environment than across all production environments.
Feature flags make it especially easy to gradually rollout features, which is important when making changes to the user interface. Rolling back changes using feature flags doesn’t require deploying any changes as problematic features can just be disabled. These techniques allow development teams to release more frequently, leading to safer rollbacks and most importantly tighter feedback loops, allowing businesses to shape future decisions on customer feedback, rather than assumptions.
Deploying and releasing code are vastly different concepts, but for new software engineers this may not be so obvious. Learning about deploys and releases and how to separate them, will ultimately help teams improve their processes and in turn minimize risk, reduce turnaround time for user feedback and improve the overall software development experience.