Continuous Integration Versus Continuous Delivery Versus Continuous Deployment
Here’s What They Mean (and How To Use Them)
Editor’s Note: Brent Laster is a global trainer, author, and speaker on open source technologies as well as an R&D director at SAS. He’s been involved in the software industry for more than 25 years, holding various technical and management positions.
In this excerpt from his report, Brent Laster explains the practical meaning of the terms Continuous Integration, Continuous Delivery, and Continuous Deployment in product development and release cycles, with an emphasis on and the actual practices and tooling associated with them. He places these terms in the larger context of DevOps, and explains how they combine to form a highly productive set of best practices and disciplines for development, testing, and deployment teams.
Continuous integration, continuous delivery, and continuous deployment — if you do any work in the area of software delivery, it’s impossible not to encounter these terms on a regular basis. But what does each of these processes do for your product development and release cycles? I’ll explain what they really boil down to, what practices are associated with them, and what kinds of tooling are available to implement them in a manner that clearly depicts what they offer, how they differ, and how they work, both separately and together, to help release software to customers frequently, reliably, and with high quality.
Defining “Continuous”
Continuous here implies automation — automation in transforming source code into deliverables, in testing and validation, and even in installing and configuring software. It doesn’t imply that processes are continuously executing; rather, that when a new change is introduced into the system it can be automatically processed quickly and thus enables more frequent builds and faster testing, packaging, and releases than without continuous practices. Further, it enables quick identification and reporting of failures so that they can be fixed swiftly. This is commonly known as fail-fast.
Continuous here also implies that a change can proceed, without human intervention, through the stages of builds, testing, packaging, and so on. Combined, these stages form a “pipeline” of processes and applications to take source code changes and turn them into a releasable product. This pipeline can go by various names, including continuous delivery pipeline, deployment pipeline, or release pipeline. Although hand-offs between stages are intended to be automatic as the sequence progresses, there are two cases in which human intervention can be required:
- When a change breaks and should be fixed before proceeding
- When you want a human to validate something before it proceeds (e.g., user acceptance testing or choosing to manually approve deploying to customers)
With this background in mind, let’s look closer at what each of these three terms encompasses.
Continuous Integration
In the continuous integration (CI) phase, individual changes from a developer are merged and tested. The goal of CI is to quickly validate individual code changes that have been submitted for inclusion in the code base. An intended consequence of this is quickly identifying any problems in the code and automatically notifying the developer of problems so that the code base is not broken any longer than necessary. The mechanism for CI detects that code changes have been made and then runs targeted testing to prove that the code changes work in isolation (function inputs produce the desired outputs, bad inputs are flagged appropriately, and so on).
Unit Tests
These targeted tests are called unit tests, or commit tests, and developers are responsible for creating them to exercise their code. In fact, one model (known as test-driven development [TDD]) requires unit tests to be designed first — as a basis for clearly identifying what the code should do.
In the standard CI workflow, a developer creates or updates source in their local working environment and uses the unit tests to ensure that the newly developed function or method works. Because such changes can be frequent and numerous, and because they are at the beginning of the pipeline, the unit tests must be fast to execute, must not depend on (or make calls to) other code that isn’t directly accessible, and should not depend on external data sources or other modules. If such a dependency is required for the code to run, those resources can be mocked by using a stub that looks like the resource and can return values, but doesn’t actually implement any functionality.
Typically, these tests take the form of asserting that a given set of inputs to a function or method produce a given set of outputs. They generally test to ensure that error conditions are properly flagged and handled. Various unit testing frameworks, such as JUnit for Java development, are available to assist with this.
CI Tools
The developer can then push their code and unit tests into the source code repository. A variety of different source code management systems, such as Git, are available for use today. If the code is pushed and does not merge cleanly with other users’ changes, the developer must manually fix the conflicts and try again.
After the code is merged in, a CI tool such as Jenkins can detect (or be notified) that a change has been made in the source code repository. It will then automatically grab the latest set of code from the repository and attempt to build it and execute the unit tests. Jenkins can detect changes by regularly polling the source control system to check for changes or by having a scheduled, periodic build of whatever is current. Alternatively, most source control systems can have a hook set up to send a notification to tools like Jenkins that a change has occurred.
Prechecks
A variation on this simple workflow involves adding in prechecks on the code before it makes it all the way into the source code repository. For example, a tool called Gerrit can intercept pushes to a remote Git repository and wait for a person to sign off on a review of the code before allowing it to go into the repository. It can also initiate early Jenkins builds of the code at that point as another pass/fail check.
GitHub, a public hosting site for Git source repositories, offers a further variation. Users wanting to contribute changes to a source repository on GitHub can submit a pull request to ask the owner of a repository to review and then merge in code changes from the user’s copy of the project.
These variations provide additional checking and validation by multiple people earlier in the process — at the expense of reduced frequency of integration.
Continuous Delivery
In the continuous delivery cycle, the isolated changes merged and validated during CI can be combined with the remaining product code. Then, the set is put through progressively more encompassing types of testing. The goal of continuous delivery is not necessarily to deploy the end result, but to prove that the end result is deployable. The mechanism of continuous delivery is the continuous delivery pipeline (CDP).
Artifacts
The CDP is made up of stages connected in a sequence. Each stage in turn can be made up of smaller “jobs” with specialized tasks. For example, what’s typically referred to as the commit stage usually consists of jobs to compile the code, run the unit tests, do integrated testing, gather metrics, and publish artifacts. This then could have an automated handoff to an acceptance stage in which artifacts are retrieved and functional testing and verification are done.
An artifact refers to a file that is either a deliverable (something directly used by the final product) or included in a deliverable. For example, you might have an executable file created by compiling source that links in several other libraries. Or you might have a compressed file such as a .war or .zip file that contains another set of files within it.
Versioning
Regardless of the type of artifact, an important dimension across the pipeline is the artifact versioning. Although deployment technology has made versioning less of a concern to users (think app updates on your phone), it is critical to work with the desired versions of artifacts through the pipeline. This is usually managed via an artifact repository tool such as Artifactory. These tools work like a source management system for artifacts, allowing multiple versions to be stored and retrieved to and from different repositories. In some cases, these repositories can serve as a dev/test/prod (development-quality/testing-quality/production-quality) organizational structure to separate multiple versions of releases in progress.
Testing
If incorrect versions of artifacts are pulled in or functionality is broken, the automated testing in the pipeline will present some symptom of this. The goal of the various testing phases in the pipeline is to instill more and more confidence as we progress toward the final product. There are several types of testing, including:
Integration testing
To validate that groups of components and/or services all work together
Functional testing
To validate the end result of executing functions in the product are as expected.
Acceptance testing
To measure some characteristic of the system against acceptable criteria. Examples include performance, scalability, and capacity.
Coding Metrics and Analysis
The amount of code covered by testing is one example of a metric that can be used for ensuring code quality in the stack. This metric is called code-coverage and can be measured by tools (such as JaCoCo for Java code). Many other types of metrics exist, such as counting lines of code, measuring complexity, and comparing coding structures against known patterns. Tools such as SonarQube can run these sorts of checks, measure results against desired thresholds, and provide integrated reporting on the results. If automatic testing or metrics gathering finds a problem, the code’s progress through the pipeline is stopped. In this way, you can ensure that only the correct version of code that passes all of your testing and other metrics makes it to the end of the continuous delivery pipeline in a deployable state.
Infrastructure-as-Code
One other recommended practice that DevOps culture has introduced to continuous delivery is the infrastructure-as-code ideal. In this approach, configuration and setup of the tools used in the pipeline are automated and described in files that can be stored in source control. A change to any of these files drives a reconfiguration or update of the tools used in the pipeline, and triggers a new run of the pipeline, just as a source code change for the product code would.
Continuous Deployment
Continuous deployment is a process that can be added on at the end of the continuous delivery pipeline. The idea is that since the continuous delivery pipeline has proven the latest changes to be deployable, we should automate the deployment process and go ahead and deploy the result. Deployment can mean a lot of things, but for simplicity can just think of it as making the release available for the customer to use. That might mean that it’s running in the cloud, made downloadable, or updated in some other known location.
Manual Checks
If a deployment is not automated, there are limitations on what can be done here. Even if it is automated, an organization might not choose to always deploy every release, so a manual check might be added in. More commonly, though, such a check would take the form of a user acceptance test (UAT) earlier in the pipeline.
Staged Deployments
Even if you want to automatically deploy the latest version from the pipeline, you can still do it in a staged way. For example, you can deploy it to a location that is accessible only by a subset of users who might want to try out the new release, or you can redirect some users to the new release transparently. Then, if any problems surface during an evaluation period, not all users are affected, and the subset can be directed back to the previous release. If all is well, the remaining users can then be redirected to the new release. This sort of deployment is called a canary deployment.
Blue/Green Deployments
A broader form of the canary deployment is called blue/green deployment. In this case, two deployment environments are always available: one labeled blue and one labeled green. At any given time, one of these is production and one is in development or testing (nonproduction). When a new release is deployed, it is deployed to the nonproduction environment for evaluation. When it is deemed production ready, the nonproduction environment is toggled to be production (that is, customer interaction is redirected to it). And the previous production environment is toggled to be nonproduction, ready for the next candidate release.
Conclusion
As you can see, continuous integration, continuous delivery, and continuous deployment form a powerful set of disciplines, best practices, and technologies for transforming changes from source to production quickly and reliably. The end result is high quality and easily reproducible. This process not only benefits the customer, but also greatly simplifies the worlds of the development, testing, and deployment teams. It lets them work together in a common environment from start to finish, helping to realize one of the core goals of the DevOps movement.
Learn faster. Dig deeper. See farther.
Join the O’Reilly online learning platform. Get a free trial today and find answers on the fly, or master something new and useful.
Brent Laster is a global trainer, author, and speaker on open source technologies as well as an R&D director at SAS. He’s been involved in the software industry for more than 25 years, holding various technical and management positions.
