DevOps Handbook Series Part 3: Exploit Continuous Delivery

6 min readJan 17, 2018

As ever, don’t forget to read up on the previous installments to get a full, clear picture of The DevOps Handbook.

The DevOps Handbook Summary Series (so far):

This week, we examine how to execute the technical practices that create The First Way: Flow. Parts 3–5 of The Handbook explore in greater detail the realization of The Three Ways and its core principles. Without which, your DevOps transformation cannot succeed to its full potential.

Establishing Continuous Delivery to Enable Flow

To enhance the flow of work in the technology value stream between Development and Operations, it is essential to implement Continuous Delivery. This set of technical practices serve to reduce the risks that occur in production when deploying code and releasing changes.

The focus of the first two chapters of Part 3 in The DevOps Handbook is about laying the foundation for your deployment pipeline and facilitating the continuous integration of quick and reliable automated tests. These factors will help ensure that we are releasing code daily on a routine basis in a low-risk, deployable state which optimizes our continuous delivery environment architecture.

Using production-like environments throughout the value stream ensures that we pro-actively build applications with fault-finding in mind. Rather than discovering issues during production deployment as the traditional method dictates.

This Continuous Delivery strategy calls for teams to:

“Enable on Demand Creation of Dev, Test, and Production Environments.” The most significant factor behind issues that crop up during software releases tends to be because only in deployment do we introduce apps to a production-like environment. Instead, enable Devs to create and run on demand and self-serviced test environments at their workstation. Achieve this by defining and creating an automated common build mechanism for all environments.
“Create Our Single Repository of Truth for the Entire System.” Ensure that all elements of the software system are accessible to the entire DevOps team within a shared version control repository. Documenting any changes and revisions to both code and environments within this ‘Single Repository of Truth’ allows everyone to compare, combine, and revert to previous revisions as needed. Optimizing version control is “the highest predictor of both IT performance and organizational performance.”
“Make Infrastructure Easier to Rebuild Than to Repair.” The capability to restore and recreate our apps and environments on demand ensures that we can rebuild, rather than repair, when we encounter issues. These repeatable environment creation systems enable us to increase server capacity and avoid disaster quickly.
“Modify Our Definition of Development ‘Done’ to Include Running in Production-Like Environments.” Modern software development methodologies advocate short and iterative development intervals to fuel continuous integration. As opposed to singular large software releases which inevitably create more issues. The Scrum sprint is a great example of this short time frame which increases productivity and aims for working shippable code at the end of each interval.

Approve Automated Testing

Having implemented practices which enable the successful integration of version controlled production-like environments throughout the value system, it is important to merge automated testing too.
Continuous integration practices automatically test code against any number of automated tests. This means that instead of discovering problems months after the code is developed archaeologically, teams can find and fix mistakes at an amazing pace.

Aim to:

“Continuously Build, Test, and Integrate Our Code and Environments.” Make automated tests a daily part of your team’s work. Achieve this by creating a deployment pipeline that runs in dedicated environments and performs code integration whenever a new change is submitted to version control. This continuous integration begins with executing the commit stage — code build/packaging — before implementing automated unity tests and other validation including analysis of static code, duplication/test coverage analysis, and style checking. The result: a green build (where everything in version control is in a deployable state).

The Deployment Pipeline: The DevOps Handbook (1st ed. 2016) Kim et al.

“Build a Fast and Reliable Automated Validation Test Suite.” If tests are only run on an intermittent basis, we expose our value system to slow and periodic feedback. Alternatively, run fast automated tests in a production-like environment everytime a new change is checked into the version control. In this manner, work batches remain small, code is in a consistently deployable state, and errors are located and fixed quickly before gaining a chance to escalate. Automated tests fall into the following types:
Unit tests: Which test the designed operation of individual components or functions within the application
Acceptance tests: To test the overall design and functionality of the application
Integration tests: These test that the build (which should pass unit and acceptance tests) interacts appropriately with other services and products
“Catch Errors as Early in Our Automated Testing as Possible.” Running faster-automated tests (unit tests) before slower ones (acceptance/integration tests) enables early error detection. Martin Fowler’s “ideal testing pyramid” optimizes the flow of testing by recommending the creation of an automated unit test to locate any error discovered by an acceptance/integration test. Thus, improving the continuous delivery feedback loop.

Ideal Vs. Non Testing Pyramids: The DevOps Handbook (1st ed. 2016) Kim et al.

“Ensure Tests Run Quickly (in Parallel, If Necessary).” Design tests to run in parallel. This will maximize speed. All testers (including devs) should be able to test the latest build in version control. Rather than having to wait for a build to be flagged.

Running Automated and Manual Tests in Parallel: The DevOps Handbook (1st Ed. 2016) Kim et al.

“Write Our Automated Tests Before We Write the Code (‘Test-Driven Development’).” To guarantee the use of automated testing, DevOps teams should integrate test writing into their daily work. This method works best by writing tests with the expectation that they will fail. Developers should then write the code in a way that passes the tests. Follow this method:
Write a test that will fail the functionality you are creating ➨Check in
Write the code to pass the test ➨ Check in
Rework old and new code to improve structure ➨ Ensure tests pass ➨ Check in
“Automate as Many of Our Manual Tests as Possible.” Automating testing ensures that our developers are put to more high-quality use on activities that cannot be automated. This strategy only works if tests are reliable. Undesirable outcomes such as false positives are a waste of valuable time and increase the stress load on tests and developers.
“Integrate Performance Testing into Our Test Suite.” Rather than waiting to discover performance issues at an application’s deployment stage, teams should endeavor to write and run performance tests throughout the delivery pipeline. Building the performance test environment early in the project is the key to avoiding its construction becoming more complex than the application’s actual production environment.
“Pull Our Andon Cord When the Deployment Pipeline Breaks.” Create a virtual Andon Cord which implements a “no new work” rule. This will make rollbacks easier and keep maintaining a ‘green build’ in the deployment pipeline possible. It also encourages the active team goal of consistently restoring to a green state. Such virtual Andon Cords should notify every on-call tester and developer, inciting a swarm to fix the problem. This is especially helpful during the later stages of production as opposed to just stopping new work. Visible/audio indicators are extremely helpful for when build/automated tests fail
“Why We Need to Pull the Andon Cord.” Failing to pull the Andon Cord will only increase the potential risk of finding problems in production. At this point, we are also subject to all the undesirable issues that come with “big bang integrations” such as feature delivery grinds, major firefighting, and archaeology — working on issues from a long time ago.

Check out the next installment now, DevOps Handbook Series Part 3: Practice Continuous Integration