CONTINUOUS DELIVERY — BOOK REVIEW

The book consists of three parts:

Part I — Foundations
Describes the prerequisites for understanding the deployment pipeline.

• Chapter 1 — The Problem of Delivering Software
• Chapter 2 — Configuration Management
• Chapter 3 — Continuous Integration
• Chapter 4 — Implementing a Testing Strategy

Part II — The Deployment Pipeline
How to implement the various stages in the pipeline.

• Chapter 5 — Anatomy of the Deployment Pipeline
• Chapter 6 — Build and Deployment Scripting
• Chapter 7 — The Commit Stage
• Chapter 8 — Automated Acceptance Testing
• Chapter 9 — Testing Nonfunctional Requirements
• Chapter 10 — Deploying and Releasing Applications

Part III — The Delivery Ecosystem
Discusses crosscutting practices and techniques that support the deployment pipeline.

• Chapter 11 — Managing Infrastructure and Environments
• Chapter 12 — Managing Data
• Chapter 13 — Managing Components and Dependencies
• Chapter 14 — Advanced Version Control
• Chapter 15 — Managing Continuous Delivery

PART I — FOUNDATIONS

Chapter 1 — The Problem of Delivering Software
The pattern that is central to this book is the deployment pipeline, that is, an automated implementation of your application’s build, deploy, test, and release process.

The aim of the deployment pipeline is threefold:

• First: it makes every part of the process of building, deploying, testing, and releasing software visible to everybody involved, aiding collaboration.
• Second: it improves feedback so that problems are identified, and so resolved, as early in the process as possible.
• Third: it enables teams to deploy and release any version of their software to any
  environment at will through a fully automated process.

The main purpose here is how to reduce the stress on release days, and how to ensure that each release is predictably reliable.

Antipattern: Deploying Software Manually — the main signs of this antipattern are:

• The production of extensive, detailed documentation that describes the
  steps to be taken and the ways in which the steps may go wrong
• Reliance on manual testing to confirm that the application is running
  correctly
• Frequent corrections to the release process during the course of a release
• Environments in a cluster that differ in their configuration, for example
  application servers with different connection pool settings, etc.
• Releases that are unpredictable in their outcome, that often have to be
  rolled back or run into unforeseen problems

Over time, deployments should tend towards being fully automated.

Here are a few things that can exacerbate the problems associated with a release:

• When working on a new application, the first deployment to staging is
  likely to be the most troublesome.
• The longer the release cycle, the longer the development team has to make
  incorrect assumptions before the deployment occurs, and the longer it will
  take to fix them.
• The bigger the difference between development and production environments, the less realistic are the assumptions that have to be made during development.

The remedy is to integrate the testing, deployment, and release activities into the development process. Make sure everybody involved in the software delivery process, from the build and release team to testers to developers, work together from the start of the project.

Antipattern: Manual Configuration Management of Production Environments — the main signs of this antipattern are:

• Having deployed successfully many times to staging, the deployment into
  production fails.
• The operations team take a long time to prepare an environment and different versions of operating systems, libraries, or patch levels.
• You cannot step back to an earlier configuration of your system, which
  may include OS, application server, web server, RDBMS, or other settings.

Key practices — Configuration Management: being able to re-create every piece of infrastructure used by your application and your production environment exactly, preferably in an automated fashion. Virtualization can help you get started with this.

“In software, when something is painful, the way to reduce the pain is to do it more frequently, not less. So instead of integrating infrequently, we should integrate frequently; in fact, we should integrate as a result of every single change to the system”.

Create a Repeatable, Reliable Process for Releasing Software: releasing software should be easy (as simple as pressing a button), because you’ve tested every single part of the release process many times before. The two derived principles:

• Automate almost everything
• Build process up to the point where it needs specific human direction or decision making
• Deployment and release process
• Acceptance tests
• Database upgrades and downgrades
• Even network and firewall configuration can be automated
• Keep everything you need to build, deploy, test, and release your application in version control

Things that cannot be automated is much smaller than many people think. Here are some things that it’s impossible to automate:

• Exploratory testing relies on experienced testers
• Demonstrations of working software to representatives of your user community cannot be performed by computers
• Approvals for compliance purposes by definition require human intervention

You should start by looking at that part of your build, deploy, test, and release process that is currently the bottleneck. You can, and should, automate gradually over time.

The DevOps movement is focused on encouraging greater collaborationbetween everyone involved in software delivery in order to release valuable software faster and more reliably.

Principles of Software Delivery

• Create a Repeatable, Reliable Process for Releasing Software
• Automate almost everything
• Keep Everything in Version Control
• If It Hurts, Do It More Frequently, and Bring the Pain Forward
• Build Quality In (The earlier you catch defects, the cheaper they are to fix)
• Done Means Released (Really, a feature is only done when it is delivering
  value to users; not always means released into production)
• Everybody is responsible for the Delivery Process
• Continuous Improvement

The Antipatterns that we’ve seen in this chapter

• Deploying Software Manually
• Deploying to a Production-like Environment Only after Development is Complete
• Manual Configuration Management of Production Environments

Chapter 2 — Configuration Management
Refers to the process by which all artifacts relevant to your project, and the relationships between them, are stored, retrieved, uniquely identified, and modified.

• Every single artifact related to the creation of your software should be under version control
• It includes source code, database scripts, build and deployment scripts, documentation, libraries, configuration files, etc.

Using version control is important in this context to provides access to, every version of every file that has ever been stored. Also, allows teams that may be distributed across space and time to collaborate.

It is impossible to do continuous integration, release management, and deployment
pipelining without the Configuration Management.

And how your team manage check-ins frequently? A recommended solution is to create a separate branch within the version control system for new functionality. At some point, when the changes are deemed satisfactory, they will be merged into the main development branch. This is a bit like a two-stage check-in.

To ensure you aren’t going to break the application when you check in, two practices are useful: 1) Run your commit test suite before the check-in 2) Introduce changes incrementally, at the conclusion of each separate incremental change or refactoring.

Managing Application Configuration means that you need to consider: A) Configuration Information B) Deployment Scripts access and how it vary between environments, applications, and versions C) Authentication and passwords D) Infrastructure

Puppet and CfEngine are two examples of tools that make it possible to manage operating system configuration in an automated fashion

There are two principles that, as we have found, form the basis of an effective configuration management strategy:

• Keep binary files independent from configuration information
• Keep all configuration information in one place

Chapter 3 — Continuous Integration
The teams that use continuous integration effectively are able to deliver software much faster, and with fewer bugs, than teams that do not. Bugs are caught much earlier in the delivery process when they are cheaper to fix, providing significant cost and time savings.

There are three things that you need before you can start with continuous integration: 1) Version Control 2) Automated Build 3) Agreement of the team

Prerequisites for Continuous Integration:

• Check in Regularly: the most important practice for continuous integration to work properly is frequent check-ins to trunk or mainline
• Create a comprehensive automated test suite: unit, component and acceptance tests (if the application meets the acceptance criteria)
• Keep the Build and Test process short: avoid waiting too long for the software building and multiple commits
• Managing your development workspace: developers should be able to run the build, execute the automated tests, and deploy the application in an environment under their control

Essential Practices
Continuous integration is a practice, not a tool, and it depends upon discipline to make it effective. T he objective of our CI system is to ensure that our software is working, in essence, all of the time. Here are the practices that we enforce on our teams:

• Don’t check in on a broken build
• Always Run All Commit Tests Locally before Committing (or Get Your CI Server to Do It for You)
• Never go home on a broken build
• Always be prepared to revert to the previous revision
• Time-Box Fixing before reverting (for example, after 10 minutes you aren’t finished with the solution, revert to the previous version)
• Test-Driven Development

Chapter 4 — Implementing a Testing Strategy
Building quality in means writing automated tests at multiple levels (unit, component, and acceptance) and running them as part of the deployment pipeline. Manual testing is also an essential part of building quality in: Showcases, usability testing, and exploratory testing need to be done continuously throughout the project.

The automated tests ensure that any problems that compromise the fulfillment of the requirements are caught early when the cost of fixing them is low. Brian Marick categorized tests according to whether they are business facing or technology facing, and whether they support the development process or are used to critique the project

Regression tests aren’t mentioned on the quadrant diagram because they are a crosscutting category, but are particularly important.

High-quality software is only possible if testing becomes the responsibility of everybody involved in delivering software and is practiced right from the beginning of the project and throughout its life. Automated tests should be supplemented with manual testing such as exploratory testing and showcases.

PART II — THE DEPLOYMENT PIPELINE

The five most likely causes of a failure in pipeline are:

• A bug in the application’s code
• A bug or invalid expectation in a test
• A problem with the application’s configuration
• A problem with the deployment process
• A problem with the environment

Chapter 5 — Anatomy of the Deployment Pipeline
A deployment pipeline is an automated manifestation of your process for getting software from version control into the hands of your users. The entire process — from concept to cash — can be modeled as a value stream map. A high-level value stream map for the creation of a new product is shown below.

The part of value stream, considering the shaded boxes (from development through to release) considers:

• Builds pass through it many times on their way to release
• Every change creates a build, begun with commit to the version control
• As the build passes each test of its fitness, confidence in it increases, which means that the environments the build passes through become more productionlike
• Eliminate unfit release candidates as early in the process
• Get feedback on the root cause of failure to the team as rapidly as possible
• We must also automate deployment to testing, staging, and production environments to remove these manually intensive, error-prone steps

The stages of deployment pipeline ensure that error-prone and complex steps are automated, reliable, and repeatable in execution. The ability to deploy the system at all stages of its development by pressing a button encourages its frequent use by testers, analysts, developers, and (most importantly) users.

There are some practices you should follow to get the benefits in the deployment pipeline:

• Only Build your binaries once: the code will be compiled repeatedly in different contexts: during the commit process, again at acceptance test time, again for capacity testing, and often once for each separate deployment target. Deploying the binaries to every environment will lead you to managing your configuration correctly, better-structured build systems
• Deploy the same way to every environment: it’s important to use the same deploy-time configuration mechanism for each of your applications to improve efficiency when you’re trying to work out the root cause of some bug.Using the same script to deploy to production that you use to deploy to development environments is a fantastic way to prevent the “it works on my machine” syndrome
• Smoke-Test your deployments: you should have an automated script that does a smoke test to make sure that it is up and running (dependencies such as a DB, messaging bus, or external service)
• Deploy into a copy of production
• If Any Part of the Pipeline Fails, Stop the Line
• The Commit Stage. Useful metrics include:
• Test coverage (if your commit tests only cover 5% of your codebase, they’re pretty useless)
• Amount of duplicated code
• Cyclomatic complexity
• Afferent and efferent coupling
• Number of warnings
• Code style
• Automated Acceptance Test Best Practices: analyze if the tests cost more effort than they save. Keep in mind that changing the way the creation and maintenance of the tests are managed can dramatically reduce the effort expended and change the cost-benefit equation significantly
• Manual Testing and Nonfunctional Testing
• Automated Deployment and Release
• Backing Out changes: on no account should you have a different process for backing out than you do for deploying, or perform incremental deployments or rollbacks
• Building on sucess

Implementing a Deployment Pipeline
In general, the steps to implement an incremental deployment pipeline approach are:

1. Model your value stream and create a walking skeleton: map out the part of your value stream that goes from check-in to release
2. Automate the build and deployment process: the build process should be performed every time someone checks in by your CI server software; automating deployment happens in one or several machines. In some companies, can be called the staging or UAT environment
3. Automate unit tests and code analysis: full commit stage (running unit tests, code analysis, and a selection of acceptance and integration tests on every check-in)
4. Automate acceptance tests: functional and nonfunctional tests
5. Automate releases: the whole pipeline should be implemented incrementally. Records manual process (when the process is started and when it completes) to identify bottlenecks

Metrics
Feedback is at the heart of any software delivery process. The best way to improve feedback is to make the feedback cycles short and the results visible. Measure what really matters:

• Cycle time — the most importante global metric in delivery process. Encourages the practices that increase quality, such as the use of a comprehensive automated suite of tests that is run as a result of every check-in. A number of other diagnostics that can warn you of problems:
• Automated test coverage
• Properties of the codebase (amount of duplication, cyclomatic, complexity, efferent and afferent coupling, style problems, and so on)
• Number of defects
• Velocity (the rate at which your team delivers working, tested, ready for use code)
• Number of commits to the version control system per day
• Number of builds per day
• Number of build failures per day
• Duration of build, including automated test
• Number of defects — secondary measure, considering the quality process above

Once you know the cycle time for your application, you can work out how best to reduce it. You can use the Theory of Constraints to do this by applying the following process.

1. Identify the limiting constraint on your system. To pick an example at random, perhaps it’s the manual testing process
2. Exploit the constraint. This means ensuring that you should maximize the throughput of that part of the process
3. Subordinate all other processes to the constraint. For example, if your developers work developing stories at full capacity, the backlog of stories waiting to be tested would keep on growing. Instead, have your developers work just hard enough to keep the backlog constant and spend the rest of their time writing automated tests to catch bugs so that less time needs to be spent testing manually
4. Elevate the constraint. if your cycle time is still too long, you need to increase the resources available — hire more testers, or invest more effort in automated testing
5. Rinse and repeat. Find the next constraint on your system and go back to step 1

The purpose of the deployment pipeline is to give everyone involved in delivering software visibility into the progress of builds from check-in to release. A deployment pipeline, in turn, depends on having some foundations in place:

• Good Configuration Management
• Automated scripts for building and deploying
• Automated tests to prove that your application will deliver value to its users

Chapter 6 — Build and Deployment Scripting
There are two types of build tools: task-oriented (for example, Ant, NAnt and MsBuild) describe the dependency network in terms of a set of tasks; product-oriented (such as Make), describes things in terms of the products they generate, such as an executable.

• Task-oriented: each task will know whether or not it has already been executed as part of the build.
• Product-oriented: the world is modeled as a set of files. The tools keep state on the files generated by each of the tasks. This is great when you’re compiling C or C++ (for example), because Make will ensure that you only compile those source code files that have changed since the last time the build was run.

There are some principles and practices of build and deployment scripting which should apply whichever technology you use:

• Create a Script for Each Stage in Your Deployment Pipeline
• Use an Appropriate Technology to Deploy Your Application
• Use Your Operating System’s Packaging Tools
• Evolve Your Deployment System Incrementally

Changes to testing and production environments should be made through an automated process, using three kinds of scripts (1- Script that logs into each box and runs the appropriate commands; 2- Script that runs locally, and have agents that run the script on each of the remote machines; 3- Package your application and push out new versions).

Here are some solutions and strategies used to solve common build and deployment problems:

• Always Use Relative Paths (not dependent with the configuration of a specific machine)
• Eliminate Manual Steps
• Build In Traceability from Binaries to Version Control
• Don’t Check Binaries into Version Control as Part of Your Build

Chapter 7 — The Commit Stage
The commit stage represents the entrance into the deployment pipeline. Should be focused on one thing: detecting, as fast as possible, the most common failures that changes to the system may introduce, and notifying the developers so they can fix the problems quickly.

The inputs are source code, and the outputs are binaries and reports (the outputs are stored in the Artifact Repository). Somebody checks a change into mainline (trunk) in version control. Your CI server detects the change, checks out the source code, and performs a series of tasks. You need to give developers ownership. It’s intimately tied to their work and their productivity.

A commit stage provides a huge advance in the quality and reliability of your delivery process — assuming you follow the other practices involved in continuous integration, such as checking in regularly and fixing any defects as soon as they are discovered.

Mike Cohn recommends to use test automation pyramid to structure your automated test suite:

• Unit tests: form the vast majority of the tests. But since they execute so fast, the unit test suite should still complete in just a few minutes
• Acceptance tests: subdivided into service and UI tests; run against the full running system, ensuring the application is working and delivering the expected business value

In particular, running unit tests shouldn’t touch the filesystem, databases, libraries, frameworks, or external systems. Any calls to these parts of your environment should be replaced with test doubles, such as mocks and stubs.

And here are some strategies to design commit test:

• Avoid the User interface: it tends to involve a lot of components or levels of the software under test; and timescales are different between human and computer
• Use Dependency Injection between objects
• Avoid the database: not store the results in DB, it’s slower to run and more complex to establish and manage
• Avoid asynchrony in Unit Test
• Using Test Doubles

Chapter 8 — Automated Acceptance Testing
Once you have automated acceptance tests in place, you are testing thebusiness acceptance criteria (functional or nonfunctional) of your application, that is, validating that it provides users with valuable functionality. It acts to focus the attention of all members of a delivery team on what really counts: the behavior that the users need from the system.

Acceptance tests are typically run against every version of your software that passes the commit tests. They are business-facing, not developer-facing as unit tests. They test whole stories at a time against a running version of the application. The workflow of the acceptance test stage:

The real reason people don’t like automated acceptance testing is that it is perceived as being too expensive. However, it is possible to decrease the cost to well below the level where it becomes efficient and cost-effective.

Since the feedback loop is much shorter, defects are found much sooner, when they are cheaper to fix. The collaboration is also better, considering testers, developers and customers suite working together, focusing on the business value that the application needs to deliver.

Creating and maintaining effective automated acceptance tests:

• Follow the INVEST (independent, negotiable, valuable, estimable, small, and testable) principles to write stories and maintainable acceptance tests
• Once you have a set of acceptance criteria describing the value to be delivered to users, the next step is to automate them

• Iterative development processes are essential to the creation of quality software. Analysts spend much of their time defining acceptance criteria
• Behavior-driven development helps to meet customer’s expectation (behavior of the application), writing acceptance criteria in order to automate tests
  Given some initial context,
  When an event occurs,
  Then there are some outcomes.

Chapter 9 — Testing Nonfuncional Requirements
The crosscutting nature of many NFRs means that it is hard to manage the risks that they pose to any given project. This, in turn, can lead to two paralyzing behaviors:

• Not paying enough attention to them from the start of the project
• At the other extreme, defensive architecture and over-engineering

We must work closely with customers and users to determine the sensitivity points of our application and define detailed NFRs based upon real business value.

It helps to decide the correct architecture and create requirements and acceptance criteria. The delivery team needs to create and maintain automated tests to ensure that these requirements are met.

Michael Nygard has an important definition about performance and throughput:

• Performance: is a measure of the time taken to process a single transaction, and can be measured either in isolation or under load
• Throughput: is the number of transactions a system can process in a given timespan. It is always limited by some bottleneck in the system

The strategy to address capacity problems include: application architecture (particular attention to process and network boundaries), patterns, boundaries and data structure. And some types of measurements that can be performed:

• Scalability testing: how do the response time of an individual request and the number of possible simultaneous users change as we add more servers, services, or threads?
• Longevity testing: this involves running the system for a long time to see if the performance changes over a protracted period of operation. This type of testing can catch memory leaks or stability problems
• Throughput testing: how many transactions, or messages, or page hits per second can the system handle?
• Load testing: what happens to capacity when the load on the application increases to production-like proportions and beyond?

Automating the Capacity Testing needs to define the strategy about recording (interactions performed via the user interface) and simulating realistic use of the system.

Chapter 10 — Deploying and Releasing Applications
The focus here is on the problem of deploying application software to environments shared by multiple users. Also, discuss releasing products and ensuring continuous delivery of client-installed software.

The release strategy considers functional and nonfunctional requirements for both software development and for the design, configuration, and commissioning of hardware environments. The release plan is a vital component of release strategy. It involves automated scripts, documentation, or other procedures needed to reliably and repeatedly deploy the application into the production environment.

The key to deploying any application in a reliable, consistent manner is constant practice: Use the same process to deploy to every environment, including production. Automating the deployment should start with the very first deployment to a testing environment.

A recommended deployment strategy is pick some stories and create iterations to deliver them. You should have the following in place:

• Your deployment pipeline’s commit stage
• A production-like environment to deploy to
• An automated process that takes the binaries created by your commit stage and deploys them into the environment
• A simple smoke test that verifies that the deployment worked and the application is running

There are several methods of performing a rollback (in case a deployment goes wrong), the most advanced techniques are blue-green deployments and canary releasing. Both can be used to perform zero-downtime releases and rollbacks.

Blue-green deployments
The idea is to have two identical versions of your production environment — blue and green. The users of the system are routed to the green environment, which is the currently designated production. So, releasing a new version of the application, will deploy it to the blue environment.

We can run smoke tests against the blue environment to check it is working properly. Moving to the new version is as simple as changing the router configuration to point to the blue environment instead of the green environment. The blue environment thus becomes production.

If something goes wrong, we simply switch the router back to the green environment. We can then debug what went wrong on the blue environment.

Canary releasing
Involves rolling out a new version of an application to a subset of the production servers to get fast feedback. This quickly uncovers any problems with the new version without impacting the majority of users. It’s a great way to reduce the risk of releasing a new version and select users to use a new version of the application. Also used for A/B tests.

Continuous Deployment — “more frequent releases lead to lower risk in putting out any particular release”.

Don’t Make Changes Directly on the Production Environment! Most downtime in production environments is caused by uncontrolled changes. Production environments should be completely locked down, so that only your deployment pipeline can make changes to it.

PART III — THE DELIVERY ECOSYSTEM

Chapter 11 — Managing Infrastructure and Environments
The degree to which you need to take the configuration management of your infrastructure will depend on its nature. In a large and complex system with many configuration points, the approach in this chapter can save your project.

In large organizations, operations teams usually use metrics such as MTBF, MTTR and SLAs to manage their performance. It’s necessary to support the development team in delivering software with an effective change management process and controlling risks to release new features in production.

The main concerns are: documentation and auditing; alerts for abnormal events; continuity planning (RPO — recovery point objective). The development team and operations team should sit down at the beginning of every project and decide how deployment of the application will be performed.

Even if you don’t have control over the selection of your infrastructure, if you intend to fully automate your build, integration, testing, and deployment, you must address each of the following questions:

• How will we provision our infrastructure?
• How will we deploy and configure the various bits of software that form
  part of our infrastructure?
• How do we manage our infrastructure once it is provisioned and configured?

Controling access to infrastructure (to prevent anyone from making a change without approval); Create and Maintain your infrastructure under version control (scripts, configurations, OS definitions, etc.); Changesto infrastructure (needs to follow change management process too);

You should be able to take a vanilla set of servers and deploy everything to them from scratch. Indeed, a great way to introduce automation or virtualization is to make it a test of your environment provisioning process.

Puppet is a popular open source systems that manages configuration through a declarative, external domain-specific language (DSL) tailored to configuration information. There is a master server with a list of machines that it controls and communicates with the server to ensure that the servers under Puppet’s control are synchronized with the latest version of the configuration.

Managing the Configuration of Middleware
You also need to think about the management of the middleware — whether web servers, messaging systems, or commercial off-the-shelf software (COTS).

They can be decomposed into three parts: binaries, configuration, and data. The three have different lifecycles which makes it important to treat them independently. Database schemas, web server configuration files, application server configuration information, message queue configuration, and every other aspect of the system that needs to be changed for your system to work should be under version control.

Managing Infrastructure Services
Network configuration (DNS, DHCP, firewall, SMTP, etc.), Network monitoring system (Nagios, Zabbix, etc.), Logging and Tests (smoke tests and integration tests) are important things to check in production environment.

Virtualization
Provides a way to amplify the benefits of the techniques, for automating the provisioning of servers and environments. It adds a layer of abstraction on top of one or more computer resources.

Platform virtualization is an important concept that simulates an entire computer system so as to run multiple instances of operating systems simultaneously on a single physical machine. In this configuration, there is a virtual machine monitor (VMM), or hypervisor, which has full control of the physical machine’s hardware resources.

In particular, virtualization provides the following benefits:

• Fast response to changing requirements
• Consolidation (CI and testing infrastructure)
• Standardizing hardware
• Easier-to-maintain baselines (images)

Also consider three categories of cloud computing: applications in the cloud, platforms in the cloud, and infrastructure in the cloud. Finally, there are BDD(Behavior-Driven Monitoring) in operations team to write automated tests to verify the behavior of their infrastructure.

Chapter 12 — Managing Data
The fundamental principles that govern data management are the same than Deployment Pipeline. The key is to ensure that there is a fully automated process for creating and migrating databases. This process is used as part of the deployment process, ensuring it is repeatable and reliable.

The same process should be used whether deploying the application to a development or acceptance testing environment with a minimal dataset, or whether migrating the production dataset as part of a deployment to production.

Even with an automated database migration process, it is still important to manage data used for testing purposes carefully. While a dump of the production database can be a tempting starting point, it is usually too large to be useful. Instead, have your tests create the state they need, and ensure they do this in such a way that each of your tests is independent of the others.

In this chapter we’ll see database tests, how to manage data and changes in database. First, considering database tests we have the below concepts:

• Test application: to assert that it possesses characteristics that we desire
• Unit tests: to protect from effects of inadvertently changes that breaks application
• Acceptance tests: to assert the application delivers the expected value to users
• Capacity tests: to assert the application meets our capacity requirements
• Suite of integration tests: to confirm that our application communicates correctly with services it depends on

Versioning database is one demand that we need to work on CI/CD process. You need to track the changes to the database. From a version x to version x + 1 (a roll-forward script) and From version x + 1 to version x (a roll-back script).

The technique of managing database changes achieves two goals:

• Allows you to continuously deploy your application without worrying about the current state of the database in the environment you’re deploying to
• Your deployment script simply rolls the database back or forward to the version your application is expecting

There are two concerns to consider Managing Test Data:

• Test performance: to make sure our tests run as fast as possible. Using unit tests (not running against a database at all, or running against an in-memory database) and other types of tests (test data carefully, not using a dump of the production database except in a few limited cases).
• Test isolation: is a durable store of information that allows changes to persist between test invocations — unless you explicitly do something to prevent it.

One way to manage incremental change is to make applications work with multiple versions of your database, so that the database can be migrated independently of the applications it depends on. This technique is also useful for zero-downtime releases.

Another approach to managing database changes and refactorings is to use an abstraction layer, in the form of stored procedures and views. If the application accesses the database through such an abstraction layer, it is possible to make changes to the underlying database objects while keeping the interface presented to the application by the views and stored procedures constant.

In summary, here are some important principles and practices from this chapter:

• Version your database and use a tool like DbDeploy to manage migrations
• Strive to retain both forward and backward compatibility with schema changes so that you can separate data deployment and migration issues from application
• Make sure tests create the data they rely on as part of the setup process, and that data is partitioned to ensure it does not affect other tests that might be running at the same time
• Use the application’s public API to set up the correct state for tests when possible
• In most cases, don’t use dumps of the production dataset for testing purposes. Create custom datasets by carefully selecting a smaller subset of production data, or from acceptance or capacity test runs.

Chapter 13 — Managing Components and Dependencies
This chapter describes how to keep your application releasable at all times, despite being under constant change. One of the key techniques for this is componentization of larger applications, so we will treat componentization, including building and managing large projects with multiple components, at length. Also describe how to create and manage build systems for component-based applications.

Components are large-scale code structure within an application, with a well-defined API, that could potentially be swapped out for another implementation. A component-based software system is distinguished by the fact that the codebase is divided into discrete pieces that provide behavior through well-defined, limited interactions with other components.

Continuous integration is designed to give you a high level of confidence that your application is working at the functional level. The deployment pipeline, an extension of continuous integration, is designed to ensure that your software is always releasable.

The aim of continuous delivery is for the application to always be in a releasable state. How can we achieve this? One approach is to create branches in version control that are merged when work is complete, so that mainline is always releasable. How is it possible to have everybody working on mainline, and still keep your application in a releasable state at all times?

There are four strategies to employ in order to keep your application releasable in the face of change:

• Hide new functionality until it is finished
• Make all changes incrementally as a series of small changes, each of which is releasable
• Use branch by abstraction to make large-scale changes to the codebase
• Use components to decouple parts of your application that change at different rates

Dependencies
First, it’s necessary to understand the difference between components and libraries:

• Libraries: refer to software packages that your team does not control, other than choosing which to use and are usually updated rarely
• Components: are pieces of software that your application depends upon, but which are also developed by your team, or other teams in your organization and are usually updated frequently

This distinction is important because when designing a build process, there are more things to consider when dealing with components than libraries.

For example, do you compile your entire application in a single step, or compile each component independently when it changes? How do you manage dependencies between components, avoiding circular dependencies?

The distinction between build-time and runtime dependencies is as follows:

• Build-time dependencies: must be present when your application is compiled and linked (if necessary);
• Runtime dependencies: must be present when the application runs, performing its usual function.

In your deployment pipeline you will be using many different pieces of software that are irrelevant to the deployed copy of the application, such as unit test frameworks, acceptance test frameworks, build scripting frameworks, and so forth.

Almost all modern software systems consist of a collection of components. These components may be DLLs, JAR files, OSGi bundles, Perl modules, etc.

Implementing a deployment pipeline that takes account of the interactions between components, is a nontrivial task. The results of this complexity are often demonstrated
by builds that take many hours to assemble a deployable, testable application.

Most applications start off as a single component. Some start off as two or three (for example, a client-server application). So why should a codebase be split into components, and how should the relationships between them be managed? Unless these relationships are managed effectively, it can compromise the ability to use them as part of a continuous integration system.

Here are some good reasons to separate out a component from your codebase:

1. Part of your codebase needs to be deployed independently
2. You want to turn a monolithic codebase into a core and a set of plugins, perhaps to replace some part of your system with an alternative implementation, or to provide user extensibility
3. The component provides an interface to another system (for example a framework or a service which provides an API)
4. It takes too long to compile and link the code
5. It takes too long to open the project in the development environment
6. Your codebase is too large to be worked on by a single team

We do not recommend making teams responsible for individual components. This is because in most cases, requirements don’t divide along component boundaries.

In our experience, cross-functional teams in which people develop features end-to-end are much more effective. It’s better to split teams up so that each team takes on one stream of stories (perhaps all with a common theme), and touches whatever components they need to in order to get their work done.

Organize teams by functional area rather than by component, ensure that everybody has the right to change any part of the codebase, rotate people between teams regularly, and ensure that there is good communication between teams.

Here are a few examples of circumstances where it makes sense to have separate pipelines:

• Parts of your application that have a different lifecycle (perhaps you build your own version of an OS kernel as part of your application, but you only need to do this once every few weeks)
• Functionally separate areas of your application that are worked on by different (perhaps distributed) teams may have components specific to those teams
• Components that use different technologies or build processes
• Shared components that are used by several other projects
• Components that are relatively stable and do not change frequently
• It takes too long to build your application, and creating builds for each component will be faster (but beware, the point at which this becomes true is much later than most people think)

The build for each component or set of components should have its own pipeline to prove that it is fit for release. This pipeline will perform the following steps:

• Compile the code, if necessary
• Assemble one or more binaries that are capable of deployment to any environment
• Run unit tests
• Run acceptance tests
• Support manual testing, where appropriate

The integration pipeline takes as its starting point the binary output from each of the components that comprise your system

• First stage: create a package suitable for deployment by composing the appropriate collections of binaries
• Second stage: deploy the resulting application to a production-like environment and run smoke tests against it to give early indication of integration problems. If this stage is successful, then the pipeline should move on to a conventional acceptance test stage, running whole application acceptance tests in the usual way

A good CI tool can trace the origins of the components that went into a particular build of the application, and also show you which versions of your components integrated together successfully.

Your CI tool should also ensure that consistent versions of components are used in the pipeline. It should prevent dependency hell, and ensure that a change in version control which affects multiple components only propagates through the pipeline once

Consider a pipeline with the following stages: compile, unit test, acceptance test, manual acceptance test, and production.

Maven is an extensible build management tool for Java projects. In particular, it provides a sophisticated mechanism for managing dependencies.

Chapter 14 — Advanced Version Control
They enable teams to work together on separate parts of an application while maintaining a system of record. There are three good reasons to branch your code:

1. A branch can be created for releasing a new version of your application. This allows developers to continue working on new features without affecting the stable public release. When bugs are found, they are first fixed in the relevant public release branch, and then the changes are applied to the mainline. Release branches are never merged back to mainline.
2. When you need to spike out a new feature or a refactoring; the spike branch gets thrown away and is never merged.
3. It is acceptable to create a short-lived branch when you need to make a large change to the application that is not possible with any of the methods described in the last chapter.

The main purpose of branches is to facilitate parallel development: the ability to work on two or more work streams at the same time without one affecting the other. For example, it is common to branch on release, allowing for ongoing development on mainline and bugfixing in the release branch.

There are several other reasons why teams may choose to branch their code:

• Physical: system’s physical configuration — branches are created for files, components, and subsystems.
• Functional: system’s functional configuration — branches are created for features, logical changes, both bugfixes and enhancements, and other significant units of deliverable functionality (e.g., patches, releases, and products).
• Environmental: system’s operating environment — branches are created for various aspects of the build and runtime platforms (compilers, libraries, hardware, operating systems, etc.) and/or for the entire platform.
• Organizational: team’s work efforts — branches are created for activities/tasks, subprojects, roles, and groups.
• Procedural: team’s work behaviors — branches are created to support various policies, processes, and states.

Perhaps the most important practice that makes continuous integrationpossible
is that everybody checks in to mainline at least once a day. So if you merge your branch to (not just from) mainline once a day, you’re OK.

If you’re not doing that, you’re not doing continuous integration. Indeed, there is a school of thought that any work on a branch is, in the lean sense, waste — inventory that is not being pulled into the finished product.

It’s not uncommon to see continuous integration basically ignored and people branching promiscuously, leading to a release process that involves many branches. A more manageable branching strategy is to create long-lived branches only on release.

In this model, new work is always committed to the trunk. Merging is only performed when a fix has to be made to the release branch, from which it is then merged into mainline.

This model is better because the code is always ready to be released, and the releases are therefore easier. There are fewer branches, so much less work has to be done merging and keeping track of the branches.

The incremental approach certainly requires more discipline and care — and indeed more creativity. But it significantly reduces the risk of your changes breaking the application, and will save you and your team a great deal of time merging, fixing breakages, and getting your application into a deployable state.

Distributed Version Control Systems
In DVCS each user keeps a selfcontained, first-class repository on their computer. There is no need for a privileged “master” repository, although most teams designate one by convention (otherwise it is impossible to do continuous integration).

From this design principle, many interesting characteristics follow:

• You can start using a DVCS in a few seconds — just install it, and commit your changes into a local repository
• You can pull updates individually from other users without them having to check their changes into a central repository
• You can push updates to a selected group of users without everyone being forced to take them
• You can check your changes into source control while you are working offline
• You can commit incomplete functionality regularly to your local repository to check point without affecting other users

Since there are many copies of the full repository, DCVSs are more fault-tolerant, although master repositories should still be backed up. If you think that using a DVCS sounds rather like everyone having their own SCCS or RCS, you are right.

Where distributed version control systems differ from the approaches in the previous section is in the way they handle multiple users, or concurrency. Instead of using a central server with a version control system on it to ensure that several people can work on the same branch of the codebase at the same time, it takes the opposite approach:

• Every local repository is effectively a branch in its own right, and there is no “mainline”
• Much of the work that goes into the design of a DVCS is spent on making it easy for users to share their changes with each other

The beauty of distributed version control comes in the form of spontaneous team formation, as people with a common interest in a bug or feature start to work on it, bouncing that work between them by publishing branches and merging from one another.

Stream-Based Version Control Systems
IBM’s ClearCase is one example of Stream-based version control systemsthat are designed to ameliorate the merge problem by making it possible to apply sets of changes to multiple branches at once.

In the stream paradigm, branches are replaced by the more powerful concept of streams, which have the crucial distinction that they can inherit from each other. Thus, if you apply a change to a given stream, all of its descendent streams will inherit those changes.

Consider how this paradigm helps with two common situations: applying a bugfix to several versions of your application and adding a new version of a third-party library to your codebase.

In stream-based systems, developers are encouraged to develop in their own workspaces, developing functionality without affecting other users. When they are ready, they can promote their changes to make them available to others.

For example, you might be working on a stream you’ve created for a particular feature. When the feature is complete, you can promote all the changes in that stream to the team’s stream, which can be continuously integrated.

When your testers want to test completed functionality, they could have their own stream, to which all functionality ready for manual testing can be promoted. Functionality that has passed testing can then be promoted to a release stream.

One of the problems with the stream model of development is that promotionis done at the source level, not the binary level. As a result, every time you promote a change to a higher stream, you have to check out source and rebuild the binary.

Develop on Mainline
It is an extremely effective way of developing, and the only one which enables you to perform continuous integration. In this pattern, developers almost always check in to mainline. Branches are used only rarely. The benefits of developing on mainline include:

• Ensuring that all code is continuously integrated
• Ensuring developers pick up each others’ changes immediately
• Avoiding “merge hell” and “integration hell” at the end of the project

In this pattern, in normal development, developers work on mainline, committing code at least once a day.

Branch for Release
By creating a release branch, developers can keep checking in to mainline, while changes to the release branch are made for critical bugfixes only. In this pattern:

• Features are always developed on mainline.
• A branch is created when your code is feature-complete for a particular release and you want to start working on new features.
• Only fixes for critical defects are committed on branches, and they are merged into mainline immediately.
• When you perform an actual release, this branch is optionally tagged (this step is mandatory if your version control system only manages changes on a per-file basis, like CVS, StarTeam, or ClearCase).

Branch by Feature
Every story or feature is developed on a separate branch. Only after a story is accepted by testers, it is merged to mainline so as to ensure that mainline is always releasable.

It is worth emphasizing that branching by feature is really the antithesis of continuous integration, and all of our advice on how to make it work is only about ensuring that the pain isn’t too horrible come merge time. There are exceptions where this may make sense, such as open source projects or small teams of experienced developers working with distributed version control systems.

Branch by Team
This pattern is an attempt to address the problem of having a large team of developers working on multiple work streams while still maintaining a mainline that can always be released.

As with branch by feature, the main intent of this pattern is to ensure that the trunk is always releasable. A branch is created for every team, and merged into trunk only when the branch is stable. Every merge to any given branch should immediately be pulled into every other branch.

Here is the workflow for branching by team:

1. Create small teams, each working on its own branch
2. Once a feature/story is completed, the branch is stabilized and merged to trunk
3. Any changes on trunk get merged to every branch daily
4. Unit and acceptance tests are run on every check-in on the branch
5. All tests, including integration tests, are run on trunk every time a branch is merged into it

In this pattern, developers only check in to their team’s branch. This branch is only merged to trunk when all the features being worked on are complete.

From a CI perspective, this strategy has some drawbacks like the unit of work under this strategy is scoped to a whole branch, not just a particular change. In other words, you can’t merge a single change to mainline — you have to merge the whole branch, otherwise there is no way of knowing whether you have violated the mainline policy.

If the team discovers a bug after it has merged to trunk, and there are other changes in the branch, they can’t just merge the fix. In this situation, the team would either have to get the branch stable again, or create yet another branch just for the fixes.

Chapter 15 — Managing Continuous Delivery
Implementing continuous delivery depends on effective collaboration between everyone involved in delivery, support from executive sponsors, and willingness of people on the ground to make changes.

Using these practices, even large organizations with complex applications can deliver new versions of their software rapidly and reliably. That means not only that businesses can get a faster return on their investment, but that they can do so with reduced risks and without incurring the opportunity cost of long development cycles — or worse, delivering software that is not fit for purpose.

To use an analogy with lean manufacturing, software that is not being delivered frequently is like inventory stored up in a warehouse. Indeed, it’s costing you money to store it.

A Maturity Model for Configuration and Release Management
This model helps to identify where an organization stands in terms of the maturity of its processes and practices and defines a progression that an organization can work through to improve.

The ultimate aim is for your organization to improve. The outcomes you want are:

• Reduced cycle time, so that you can deliver value to your organization faster and increase profitability
• Reduced defects, so that you can improve efficiency and spend less on support
• Increased predictability of your software delivery lifecycle
• The ability to adopt and maintain an attitude of compliance to any regulatory regime that you are subject to
• The ability to determine and manage the risks associated with software delivery effectively
• Reduced costs due to better risk management and fewer issues delivering software

We recommend, as ever, that you apply the Deming cycle — plan, do, check, act:

1. Use the model to classify your organization’s configuration and release management maturity
2. Choose an area to focus on where your immaturity is especially painful
3. Implement the changes
4. Once the changes have been made, use the acceptance criteria you created to measure if the changes had the desired effect
5. Repeat these steps, building upon your knowledge. Roll out improvements incrementally, and roll them out across your whole organization

An initial high-level picture might include the following phases:

• Identification: strategic objectives, projects and stakeholders
• Inception: requirements are gathered and analyzed. It usually includes:
• Business case
• High-level functional and nonfunctional requirements
• Release plan
• Testing strategy
• Architecture evaluation
• A risk and issue log
• Initiation: establish initial project infrastructure (team, basic infrastructure, emails, version control, environments, etc.)
• Development and deployment: it’s recommended an iterative and incremental process for developing and releasing software
• Operation: most projects don’t stop at the point of first release, and will continue to develop new functionality

Although many people agree on the benefits of an iterative process, we have often seen teams that claim to be doing iterative development but actually aren’t. So it’s worth reiterating what we consider to be the essential, basic conditions for an iterative process:

• Your software is always working, as demonstrated by an automated test suite including unit, component, and end-to-end acceptance tests that run every time you check in
• You deploy working software, at every iteration, into a production-like environment to showcase it to users (this is what makes the process incremental in addition to being iterative)
• Iterations are no longer than two weeks

There are many approaches to iterative and incremental development. One of the most popular is Scrum, an agile development process. We have seen Scrum succeed on many projects, but we have also seen it fail. Here are the three most common reasons for failure: 1) Lack of commitment 2) Ignoring good engineering 3) Adapting until the process is no longer an agile.

Risk Management Process
Risk management is the process of making sure that:

• The main project risks have been identified
• Appropriate mitigating strategies have been put in place to manage them
• Risks continue to be identified and managed throughout the course of the project

A good starting point to analyze any project is to pose these questions (this list has worked well for us on several projects):

• How are you tracking progress?
• How are you preventing, discovering and tracking defects?
• How do you know a story is finished?
• How are you managing your environments?
• How are you managing configuration, such as test cases, deployment scripts, environment and application configuration, database scripts, and external libraries?
• How often do you run your automated tests?
• How are you building and deploying your software?
• How are you ensuring that your release plan is workable and acceptable to the operations team?
• How are you ensuring that your risk-and-issue log is up-to-date?

These questions are not prescriptive, which is important because every team needs to have a certain amount of flexibility to choose the most suitable process for their specific needs, validating that the team will actually be able to deliver.

Common Delivery Problems — Their Symptoms and Causes
When things do go wrong (building, deploying, testing, and releasing software), work out how that could have been spotted early, and ensure that these symptoms are monitored.

Once you have observed the symptoms, you need to discover the root cause. “Why?” It is recommended that you ask “Why?” at least five times. Although this process sounds almost absurd, we have found it to be incredibly useful and totally foolproof. Once you know the root cause, you have to actually fix it.

Here’s a list of common symptoms, grouped by their root cause:

• Infrequent or Buggy Deployments
• Problem
• It takes a long time to deploy the build, and the deployment process is brittle
• Symptoms
• It takes a long time for bugs to be closed by testers.
• It takes a long time for stories to be tested or signed off by the customer
• Testers are finding bugs that developers fixed a long time ago
• Nobody trusts the UAT, performance, or CI environments, and people are
  skeptical as to when a release will be available
• Possible causes
• The deployment process is not automated
• There is not enough hardware available
• The hardware and operating system’s configuration are not managed correctly
• The deployment process depends on systems outside the team’s control.
• Poor Application Quality
• Poorly Managed Continuous Integration Process
• Poor Configuration Management

Compliance and Auditing
Many such regulatory regimes require audit trails that make it possible to identify for every change in a production environment, what were the lines of code that it came from, who touched them, and who approved the steps in the process.

Here are some common strategies we have seen employed for enforcing these kinds of regulations:

• Locking down who is able to access “privileged” environments
• Creating and maintaining an effective and efficient change management process for making changes to privileged environments
• Requiring approvals from management before deployments can be performed
• Requiring every process, from building to release, to be documented
• Creating authorization barriers to ensure that the people who create the software are not able to deploy it into production environments, as a protection against potential malicious interventions
• Requiring every deployment to be audited to see exactly what changes are being made

Strategies like these are essential in organizations subject to regulation, and can lead to drastic reductions in downtime and defect counts. Nonetheless they have a bad reputation because it is all too easy to implement them in ways that make change more difficult. However, the deployment pipeline makes it possible to enforce these strategies fairly easily while enabling an efficient delivery process.

In this section, we present some principles and practices to ensure compliance
with such regulatory regimes while maintaining short cycle times
Change Management
In regulated environments, it is often essential for parts of the build, deploy, test, and release process to require approval. In particular, manual testing environments, staging, and production should always be under strict access control so that changes to them can only be made through the organization’s change management process.

This may seem unnecessarily bureaucratic, but in fact research has demonstrated that organizations which do this have lower MTBF and MTTR. If your organization has a problem meeting its service levels due to uncontrolled changes to testing and production environments, we suggest the following process for managing approvals:

• Create a CAB (Change Advisory Board) with representatives from your development team, operations team, security team, change management team, and the business
• Decide which environments fall under the purview of the change management process. Ensure that these environments are access-controlled so that changes can only be made through this process
• Establish an automated change request management system that can be used to raise a change request and manage approvals. Anyone should be able to see the status of each change request and who has approved it
• Any time anybody wants to make a change to an environment, whether deploying a new version of an application, creating a new virtual environment, or making a configuration change, it must be done through a change request
• Require a remediation strategy, such as the ability to back out, for every change
• Have acceptance criteria for the success of a change. Ideally, create an automated test that now fails but will pass once the change is successful. Put an indicator on your operations management dashboard with the status of the test
• Have an automated process for applying changes, so that whenever the change is approved, it can be performed by pressing a button (or clicking a link, or whatever)

Finally, there are three more principles that should be followed when implementing and managing a change approval process:

1. Keep metrics on the system and make them visible. How long does it take for a change to be approved? How many changes are waiting for approval? What proportion of changes are denied?
2. Keep metrics that validate the success of the system and make them visible. What’s the MTBF and MTTR? What is the cycle time for a change? There is a more complete list of metrics defined in the ITIL literature
3. Hold regular retrospectives on the system, inviting representatives from each of your organization’s units, and work to improve the system based on feedback from these retrospectives