Continuous Delivery (CD)
[Article 4 of 6 in series Software Delivery, DevOps and CICD for the uninitiated]
In this article, we will cover:
- typical steps involved in software delivery
- the need for multiple phases and delivery environments
- a high level software architecture and process flow that achieves ‘Continuous Delivery’.
- strategies for avoiding and recovering from disrupted service during releases
For a reminder of what we mean by ‘Delivering’ software, including the distinction between ‘Customer-run’ and ‘Self-run’ applications, visit: Software Delivery 101 (Environments and Applications).
How do we deliver?
Customer run applications for which the target audience is not necessarily technical (e.g. a social media app) usually have a neatly packaged installation process (or ‘wizard’, think ‘Download, Open, Accept Terms, Accept Permissions, Install, Launch’).
Such users might be surprised to learn that for other applications which either have more technical audiences (e.g. a database server), or are self-run applications with an installation process that will never be exposed publicly, the procedure is very often far less polished. A typical installation process will likely involve:
- copying or unzipping files to certain directories
- editing configuration text files
- adding hostnames and ip addresses to firewall configurations
- stopping and starting services
- setting operating system environment variables
- creating operating system or admin users
- setting directory permissions
- generating security keys (e.g. SSH, PGP)
- installing libraries
- … etc.
For example, I won’t be volunteering to install this application: https://groups.google.com/forum/#!msg/turtl/q3kAYnAcH0s/AHhKABu2BQAJ)
The result is that delivery requires users with a high degree of skill, and takes a long time. Although one upside is that users are given more control, there is a high degree of risk: one mis-followed step or one slight deviation in the host environment from the expectation is likely to result in failure. Add to that the need to repeat these steps across multiple environments every time we have a new version… we’re in for a nightmare.
“Many releases were a “scary” experience because the release process wasn’t often practiced and there were many error-prone manual activities. Priority 1 incidents caused by manual-cofiguration mistakes weren’t uncommon. In addition, the release activities weren’t efficient. Just setting up the testing environment could take up to three weeks.”
Credit: Continuous Delivery: Huge Benefits, but Challenges Too
Assuming we’re delivering a product that either is itself, or comprises a self-run application, we’re going to need a hero of a tool to help us…
G. Configuration Manager
What does it do?: Allows the automation of operating system operations via runnable code.
Imperative programming is when you say how to get what you want. I.e. ‘do x, then do y’). By contrast, Declarative programming is when you say what you want. I.e. the code asserts a desired final state (make this be x, then make this be y) and leaves how to get there up to the software. With Declarative code, the programmer doesn’t need to worry about the initial state, and can often use the same code in different types of operating systems (e.g. Windows vs Linux), making it hyper-flexible.
Configuration Management tools tend be Declarative, allowing programmers to write instructions in the following vein:
- ensure certain files are unzipped and present in particular directories
- ensure text files contain certain content
- ensure certain hostnames and ip addresses are present in firewall configurations
- ensure certain services are running
- ensure environment variables are set to particular values
- ensure certain operating system users exist
- ensure a directory’s permissions are set
- ensure certain security keys are present
- ensure libraries are installed
Ensuring server configuration is always codified as a desired state and checking this code into a source code respository means that:
- you always have a description of a server’s (intended) present state and changes are controlled and auditable.
- configuration scripts can be run over and again on different serves and always result in a predictable end result.
Examples: Puppet, Ansible, Chef
As a decent Configuration Manager will support integrations with the key Binary Repository Managers (e.g. Artifactory), they can be used to retrieve an application and its dependencies and deploy it to a target environment. Since the result is consistent every time it runs, we can help ensure that the deployed application behaves in the same way every time.
Our Automation Server (e.g. Jenkins) also supports inclusion of the running of a configuration script within a job. This job could be kicked off by manual instruction, but the most powerful use is to add a step to the job we defined in our Continuous Integration process so that new releases which pass our tests are automatically deployed to an environment of our choice. When Continuous Integration (CI) is combined with Continuous Delivery (CD) like this, the resulting process is nicknamed ‘CICD’.
To where do we deliver?
As part of the Continuous Integration (CI) process we defined, our application has already successfully passed unit testing, integration testing and code quality checks, so we can be fairly confident about the quality of the application we’ve produced. Unfortunately ‘quite confident’ isn’t good enough; we need to be as close to certain as possible. The potential consequences if not could be dire (security breaches, data losses, … Armageddon):
9 Major Computer Bugs That Wreaked Havoc
iOS 11.0.1 update causes havoc for some iPhone and iPad users
We need to undertake further testing of our application in one or more test environments before we can consider delivering to production.
To keep things simple, we will assume that we have a single type of target production environment, but we will adopt the two-phase test environment approach as described in Software Delivery 101 (Environments and Applications):
- QA (or ‘Test’), followed by;
- Pre-Production (or ‘UAT’, ‘Staging’)
Importantly:
- To be meaningful, our test environments should be configured identically (or as closely as possible) to the target production environment. Using the same Configuration management script across the lot will help ensure this.
- For applications with multiple non-identical target production environments (for example, customer-run applications, like mobile apps which should support multiple versions of the Android operating system) then there should be test environments for each target production environment.
What additional tests do we run, when and how?
As a reminder of some of the takeaways from our previous article on Continuous Integration (CI):
- There are many different types of software testing, and they are not all mutually exclusive with one another (see diagram below)
- The testing needed will depend on the software project
- Unit and Integration testing should be conducted in the Integration environment as part of the Continuous Integration process
Automated testing in QA and the next steps
The natural progression to our existing tests is to undertake System testing — which looks at the system’s behaviour with respect to its business requirements and does so from an outsiders perspective, without caring about the inner workings of the underlying architecture or logic, making it a form of ‘black-box’ testing.
Nb.;
- Some systems should interact with multiple possible ‘clients’ e.g. web applications might support the various common web browsers (Firefox, Chrome, Microsoft Edge, …) and including different versions. Testing should be replicated across all such uses.
- A similar but sometimes more appropriate form of testing to System is End to End testing, which focuses on the user (which could be a person or a machine) and their journey. As such the test’s perspective covers the various systems which work together to produce the end effect.
We discussed previously that:
- Quick feedback loops from tests mean the value of changes can be realised sooner and allow businesses to respond more quickly to changing landscapes and opportunities
- Tests that can be written as code can be run automatically using test execution engines to maximise feedback quality and speed
So let’s do that!
D. Automated Test Execution Engine(s) for System Tests and other forms of testing
What does it do?: Sets up and executes automated tests against a software application, assessing and reporting on their results against expected behaviour. Automated tests provide feedback to developers within minutes, to accelerate delivery cycles and realise returns on investment sooner.
In contrast to lower lever tests such as Unit and Integration testing conducted during Continuous Integration (CI), many later stage tests treat the application as a black box and depend less on the language the software was written in and more on whatever interface it exposes (e.g. a Graphical User Interface or a set of RESTful Web Services). The key driver is instead the type of test being run, meaning a combination of test tools might be required.
Examples: Cucumber, Selinium, Apache JMeter
Our System or End to End tests are run on our QA environment. If any automated tests expose an unexpected result, then a notification should inform the development team who should make it a high priority task to address the issue so that continuous releasability (a key goal of CICD) can be restored.
Barring a good reason not to (see ‘Release planning’ below), releases which pass our automated tests in QA should be automatically promoted to the Pre-Production environment. Versions which reach this stage can rightfully be considered HOT-STUFF, and all triggered by a single commit from a developer!
The Pre-Production environment, as the name suggests, is the last stop before Production, and therefore the last chance to perform any other types of pre-release testing that may be desired. As well as re-running the Integration and System tests to ensure behaviour consistency between environments. there may be tests which were not possible in prior steps due to intentional / understood environmental differences (e.g. access to expensive Production-like hardware for performance testing, or access to a third party system not available elsewhere) which should be tested. As always, this should be with as much automation as possible, though manual testing is by no means a thing of the past: some types of test are impossible to put into code, such as showcases to stakeholders or usability studies — others might just be very difficult to codify!
As discussed in An introduction to DevOps a key part of CICD is putting the processes and infrastructure in place to ensure deployments minimise or eradicate downtime (e.g. by using a blue/green deployment process) release issues are identified quickly, and remedies to release issues can be rolled out just as speedily.
Eliminating manual steps and the strive for CICD perfection
Many organisations find themselves not quite able to relinquish control and continue to kick off deployments to production manually. The overall CICD pipeline in such a situation might look like the following:
Delivering applications into Production frequently, automatically safely is however the holy grail of CICD that relatively few manage to obtain. The automation of all steps, especially testing, is key to success:
“Without a doubt, automated test are the most important part of any CI/CD pipeline. Let me repeat that since it’s so important: without proper automated tests that run fast, have good coverage, and no erroneous results, there can be no successful CI/CD pipeline. It’s that simple.”
Credit: Test automation and Continuous Integration & Deployment (CI-CD)“ While [manual testing] does serve a distinct purpose and will always be needed for specific types of testing, such as exploratory testing, it can hinder an iterative delivery process. Time is of the essence in a continuous environment and manual testing can be painfully sluggish. The slower the feedback loops, the longer it will take to make adjustments and release software. Automation accelerates the build and deploy cycle by checking code and running tests continuously and providing feedback within minutes. This is why as much of the process as possible should be automated”
Credit: The key to an effective CI/CD pipeline: Automated Testing
The benefits of Continuous Delivery
We can now sit back in our chairs and marvel at all the benefits our continuous delivery processes can bring to our team:
- Accelerated time to market and better targeted products
Compared to release cycles that last months, in which features completed early on sit there dormant until release day, in CICD these features are released as soon as they are complete, delivering value instantly and receiving feedback, enabling the organisation to better tailor subsequent developments. - Improved quality, release reliability and customer satisfaction
Instead of the usual saga (dread, manual procedures, mystery priority 1 bugs and all nighters), releases are so common and so smooth they’re barely noticed. Everything is thoroughly tested, using consistent binaries and environments, and deployment is automated. - Improved productivity/efficiency
Because developers, testers and ops aren’t spending all their time setting up environments, deploying and finding, reporting and fixing bugs, they can all focus on delivering more value to the organisation.
Summary
In this article we explored how we can assure the quality of and accelerate the delivery of our software releases:
- covering 2 types of tool, providing Open Source examples for each:
G. Configuration Management
D (reloaded!): Test Execution Engine(s) for System Tests - describing how how these tools can be added to our technical architecture and be strung together in a CICD process flow
- discussing techniques for minimising downtime and recovering from issues quickly
- highlighting invaluable rewards such as business agility and risk mitigation
