Moving towards Continuous Delivery

by Bethan Palmer and Mat Moore (Software Development Profession)

In LAA digital, we are moving towards Continuous Delivery as part of modernising CCMS (Client and Cost Management System). CCMS is one of several legacy services we are modernising in the MOJ. So far we have been focusing mainly on one part of CCMS, the Provider User Interface.

Continuous Delivery is the ability to get changes of all types — including new features, configuration changes, bug fixes and experiments — into production, or into the hands of users, safely and quickly in a sustainable way. https://continuousdelivery.com/

Historically, the deployment process for CCMS has involved a lot of steps, and a lot of different people need to be involved throughout the process to get a change from development to production.

When a release process is complex and time consuming, releases happen infrequently. Because they happen infrequently they are risky as they contain a lot of changes. And because they are risky they tend to happen even less often. So there becomes a cycle of infrequent releases which is difficult to break out of. We wanted to avoid this situation by moving towards frequent deployments to production.

Things that we want to change

The bottlenecks in the deployment process were:

• Deployments needed to be planned, and required approval from various people to proceed
• We needed to do a lot of manual testing, and our case workers or the live support team also manually smoke tested each deployment to make sure we hadn’t broken anything
• We needed to run separate builds for each environment we deploy to, instead of building a single artefact and deploying it to each environment in turn
• We had to develop the code in a shared development environment, which was firewalled off from the internet and difficult to work on

We are working to remove these bottlenecks so that we can deploy to production quickly and easily. Our end goal is to promote changes from the development environment through to production in less than 15 minutes.

What we’ve done so far

Below are some of the changes we’ve made which are helping us move towards Continuous Delivery.

Updated the unit tests

There were already some existing unit tests, but several of them contained hardcoded paths to configuration files we didn’t have access to. We re-wrote some of these tests, removed some of them, and wrote some new unit tests to increase our test coverage.

Added CircleCI (continuous integration)

We added an integration with CircleCI, which runs our unit tests every time we create a Pull Request in GitHub. It also checks our code’s dependencies for security vulnerabilities every night. Eventually we want to automate deployments from here too.

Written some end to end tests

We’ve written some browser tests that automate someone clicking through the application. Running these gives us confidence that our changes haven’t broken any of the basic functionality.

Improved the application build process

We have replaced the old Ant build with a new Gradle build. Ant and Gradle are both build automation tools for Java applications, but Gradle has several advantages over Ant. For example with Gradle we are able to delete checked in dependencies and use remote ones, making it much easier to upgrade them. Gradle is a simpler tool for developers to use and the build process is quicker, which makes developing code changes easier.

Created a Docker/Tomcat build we can run locally

We are now able to build the application and deploy it to Tomcat running in a Docker container. This means we can develop and test our changes locally instead of relying on the shared development environment, which is very difficult to use. This gives us a much quicker feedback loop when doing development work.

Spiked out what our ideal deployment pipeline into AWS might look like

We did a spike to demonstrate how our end goal might work with a simple Hello World application. This involved creating a pipeline in CircleCI that builds the app, tests it, and deploys it to multiple environments in AWS with zero downtime. We also want to look at spinning up ephemeral environments that get deleted after a certain amount of time. Another department in the MoJ, the Office of the Public Guardian, had already created a similar deployment pipeline so we were able to use their work as a guideline for ours.

Next steps

Migrate to AWS

CCMS could not be included in our previous migration to AWS. Now that we have a better understanding of the process, we intend to repeat the same steps to move CCMS out of the current data centre and deploy into public cloud. This will reduce the cost of running the service, but will also allow us to run fully automated deployments. This enables us to make changes more quickly and safely.

Increase our test coverage

Our test coverage is still very low (6% for the Provider User Interface). We want to increase this so that we have less reliance on manual testing, and can be confident we are releasing changes safely.

Adopt Continuous Deployment

When we have achieved Continuous Delivery, the next step is to move to Continuous Deployment (where the release is automated from CircleCI rather than a manual process, and there is zero downtime).

If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can follow us on Twitter, read our other blog or check us out on LinkedIn.

If you’d like to come and work with us, please check current vacancies on our job board!