Level up Your CI/CD Pipeline with Pull Request Deployments
Leverage your infra-as-code to the max to give stakeholders a demo environment of your work in progress. Using Serverless Framework, AWS CloudFormation and Azure DevOps Pipelines.
Feature branches
The pull request (PR) model is a powerful collaboration method. When working with feature branches, the pull request can capture the discussions with your fellow developers around the code (code review), while automated checks from your CI pipeline (such as unit tests or code quality scans) give you fast feedback on your changes and prevent you from deploying broken software to the master branch.
In my team, when a pull request is merged to master, the code is automatically deployed to our test environment and then, after a manual approval step, deployed to production.
However, the pull request — and its focus on code — is usually seen as an engineering tool, whereas, when you work in a multi-disciplinary team, you want to take feedback from, say, UX Designers into account as well, before you call a feature “done”. In my experience it could happen that right after a feature is merged and is deployed to a test environment, feedback comes in from designers or business stakeholders that require rework that ends up in a new PR. This is inefficient. The delay you get in processing feedback might even tempt you in dropping pull requests altogether.
Wouldn’t it be great if one could give UX Designers and stakeholders a preview environment of their pull request? Such that…
- stakeholders are involved and engaged early in the process
- the moment on which you merge pull requests becomes more aligned with your actual Definition of Done
- having a preview environment even enables you to run an automated end-to-end test in your pull request, increasing your confidence before merging, preventing ‘rework’ pull requests
This article shows how you can achieve this.
Serverless & CloudFormation
To allow for pull request deployments, a prerequisite is that you fully rely on infrastructure as code, so that you are able to deploy a new environment from scratch. In our case we deploy our web application using a combination of the Serverless Framework and AWS CloudFormation. Each pull request then simply is another stage (in the terminology of the Serverless Framework) next to existing stages, such as ‘test’ and ‘prod’.
An important consideration is the speed of your deployment. Pull requests are all about short feedback cycles. Examine your infrastructure setup and try to make things you don’t need for a preview environment optional.
As an example, in our test and production stages we deploy our web app to an S3 public website and serve it through a CloudFront distribution. Deploying a new CloudFont distribution takes at least 20 minutes, so that’s a step we want to skip for pull request environments. An easy way to do this, is to make CloudFront depend on an environment variable that you can control in your pipeline.
Pipeline
In this example we use Azure DevOps. This is a picture of the end result.
Here, the deployments to Test and Production are existing deployment jobs. The Build stage already packages the code into an artifact that is used inside those deployment jobs. We will add an extra stage for pull requests.
Extracting your deployment code
If you haven’t already, extract the deployment step in your pipeline to a template, where the stage name (Test, Production, or a pull request ID) is a parameter.
Adding a stage for pull requests
For the existing deployment stages we use deployment jobs that automatically download the artifact. However, in this case, using a deployment job would mean that Azure DevOps would keep track of a new environment in the project for every new PR, which would soon create a mess.
Therefore, we use an ordinary job, and specifically tell Azure DevOps to download the artifact. The code snippet below demonstrates how we use pr-$(System.PullRequest.PullRequestId) to generate a new Serverless stage name for every PR.
If all goes well, the pipeline greets you with an URL to your preview environment: Visit your web app at http://my-webapp-pr-565.s3-website-eu-west-1.amazonaws.com.
Adding end-to-end tests
It is now easy to add extra steps to run an end-to-end suite against this new deployment. In the following snippet, we add extra steps to first seed our database with test data, then run our Cypress test suite, and, finally, save the test result in Azure DevOps, including a video of any failed test cases.
Clean-up
At this point we have working preview environments and end-to-end tests in our pull requests, but there is one challenge we haven’t touched upon yet: the clean-up. If you don’t deal with this, then it’s likely you will soon reach a resource limit in your AWS Account (such as the default of 100 S3 buckets) and doing manual cleanup at that point is a nightmare (you can tell I speak from experience 😉). Leave alone any costs you might incur!
As a solution I created a NodeJs script that performs the following steps:
- Request a list of all pull requests inside the project from the Azure DevOps Git API.
- Filter the list on open pull requests.
- Request the AWS CloudFormation API for all stacks, excluding the ones that have an IN_PROGRESS state.
- From the list, remove the stacks that belong to open pull requests. (Note that each stack name includes the PR number as the Serverless stage.)
- For each remaining stack, run
serverless -v —-stage $stage --forcewhere stage refers to the corresponding pull request.
Create a standalone Azure DevOps pipeline that runs this script on a daily basis using a cron schedule:
In practice, we had to face one final hurdle. It’s very likely that your infrastructure code specifies deletion policies, so that resources are retained when you delete your CloudFormation stack. This prevents you from accidentally losing production data. This also means that the clean-up script will not remove all resources belonging to your preview environment and, therefore, still leave a mess in your account. Another problem is that CloudFormation will fail to remove S3 buckets that are not empty.
To solve this problem, I wrote a Serverless plugin that detects PR stages and hooks into two lifecycle events:
- On packaging, remove all deletion policies from the generated CloudFormation code.
- On remove, first empty any S3 buckets.
That’s it!
By using an Azure DevOps pipeline and leveraging our infrastructure code, built upon the Serverless Framework and AWS CloudFormation, we were able to automatically deploy preview environments for our pull requests. This allowed us to automatically run our end-to-end test suite against pull requests, resulting in earlier feedback, and to share a preview URL with our designers and product owner.
About Erik
Erik works as a Full Stack Engineer at NN, a financial services company active in 18 countries and the leading insurance company in the Netherlands. In the Innovation department he works on developing new technology for innovations in corporate startup fashion. Engineering topics Erik likes are CI/CD, domain-driven development and test-driven development. He obtained his MSc (Computer Science) from Delft University of Technology where he graduated cum laude on the topic of re-engineering. Interested in working at NN? Check the NN careers website.
