Secure Your Containers Now! with Automated Docker Build Pipelines

Secure container images from the ground up with automated image builds.

Photo by Michael Dziedzic on Unsplash

Container image platforms, like Docker Hub, have made it easier than ever for software developers, cloud-native engineers and DevOps teams to create and share fully bundled images of their applications.

A recent study of over 200 IT Leaders by CapitalOne shows that 86% wanted to “use containers for more applications” across the enterprise; with their key drivers being compliance and portability.

Given the rapidly increasing adoption of this paradigm, containers have landed at the heart of many environment’s security profiles. Therefore, ensuring the contents of these containers are free from security vulnerabilities has become a critical exercise for maintaining a secure container infrastructure.

In this article, we will build a fully automated CI/CD pipeline that produces a Docker image for our application. We’ll focus on a combination of tools and methods that help us to produce a secure image that is both well tested and free from known security vulnerabilities.

What will I need?

We will use the following components to assemble our secure container image build pipeline:

• A GitHub Repository for holding your Dockerfile and pipeline code.
• GitHub Actions for building our automated CI/CD workflows.
• A local Docker or Docker Desktop running on your computer. This is optional but useful for testing images as you go.
• Terminal commands for Linux or macOS. If you’re on Windows, I recommend WSL2 with Ubuntu or similar.
• GNU Make we will use a Makefile to execute our pipeline commands.

We’ll also be using some awesome Open Source tools that are called via GitHub Actions plugins. These include:

• Goss Quick and Easy server validation https://github.com/aelsabbahy/goss/
• Hadolint Dockerfile Linter https://github.com/hadolint/hadolint
• Grype Vulnerability scanner for container images https://github.com/anchore/grype

What are we making?

The following diagram shows the automation workflow we will build in GitHub Actions to produce our secure Docker images:

Secure Docker Image Build Pipeline

Let’s get started!

All of the code for this exercise can be viewed below, or downloaded from the following GitHub repository:

https://github.com/obrientimothya/tutorial-secure-docker-image

Step 1. Dockerfile

In this exercise, we will build a docker image that contains some useful Terraform tools, as well as the Goss server validation tool that we will use later for testing.

Create the following Dockerfile at the root of your repository:

/Dockerfile

As you can see, we are using version Alpine 3.10.6 as our base image, with the Terraform and Goss versions controlled by ARG variables.

Note that we are including our alpine:3.10.6 image using the sha256 reference method. Image tag values can be overwritten by upstream developers, therefore they do not provide a guarantee that the contents of a version won’t change over time, or be interfered with my a malicious actor.

A more secure method is to provide a sha256 digest reference. This ensures that the exact same image is pulled every time, regardless of upstream tag changes.

Step 2. docker-compose.yaml

Create the following ‘docker-compose.yaml’ file in the root of your repository. This will be used to call the Hadolint syntax linter:

/docker-compose.yaml

Step 3. Hadolint Configuration File

In the root folder, create the following ‘.hadolint.yaml’ configuration file to control Hadolint’s parameters:

/.hadolint.yaml

This step forms part of our Secure Configuration. Note that we are only allowing docker to pull from Trusted Registries. For this exercise, we will allow docker to pull images from docker.io and ghcr.io.

Limiting the registries that docker has access to will prevent accidentally pulling base images from unapproved sources. For instance, if you have an internal docker registry in your environment (such as JFrog Artifactory) you should set the trustedRegistries to allow it only.

Step 4. Makefile

Our Makefile will be used to execute the commands required by our pipeline.

The Makefile defines the three stages, ‘Lint’, ‘Build’ and ‘Test’ that we will trigger with GitHub Actions in the next steps.

Create the following Makefile in the root folder:

/Makefile

Step 5. Goss Tests

Create the folder structure ./tests/goss.d

Add the following ‘goss.yaml’ file to the ‘./tests’ folder:

/tests/goss.yaml

Add the following file to the ‘./tests/goss.d’ folder:

/tests/goss.d/goss_files.yaml

Review ‘goss_files.yaml’.

Here, we are running tests to ensure that the binaries inside our image match an expected SHA hash. This security measure ensures that any binaries downloaded during the docker build phase are valid.

If a malicious actor modifies any of the binaries upstream, these tests will fail and prevent us from incorporating unknown content into our image.
Awesome, right!

Step 6. GitHub Actions

Create the folder structure .github/workflows and add the following to ‘.github/workflows/GHA01-pull-request.yml’ :

.github/workflows/GHA01-pull-request.yml

Have a read of the pull-request file. We are defining the ‘Lint’, ‘Build’, ‘Test’, ‘Scan’ and ‘Push’ stages that will happen automatically whenever a developer raises a pull request:

• Lint — Executes make lint to run the Hadolint linter to check the syntax of our Dockerfile.
• Build — Builds an image based on our Dockerfile
• Test — Executes make test to perform the Goss tests we defined above to ensure that the image contains the exact versions of assets we expect.
• Scan — Performs a Security Scan for any known vulnerabilities that exist in our image. Uses the ‘anchore/scan-action@v2’ GitHub Actions plugin to run the Grype scanner.
• Push — Publishes a test version of the image to the GitHub Container Registry and adds a comment to our pull-request with the details of the image.

Now, create the ‘.github/workflows/GHA02-push-main.yml’ :

The push-main workflow is automatically triggered when a passing pull-request is merged to the ‘Main’ branch. The workflow here is much the same, except for a further ‘Tag’ step:

• Tag — Adds a Git Tag to the main branch and creates a new Release for the final secure image.

Ready to Run!

Let’s get secure!

1. In your GitHub repository, raise a Pull Request to trigger GitHub Actions.
2. Under the Actions tab, open the job for your pull request, similar to below:

pull-request actions with failing scan

At this point, you will see the overview of our secure build workflow: Build and Lint, Scan and Test, and Push. Also, the page shows us any Annotations (errors) and Artefacts (our built docker image called ‘secureimage’)

As you can see, our ‘J04-scan’ job is failing. Annotations indicate that our scan discovered security vulnerabilities in our image (Oh, no! Danger! We’d better fix these in the next section!)

Photo by Kelly Sikkema on Unsplash

Security Reporting

In the previous section, we saw that our Scan job had failed. To see more detailed reporting on the exact security vulnerabilities found, GitHub can display the details of the ‘results.sarif’ that we uploaded during the pull-request job:

1. In your GitHub repo, open the ‘Security’ tab, then choose ‘Code scanning alerts’
2. In the ‘Filters’ input, enter is:open pr:<number> where <number> is the ID number of your Pull Request from the previous section.
3. Code Scanning shows the detail of the CVE’s found, per the image below:

Code scanning alerts report

Excellent! At this point we now have a working build pipeline that fails when it detects security vulnerabilities in our packages.

We also have detailed security reporting that points us to the source of any found vulnerabilities so that we can patch them before they impact our operations.

Patching Vulnerabilities

The security report showed vulnerabilities in some of the packages in our base Alpine 3.10.6 image. This release of Alpine is quite old, so we should start by trying to upgrade it to a newer release:

1. Edit the Dockerfile in your repository.
2. Remove the line starting with FROM alpine@sha256abd435...
3. Replace it with FROM alpine:3.14 (note: please use the most current Alpine version from Docker Hub as this may change by the time you read this)
4. Save and Commit, then push to the pull-request with your new Dockerfile to re-run the pipeline:

passing pull-request pipeline

Success! The pipeline is now passing and is free from known security vulnerabilities.

Photo by Mika Baumeister on Unsplash

Some Other Things for you to Explore…

• Check the comments section of your Pull Request. You’ll notice that GitHub Actions has posted a link to your new docker image so that you can test it in your environment - such as locally via docker or in a Kubernetes configuration.

link to testing image

• Once all your checks pass, you can merge your pull request into the ‘Main’ branch. This will generate a new tagged release of your image, so that you can consider it for use in production:

release generated on merge to Main

• Explore the Hadolint and Goss tools in more detail to understand their value.
• Consider how you might schedule your security scans to ensure your latest code remains secure?

Thanks!

Security is a critical focus for automated systems. By incorporating security awareness and reporting directly into our container build pipelines, we saw how we could catch and remediate security vulnerabilities as early as possible in the containerisation life-cycle.

Reach out to hello@momenton.com.au or send us a message on LinkedIn to understand how Momenton can help your organisation and how we can help mature delivery of your products.