Adding CVE Scanning to a CI/CD Pipeline

Luc Juggery
Dec 14, 2018 · 6 min read
Image for post
Image for post
Image Credit — https://cve.mitre.org/

TL;DR

A Docker image contains an application and all its dependencies. As it also contains the numerous binaries and libraries of an OS, it’s important to make sure no vulnerabilities exist in its root file system — at least no critical or major ones. Scanning an image within a CI/CD pipeline can ensure this additional level of security.


A Simple Project

In a previous article, I detailed how I set up a CI/CD pipeline on a very simple side project using GitLab, Portainer and Docker Swarm.

Three stages are defined in this pipeline:

  • The first one runs some tests on the Node.js code.
  • The second builds a Docker image and publishes it in a GitLab registry.
  • The last stage deploys the new image on Docker Swarm, using Portainer’s webhook feature.

TK *entrails* The Docker image built during the process is based on Nginx 1.14, and is deployed right away without any verification of its entrails. This could be dangerous so let’s see how we can improve that.


Adding an Image Scanning Stage to the Pipeline

There are several image scanning solutions; commercial and open source. In this article we’ll go with Clair and clair-scanner; two open source tools.

This documentation from GitLab provided all the instructions to add an additional stage dedicated to image scanning. It basically runs a Clair server that provides the existing CVEs, and then the clair-scanner binary checks each layer of the image that was built during the previous stage of the pipeline.

I’ve added the additional content to the .gitlab-ci.yml file of the project.

Note: I’ve only made some minor modifications to the stage defined in the documentation, so it fits the version of my GitLab runner (which is needed when it comes to the artifact upload).

With this new stage added, let’s see how it goes by running a new pipeline. The screenshot below shows part of the output of this new image_sanning stage.

Image for post
Image for post
image_scanning stage of the GitLab pipeline

We can see at the top, that each layer of the image is analyzed.

Note: if you want to know more about the internals of an image and explore each layer in a cool way, I highly recommend you have a look at dive.

Image for post
Image for post
Running dive on the image built in the CI/CD pipeline

Coming back to the GitLab output, we can see 100 vulnerabilities are detected. That’s a lot! Are they dangerous? Well, judging by some of the entries within the Severity column, Critical/High, they might be.


We Have a Lot of Vulnerabilities… Now What?

Well, unless you are a security person, chances are that those vulnerabilities won’t tell you much. Knowing an application has vulnerabilities can raise a lot of questions though:

  • Am I safe to run the application as is?
  • Which vulnerabilities can I ignore and which ones do I take seriously?
  • My application is not user-facing, that should be safe to go with all those CVEs, right? Even for the Critical ones?
  • Will my boss know if I just ignore all those CVEs?
  • Are there any simple things I could do to mitigate the potential risks?

In some cases, there are actually some simple steps that can help a lot as we will see below.


Check the Base Image First

The example I’m considering is a simple website, published with the official nginx:1.14 image — the last stable version at the date of writing this post. As the Dockerfile uses the multi-stage build, the final image does not contain a lot of unnecessary Node.js stuff used to build the web assets. The Dockerfile is the following one.

The multi-stage build is already a good starting point, as it helps to reduce the attack surface of the image. But still, the final nginx:1.14 image is based on Debian, so let’s change that and use nginx:1.14-alpine instead.

Note: Alpine Linux is a small distribution, it’s focused on security and exposes a very small attack surface. Using a base image built from Alpine would probably be a good move — we’ll check this out.

The second part of our multi-stage Dockerfile is now replaced with the following.

Let’s trigger a new build and see how it goes.

Image for post
Image for post

Well, it went pretty good according to the screenshot above. The scanning stage now reports… 0 vulnerabilities!


Update: Using Aquasec Microscanner

As Łukasz Lach pointed out in the Twitter feed below, Aquasec offers its own CVE scanner: Aqua Microscanner, and it’s very easy to use. Thanks Łukasz !

Image for post
Image for post

Microscanner can scan the image in different ways: during the image creation or once it has been created.

Scanning while building the image

Some changes are required in the Dockerfile to add the microscanner binary and run it so it can analyse the image filesystem that has been built.

Dockerfile updated to add the scanning step

The 2 last instructions in this Dockerfile are dedicated to the scan:

  • A token needs to be provided in the build instruction (the official documentation of Aqua Microscanner details how this token can be obtained).
  • The microscanner binary is downloaded and run against the filesystem of the image beeing created.

Several additional options are used here:

  • --html generates a html report of the scan
  • --continue-on-failure does not return an error code in case CVEs are found, and then will not stop the pipeline of the CI tool

Scanning right after the image is built

This can be done adding a Dockerfile dedicated to the scan. An example of this Dockerfile is provided below.

Dockerfile.microscanner

The FROM instruction specifies the name of the image that will be scanned. The 2 instructions that follow are the same as the ones used in the first approach (scanning while building the image).

To illustrate this second approach, we will add the following step in our CI pipeline.

Microscanner step in the .gitlab-ci.yaml

Note: There is no point of having several image scans in the same pipeline, the purpose here is just to show how this one can be put in place and it can also help to make a choose between scanning solutions.

We can then trigger a new build of the image, Clair and Microscanner will both scan the image once it’s built.

Image for post
Image for post
Result of Clair scanner
Image for post
Image for post
Results of Aqua Microscanner

Note: As this build is run a couple of weeks after the first version of this article, some CVE are found while they were not present before.

Interesting thing here: the results of the scan are different. 4 CVE are found by Clair while 5 are found by the Aqua Microscanner. This illustrates that CVE scanners do not work in the exact same way, for instance they might not operate on the same version of the CVE database, and might not work equally well.

Note: to make it even easier to use microscanner, we can use the Manager for Docker client plugins (CLIP).

With this tool, no more need to use a dedicated Dockerfile for the scanning as this one can be triggered with a simple “docker microscan” command once the microscan plugin is installed.


Summary

This short post illustrates that adding a simple image scanning stage in an existing CI/CD pipeline is not that complicated. This provides additional information such as the number and IDs of the vulnerabilities an image contains.

From this information it’s still difficult to know what to do with them, but using a base image built from Alpine can then be a first step to enhance the security of an application, without knowing a lot about CVE.

Better Programming

Advice for programmers.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store