A Docker image contains an application and all its dependencies. As it also contains the numerous binaries and libraries of an OS, it’s important to make sure no vulnerabilities exist in its root file system — at least no critical or major ones. Scanning an image within a CI/CD pipeline can ensure this additional level of security.
A Simple Project
Three stages are defined in this pipeline:
- The first one runs some tests on the Node.js code.
- The second builds a Docker image and publishes it in a GitLab registry.
- The last stage deploys the new image on Docker Swarm, using Portainer’s webhook feature.
TK *entrails* The Docker image built during the process is based on Nginx 1.14, and is deployed right away without any verification of its entrails. This could be dangerous so let’s see how we can improve that.
Adding an Image Scanning Stage to the Pipeline
This documentation from GitLab provided all the instructions to add an additional stage dedicated to image scanning. It basically runs a Clair server that provides the existing CVEs, and then the clair-scanner binary checks each layer of the image that was built during the previous stage of the pipeline.
I’ve added the additional content to the
.gitlab-ci.yml file of the project.
Note: I’ve only made some minor modifications to the stage defined in the documentation, so it fits the version of my GitLab runner (which is needed when it comes to the
With this new stage added, let’s see how it goes by running a new pipeline. The screenshot below shows part of the output of this new image_sanning stage.
We can see at the top, that each layer of the image is analyzed.
Note: if you want to know more about the internals of an image and explore each layer in a cool way, I highly recommend you have a look at dive.
A tool for exploring each layer in a docker image. Contribute to wagoodman/dive development by creating an account on…
Coming back to the GitLab output, we can see 100 vulnerabilities are detected. That’s a lot! Are they dangerous? Well, judging by some of the entries within the Severity column, Critical/High, they might be.
We Have a Lot of Vulnerabilities… Now What?
Well, unless you are a security person, chances are that those vulnerabilities won’t tell you much. Knowing an application has vulnerabilities can raise a lot of questions though:
- Am I safe to run the application as is?
- Which vulnerabilities can I ignore and which ones do I take seriously?
- My application is not user-facing, that should be safe to go with all those CVEs, right? Even for the Critical ones?
- Will my boss know if I just ignore all those CVEs?
- Are there any simple things I could do to mitigate the potential risks?
In some cases, there are actually some simple steps that can help a lot as we will see below.
Check the Base Image First
The example I’m considering is a simple website, published with the official
nginx:1.14 image — the last stable version at the date of writing this post. As the Dockerfile uses the multi-stage build, the final image does not contain a lot of unnecessary Node.js stuff used to build the web assets. The Dockerfile is the following one.
The multi-stage build is already a good starting point, as it helps to reduce the attack surface of the image. But still, the final
nginx:1.14 image is based on Debian, so let’s change that and use
Note: Alpine Linux is a small distribution, it’s focused on security and exposes a very small attack surface. Using a base image built from Alpine would probably be a good move — we’ll check this out.
The second part of our multi-stage Dockerfile is now replaced with the following.
Let’s trigger a new build and see how it goes.
Well, it went pretty good according to the screenshot above. The scanning stage now reports… 0 vulnerabilities!
Update: Using Aquasec Microscanner
Microscanner can scan the image in different ways: during the image creation or once it has been created.
Scanning while building the image
Some changes are required in the Dockerfile to add the microscanner binary and run it so it can analyse the image filesystem that has been built.
The 2 last instructions in this Dockerfile are dedicated to the scan:
- A token needs to be provided in the build instruction (the official documentation of Aqua Microscanner details how this token can be obtained).
- The microscanner binary is downloaded and run against the filesystem of the image beeing created.
Several additional options are used here:
--htmlgenerates a html report of the scan
--continue-on-failuredoes not return an error code in case CVEs are found, and then will not stop the pipeline of the CI tool
Scanning right after the image is built
This can be done adding a Dockerfile dedicated to the scan. An example of this Dockerfile is provided below.
FROM instruction specifies the name of the image that will be scanned. The 2 instructions that follow are the same as the ones used in the first approach (scanning while building the image).
To illustrate this second approach, we will add the following step in our CI pipeline.
Note: There is no point of having several image scans in the same pipeline, the purpose here is just to show how this one can be put in place and it can also help to make a choose between scanning solutions.
We can then trigger a new build of the image, Clair and Microscanner will both scan the image once it’s built.
Note: As this build is run a couple of weeks after the first version of this article, some CVE are found while they were not present before.
Interesting thing here: the results of the scan are different. 4 CVE are found by Clair while 5 are found by the Aqua Microscanner. This illustrates that CVE scanners do not work in the exact same way, for instance they might not operate on the same version of the CVE database, and might not work equally well.
Note: to make it even easier to use microscanner, we can use the Manager for Docker client plugins (CLIP).
whale: Docker Client Plugins Manager - build new plugins, publish them on Docker Hub or just try some plugins from the…
With this tool, no more need to use a dedicated Dockerfile for the scanning as this one can be triggered with a simple “docker microscan” command once the microscan plugin is installed.
This short post illustrates that adding a simple image scanning stage in an existing CI/CD pipeline is not that complicated. This provides additional information such as the number and IDs of the vulnerabilities an image contains.
From this information it’s still difficult to know what to do with them, but using a base image built from Alpine can then be a first step to enhance the security of an application, without knowing a lot about CVE.