Dockerized Dependency Check: Building NVD image
Motivation
In a previous article we discussed how we are doing our best to ensure that our software does not contain any known security vulnerabilities. For this purpose we use an OWASP Dependency Check, which is integrated as a Gradle plugin to our CI/CD pipeline.
After releasing the original article, we heard back from many readers who wanted to know more about how we integrated/implemented this security process. In fact there was one commonly asked question; How long does the analysis take? To elaborate on this we have prepared two follow-up articles. In this article we will answer the question How long does the analysis take? and discuss why the default behavior of the Dependency Check is not satisfactory and why we decided to mitigate potential issues by Dockerizing the National Vulnerability Database (NVD). Then we will demonstrate the creation of the database image using Gradle.
In the second article we will learn how to use our new image with the Dependency Check plugin without any additional configuration being required by the developers working on our applications.
How long does the analysis take?
So back to the point, how long does it take? This is an excellent question, because — if it was too slow — the teams would tend to save their time by postponing the analysis to the last possible moment, going directly against the preferred modus operandi: continuous integration. Also if the length of the analysis was not predictable, it would create unnecessary friction — who likes builds which sometime takes 7 seconds… and sometimes 7 minutes?
With our solution we have consistently seen analysis take 15–30 seconds.
How does the Dependency Check work?
Before we go deeper into our implementation, let’s just quickly cover how the Dependency Check plugin (for Java) works:
- Gradle resolves a list of dependencies
- The Dependency Check stores the evidence in Lucene index. Evidence comprises of a resolved list of dependencies + contents of their manifest and pom.xml files
- The Dependency Check downloads the NVD Database (which contains a list of known software vulnerabilities) and materializes it as a SQL database
- The Dependency Check matches the contents of the Lucene index against the SQL database to find out if any of the dependencies has a known vulnerability
The time
Steps 1, 2 and 4 are relatively fast — 5 to 15 seconds in total depending on the size of the project. That’s not an issue and its also a relatively stable/predictable number per project. On the other hand, step 3 (download NVD database and materialize it in SQL) is a different beast entirely.
The initial load, executed with first build on every machine, of the NVD dataset (vulnerabilities since 2002) and insertion to local H2 database takes 3–7 minutes. The resulting database is then cached in the Gradle cache, hence subsequent builds should not suffer from this delay. In reality we found that the cache is being rebuilt from scratch on a weekly basis.
It can be clearly seen that the default properties of the Dependency Check were a bit problematic for us, as it made the build times unpredictable — and frustrated the developers. The other, less important downside was that the analysis may occasionally require an internet connection.
The solution
The Dependency Check offers out-of-the-box the possibility of having a central corporate vulnerability database, which is centrally managed and all instances of the build plugin connect to this NVD instance. This is a great solution, if you are ok with the fact that from now on your builds will require a VPN connection. This solution also requires that the centrally managed database is available at all times(HA). Lastly, you are not able to easily “freeze” the state of the database for a given project (to allow repeatable builds).
Based on these limitations we have decided to go in a slightly different direction — to periodically build a Docker image with a H2 database containing the full NVD dataset. The image is built every week by our CI/CD engine, thereafter the individual clients/builds just need to download the latest image. In case the developer is offline, he may use the image from the previous week for local development (without any configuration change). This approach eliminates the need to have a centrally managed and highly available(HA) database.
NVD Container
Now to the code. First we need to build the Docker image. For this task we will use a Gradle build tool and the gradle-docker-plugin by Benjamin Muschko. Usage of Gradle may seem to be overkill, but keep in mind that this is a simplified version of the whole pipeline. In the real world, we also run a battery of smoke tests (written in jUnit) to verify that the image can be used in the wild. We also use the already mentioned gradle-docker-plugin to tag the images and push them to our Artifactory.
In the plugins section, we define the plugin we need to use. Aside from the Docker plugin, we need the Dependency check for Gradle which performs the database synchronization task — which is configured in the dependencyCheck section of the build.
dependencyCheck extension
In the dependencyCheck extension, we specify that the plugin should always make an attempt to load the most recent version of the NVD database. This is technically only a concern when building on a local machine, as the database will be stored in the nvd-container-resources directory set in the directory assignment, but just to be on the safe side it is better to make an attempt to update the dataset.
Once Gradle executes the task called dependencyCheckUpdate, it will download the NVD dataset, start up an instance of the H2 database and it will load it with the NVD data. The datafile called odc.mv.db will then be written to the nvd-container-resources directory as we have configured.
task copyEntryPoint
The next task copyEntryPoint does exactly what you would expect. Here is an example of the code from the entrypoint.sh:
In our case, it is a trivial implementation of the best practice provided by Docker documentation.
task createDockerFile
This is the part that does the bulk of the work. Firstly, we inherit the OpenJDK8 container provided by the AdoptOpenJDK project and set the location of the DockerFile generated by this task. The actual job starts with the first invocation of the runCommand, where we download and extract the current release of the H2 database. To make handling of the H2 db version easier, we just create a simple symlink of *jar → h2.jar. Last but not least, we copy the data file from the NVD database created by dependencyCheckUpdate to the image.
Now we just need to set the entry point with a proper default command, set the exposure of ports of the H2 database and we are done.
task buildImage
The buildImage task just wraps up our work by invoking the Docker binary, which compiles our image. The only thing we do here is that we assign the image a human friendly reference nvd-container-gradle.
Summary
In this article we have shown and discussed that default behavior of OWASP Dependency Check is slightly problematic because the NVD database build can take up to several minutes to run. To mitigate unstable build times, we have re-laid the foundations of the solution by building a Docker image of the fully populated database. This ensures consistent build times and allows for continuous integration to be implemented by all teams.
Stay tuned for the next article, where we will learn how to integrate our Dockerized version of the database with the Dependency Check Gradle plugin.
Source code
Get access to the full source code for the plugin and its usage in a showcase project in this GitHub repository.
