The chains we wear
--
A friendly reminder about the pains of container supply chain security
“Containers are great. Containers are the present and the future. They make my development so much easier. They are composable and reproducible. And Dockerhub is full of useful container images that I can download and use. Long live containers!”
“Yeah, they are cool. But images’ supply chain security is not trivial. How are you managing it?”
“I’m sorry, supply what?”
“You aren’t just downloading images from Dockerhub and building upon them for your production workloads without reviewing them, right?”
“…”
The supply chain of a container image
Container images are composed of layers. When you build a Dockerfile, you receive the layers of the parent image and stack new ones for every change you introduce. Your software supply chain includes all the image ancestors plus the software that you install yourself. And take into consideration that, depending of the image, it may have multiple different grandparents.
If an image A is built based on image B, image B is based on C, and C in D, there may be a lapse of years between the creation of A and D. Truly. Years. And D contains the base Operative System, so the situation can get pretty nasty if packages aren’t updated diligently.
Talking about Operative System packages, let’s review the steps needed to receive a security update for an OS package inside a container image in production. Let’s assume that your image is based on some official runtime image, like python:3.8, which is based on Debian:
- Vulnerability is detected in software X and reported.
- Maintainers of X release a patch of the vulnerability.
- The Debian maintainers of the package X update the package with the patched version.
- Debian container image is built with updated package, or downstream Dockerfiles (
python:3.8or your own) upgrade packages explicitly. - The
python:3.8image is rebuilt using a patched Debian base image. - You rebuild your image using a patched version of the
python:3.8image. - You deploy your patched image.
Needless to say, that if any of the links in this chain fail, the final image gets unpatched. Also, points 5 and 6 may happen several times if there are many nested parent-child images. Fun, indeed.
So, when you download a public image from Dockerhub and use it as-is, you are implicitly accepting that you may never see a security update again (or at least, not in a timely manner).
And we all have blindly downloaded public images from Dockerhub.
About CVE scans
For the rest of the article, I’ll compare container images using the number of present vulnerabilities. This is obtained using the Trivy CVE scanner like this:
$ trivy image python:3.82022-07-18T14:29:55.976+0200 INFO Vulnerability scanning is enabled
2022-07-18T14:29:55.976+0200 INFO Secret scanning is enabled
2022-07-18T14:29:55.976+0200 INFO If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-07-18T14:29:55.976+0200 INFO Please see also https://aquasecurity.github.io/trivy/v0.29.2/docs/secret/scanning/#recommendation for faster secret detection
2022-07-18T14:29:57.605+0200 INFO Detected OS: debian
2022-07-18T14:29:57.605+0200 INFO Detecting Debian vulnerabilities...
2022-07-18T14:29:57.735+0200 INFO Number of language-specific files: 1
2022-07-18T14:29:57.735+0200 INFO Detecting python-pkg vulnerabilities...python:3.8 (debian 11.4)
========================
Total: 1156 (UNKNOWN: 0, LOW: 539, MEDIUM: 240, HIGH: 335, CRITICAL: 42)...
I know that this may incite some heavy breathing in security savvy folks, so let’s add a disclaimer:
CVE scans are only a small part of securing a software system, and by no means can be used as the only metric to decide whether something is safe or not. CVE scans rely on databases of known vulnerabilities, so they offer no information about the resilience of the system against zero-day vulnerabilities; they have false positives and false negatives; and they lack context about whether a vulnerability could actually ever be triggered in the specific setup and its severity.
That being said, CVE scans are a pretty nice heuristic for simple comparisons. If two images contain the same software that I need, but image A has 1000 High or Critical CVEs and image B has 10, I’ll take a look at B first, thank you.
Again, the 10 CVEs in image B can be way more dangerous in my scenario than the 1000 of A, but at least I can manually review them and take countermeasures.
At the end of the day, heuristics don’t need to reflect reality, they just need to be useful to make decisions and save efforts.
However, CVE scans may become a burden if you start scanning an already built project and find yourself surrounded by hundreds or thousands of vulnerabilities. At that point the numbers become useless noise, cause no one in their right mind would spend work time analysing them one by one to find the root cause and (if any) solutions.
But even though your image’s CVE count is over the roof, there is still hope. Most probably, you will be able to greatly diminish the problem by choosing a better base image to build upon.
Content warning
The rest of the post contains some pretty snarky (some could say even aggressive) remarks about some free software and open source initiatives.
Let me assure you that I have the utmost respect towards the free software and open source movements, as well as their members and their tireless efforts. Their work since the last decades of the last century has stablished the foundations of our current world. If you can, please contribute in any way you can.
Now, it’s time to bully some maintainers.
The mighty Official Python images
Let’s get our hands dirty and analyse some of the most downloaded container images in the world: the Official Dockerhub Python images.
Imagine that you have been building for a while a gorgeous Flask web app based on the official Python 3.8 image and you’ve had quite a jumpscare when running Trivy on it.
How can it contain more than 1000 vulnerabilities? And what can you do about it?
Let’s begin by taking a look at the underlying Operative System of the image, and then we’ll move to the actual Python images.
OS image comparison
The official Python images are maintained by the Docker community and come in different flavours depending on the underlying OS. In the list of available image tags we find:
- Debian 10 Buster
- Debian 10 Buster Slim
- Debian 11 Bullseye
- Debian 11 Bullseye Slim
- Alpine 3.15
- Alpine 3.16
All of these Operative System versions are currently in their active security maintenance phase, so should be receiving patches regularly. Also, the “slim” versions of Debian are like their normal counterparts but with a couple less packages installed.
You’d expect no big security differences between consolidated and actively maintained operative systems like these. However, let’s see the Trivy results on each of the individual OS images (no Python yet):
+======================+=====+========+======+==========+=======+
| Image | Low | Medium | High | Critical | Total |
+======================+=====+========+======+==========+=======+
| debian:buster | 86 | 12 | 29 | 9 | 136 |
+----------------------+-----+--------+------+----------+-------+
| debian:buster-slim | 84 | 11 | 29 | 9 | 133 |
+----------------------+-----+--------+------+----------+-------+
| debian:bullseye | 61 | 0 | 13 | 3 | 77 |
+----------------------+-----+--------+------+----------+-------+
| debian:bullseye-slim | 61 | 0 | 13 | 3 | 77 |
+----------------------+-----+--------+------+----------+-------+
| alpine:3.15 | 0 | 0 | 2 | 0 | 2 |
+----------------------+-----+--------+------+----------+-------+
| alpine:3.16 | 0 | 0 | 2 | 0 | 2 |
+----------------------+-----+--------+------+----------+-------+Seems like Debian Buster is not being taken care as much as the more recent Debian Bullseye. The usual fight between siblings for the love of their package maintainers, I guess. Also, the “slim” versions only improve size, not safety.
On the other side, Alpine’s results speak for its reputation and are almost clean, except from a couple of recently reported vulnerabilities (see, since there are only 2 vulns I can actually read them).
Just out of curiosity, let’s check another Debian based operative system, Ubuntu 20.04 LTS, to see how it endures the test:
+======================+=====+========+======+==========+=======+
| Image | Low | Medium | High | Critical | Total |
+======================+=====+========+======+==========+=======+
| ubuntu:20.04 | 19 | 7 | 0 | 0 | 26 |
+----------------------+-----+--------+------+----------+-------+Oh, wow! That was unexpected. Apparently it is actually possible to patch Debian’s vulnerabilities and offer an empty OS base image without High or Critical vulnerabilities. The more you know.
Looking at the graph, we see that, paraphrasing Orwell, all OS images are equal, but some OS images are more equal than others.
The first time I ran these tests some months ago I thought that the reason for the differences must be the release cadence of the images. Most probably, Debian images were built less frequently and therefore CVEs accumulated.
So I upgraded every image to have the latest available packages and rerun the scans:
+======================+=====+========+======+==========+=======+
| Image | Low | Medium | High | Critical | Total |
+======================+=====+========+======+==========+=======+
| debian:bullseye-slim | 61 | 0 | 13 | 3 | 77 |
+----------------------+-----+--------+------+----------+-------+
| debian:updated | 61 | 0 | 13 | 3 | 77 |
+----------------------+-----+--------+------+----------+-------+
| ubuntu:20.04 | 19 | 7 | 0 | 0 | 26 |
+----------------------+-----+--------+------+----------+-------+
| ubuntu:updated | 19 | 1 | 0 | 0 | 20 |
+----------------------+-----+--------+------+----------+-------+
| alpine:3.16 | 0 | 0 | 2 | 0 | 2 |
+----------------------+-----+--------+------+----------+-------+
| alpine:updated | 0 | 0 | 0 | 0 | 0 |
+----------------------+-----+--------+------+----------+-------+
Ubuntu and Alpine indeed had upgrades waiting to be pulled and fixed some Medium and High vulnerabilities, but Debian’s results remain the same. And when you start installing more packages and dependencies, the gap only increases to the worse on the side of Debian (a lot).
Too bad so many public images’ default version is based on Debian.
Once again, I have to remind you that these CVE scans do not imply that Debian is a less secure OS than Ubuntu and Alpine. The human resources of the Debian project are limited and they have to prioritize addressing issues that they consider to have a bigger impact; a Critical level CVE may not actually be that dangerous taking into consideration the bigger picture of the whole OS security measures.
BUT, the amount of noise that Debian introduces into the scans as you install software makes it impossible to triage the CVEs in your downstream app. As a result, I personally can’t trust it as much as the others for my deployments.
So, our first two takeaways are:
- The choice of the underlying OS image matters a lot.
- Pulling a fresh latest image doesn’t imply that it is updated.
There is a gotcha about this second point: you shouldn’t just go and run a full upgrade at every Dockerfile, because layers stack the changes and the image will get fatter each time you do it. So, if you have a nested parent-child Dockerfile structure, run the upgrade only at the root and rebuild all your images regularly.
Image variants comparison
Now let’s look at the main Python 3.8 images themselves available.
+===================+=====+========+======+==========+=======+
| Image | Low | Medium | High | Critical | Total |
+===================+=====+========+======+==========+=======+
| python:3.8 | 539 | 240 | 335 | 42 | 1156 |
+-------------------+-----+--------+------+----------+-------+
| python:3.8-slim | 66 | 1 | 15 | 3 | 85 |
+-------------------+-----+--------+------+----------+-------+
| python:3.8-alpine | 0 | 0 | 2 | 0 | 2 |
+-------------------+-----+--------+------+----------+-------+Remember those more than 1000 vulns that Trivy reported in your Flask web app? Good news is that they will just go away if you switch your base image from python:3.8 to python:3.8-slim or (ideally) to python:3.8-alpine. Bad news is that they exist and other people may be sitting on them without notice.
But why on earth is there such a huge difference between the normal and the slim versions? Don’t they both come from Debian Bullseye? Good question, and scary answer. The python:3.8-slim image inherits directly from debian:bullseye-slim and then installs its stuff, but python:3.8 instead inherits from buildpack-deps:bullseye.
Wait, what is that? The buildpack-deps is a set of images that contain everything you need to compile code and a bit more, just in case. And, of course, it should NEVER EVER be the base of something that goes to production. Maybe this is fixed in the Dockerfiles of newer versions like 3.10 or 3.11? Lol, no.
The rant
First of all, let’s remember that, unless you explicitly use a “slim” or “alpine” tag (or, God forbid, “buster”), you always get a Bullseye based Python image.
Now, if you want to be truly horrified about what is installed at our lovely Bullseye Python images, they are based on buildpack-deps:bullseye, which is based on buildpack-deps:bullseye-scm, which is based on buildpack-deps:bullseye-curl, which is finally based on debian:bullseye.
Go ahead, take a look at what is in there. You didn’t know that you needed an OpenSSH client and Subversion for your lightweight Python Flask web server, but now you have them. You’re welcome. True that the image is 1GB big and contains a total of 377 High or Critical CVEs, but how could we live without curl and wget?
It is getting incrementally difficult for me not to swear in this post. I hope you appreciate the effort.
Jokes aside, let me empathise, that these Dockerfiles do not follow by any means the Dockerfile best practices. Particularly, they don’t use a multi-stage Dockerfile to separate the build process from the final image. And, what is worse, even though they only have one stage, they don’t remove the build dependencies after the build (neither do the bullseye-slim ones).
This, combined with Debian’s approach towards patching their packages (reminder: they are really hard working people, but they lack hands) leads to a bag full of problems. For real. This is endangering everyone who deploys the ~~OFFICIAL~~ Python image to prod.
“Dude, chill. Are you ok?”
No, I’m angry, and you should be too. But mostly with ourselves for not researching this stuff before using it in a professional environment. We are responsible for what we ship, including its supply chain.
Also, I know the mature next step for me is to go and open a couple of Pull Requests to address these issues instead of whining about the state of some projects maintained by volunteer effort. I KNOW! Give me a break. I’m not even finished with this article yet.
By the way, NodeJS is also affected by this. Why are they even using the buildpack-deps??? They don’t compile anything inside the Dockerfile!!!
Key takeaways:
- If you are serious about your software, read the upstream’s Dockerfiles and make informed decisions about what you use.
- Official != Good
Can anything go worse?
Yes. Of course.
Imagine that instead of using python:3.8 (which, as today, maps to python:3.8.13), when you started your project a while ago you used python:3.8.2 and never looked back:
+==============+======+========+======+==========+=======+
| Image | Low | Medium | High | Critical | Total |
+==============+======+========+======+==========+=======+
| python:3.8 | 539 | 240 | 335 | 42 | 1156 |
+--------------+------+--------+------+----------+-------+
| python:3.8.2 | 1252 | 1205 | 1138 | 170 | 3765 |
+--------------+------+--------+------+----------+-------+Yikes. Shame on you/me/us. But it happens. And in some situations, it may be too difficult/risky to change the software version (not in this example; please, always get Python’s patch updates).
The problem here is twofold:
- Pinning the patch version of Python implies that its runtime will never get security updates. If a Remote Code Execution bug is found in
python:3.8.2, it won’t be solved beforepython:3.8.3, so you are stuck with that problem. - Unlike
python:3.8, which is rebuilt every time a patch version appears (still in active support), the tagpython:3.8.2was built the day that version was released (May 14th 2020), and then never again. As a result it is based on Debian Buster and has two years due of updates.
Let’s see if we can improve the situation by upgrading the OS packages:
+======================+======+========+======+==========+=======+
| Image | Low | Medium | High | Critical | Total |
+======================+======+========+======+==========+=======+
| Python:3.8.2 | 1252 | 1205 | 1138 | 170 | 3765 |
+----------------------+------+--------+------+----------+-------+
| Python:3.8.2-updated | 1196 | 465 | 450 | 45 | 2156 |
+----------------------+------+--------+------+----------+-------+
I mean, that did something, but still far from what I call optimal, mostly because the packages of Debian Buster have more vulns than the ones of Debian Bullseye. Our CVE count won’t go any lower without an OS change.
Key takeaways here:
- Images age poorly.
- Old versions of software don’t get (usually) rebuilt upon fresh bases.
Time for a binary transplant
At this point it is pretty clear the huge liability that base OS images are and that you can’t blindly trust other third-parties on this. The situation gets even more critical when you use specific versions of a containerised software, because the images decay like crazy.
Therefore, the responsible thing to do as professionals is to take control of your whole supply chain and assemble the different elements of your image yourself. If you have multiple apps, you can maintain a base OS image and then multiple runtime images on top of that, that you can finally use to deploy you code onto.
For example, you may have a ubuntu:20.04-custom image that gets rebuilt weekly with the official image plus the latest package upgrades plus some custom security measures. Then, you maintain python:3.8.2-custom and node:16.14.3-custom, both of which inherit from ubuntu:20.04-custom, add the specific software version binaries, and get rebuilt every time the upstream does. Finally, you build you app images (regularly) on top of the Python and NodeJS ones.
Sadly, Python doesn’t offer compiled binaries for Linux. Anyway, we won’t compile them. We are not Gentoo users and we care about saving compute time.
We can do what I call a binary transplant: copy the needed binaries from the official Debian image to an updated safe base Ubuntu image, and, if needed, add the runtime dependencies and configuration. This should work without too much effort; software compiled in Debian should be fully compatible with Ubuntu, after all. Once successfully transplanted, you’ll have a way cleaner repackaged runtime image. This can be done from official Alpine to custom Alpine too, of course; the security/size impact is usually smaller, but still worth to take ownership of the supply chain.
After reading the Dockerfile of the python:3.8 image we know the runtime dependencies, and by playing around a bit with the container we get to know that the Python binaries are stored in /usr/local. Also a couple of environment variables are needed.
With that info we can write the following multi-stage Dockerfile, that upgrades Ubuntu and installs the dependencies in the RUN statement, and then transplants the binaries with the amazing COPY --from functionality:
FROM python:3.8.2 AS upstream#####################################################
FROM ubuntu:20.04
ENV PATH /usr/local/bin:$PATH
ENV LANG C.UTF-8
RUN set -eux \
&& apt-get update \
&& apt-get upgrade -y \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
libbluetooth-dev \
tk-dev \
uuid-dev \
&& rm -rf /var/lib/apt/lists/*
COPY --from=upstream /usr/local /usr/local
CMD ["python3"]
Once built, we need to ensure that this new image, that we call python:3.8.2-ubuntu, actually works. For this, we can use PyPerformance, which executes all kind of different tasks with Python. We run the following inside the container:
$ pip install pyperformance && python -m pyperformance runAfter a while, all the benchmarks finish without issues, so I stamp the “Reasonably functional” seal onto python:3.8.2-ubuntu. And when we run Trivy on it we get the following results:
+=====================+=====+========+======+==========+=======+
| Image | Low | Medium | High | Critical | Total |
+=====================+=====+========+======+==========+=======+
| python:3.8.2-ubuntu | 100 | 40 | 0 | 0 | 140 |
+---------------------+-----+--------+------+----------+-------+I hate to state the obvious, but these results are really impressive in comparison. No High nor Critical vulnerabilities at sight.
Our lazy less-than-15-lines Dockerfile for an outdated Python version outperforms every single non-Alpine Python image in Dockerhub, be it new or old, both in CVE count and size.
I honestly believe that this kind of improvement deserves a salary raise for whoever implements it in a business environment. So go and take ownership of your supply chain before somebody else does!
Last key takeaways:
- Building images yourself may lead to great results.
- It is not too hard.
- Your improvement metrics will look awesome.
Conclussions
First of all, Dockerhub is haunted. During this article I bullied Python as an example, but let me assure you that it is not an exception, but the norm.
On the bright side, having a sane container supply chain is perfectly feasible, but you have to do it yourself. Each scenario may be different and the effort may vary, but the positive impact of keeping a set of secure base images for your specific needs will be huge both in terms of improving security and reducing scan noise.
Once your foundations are solid, you’ll be able to focus all your attention to building your products and gain actionable insights from regular security scans.
Transparency note
The experiments described in this post were conducted on the July 16th 2022, including both pulling images from Dockerhub and upgrading OS packages. Scans were run with Trivy 0.30.0 using the Vulnerability DB v2 updated at 2022–07–16 12:07:59.594932682 +0000 UTC
P.S
Wait a moment… Why is the official Debian image’s Dockerfile in a repo of an account called debuerreotype that has no description info at all nor any way to validate its legitimacy? Look, I’m tired and want to finish this article; leave me alone.
Okay, here, it contains a Github Pages repo that resolves to a subdomain of the Debian project. I guess that this is the new meta for public verification; GPG is dead and we missed the memo.
Cheers.
The views, thoughts, and opinions expressed in the text belong solely to the author, and do not represent the opinion, strategy or goals of the author’s employer, organization, committee or any other group or individual.
