alpine, distroless or scratch?
I recently migrated the 4 Golang apps of the Online Boutique sample apps from alpine
to scratch
and its .NET app from alpine
to chiseled
. Here are some the things I have learned while doing that.
alpine
is a very popular container image to decrease the size of the container images for our own containerized applications as well as for improving our security posture (less surface of attack and less CVEs).
But that’s not enough, alpine
still contains packages with vulnerabilities and still have busybox
and wget
in it, which is not very secure for containerized applications in Production.
Could we do more and better than alpine
? Yup! distroless
can help.
“Distroless” images contain only your application and its runtime dependencies. They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution.
Could we do even more and better than distroless
? Yup! scratch
can help.
“Scratch” image is most useful in the context of building base images (such as
debian
andbusybox
) or super minimal images.
Let’s see the differences between the three of them.
I’m taking Kelsey Hightower’s helloworld
app for this, using scratch
, but we will perform our tests with alpine
and distroless
too:
FROM golang:1.22
WORKDIR /go/src/github.com/kelseyhightower/app/
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build .
# FROM alpine
# FROM gcr.io/distroless/static
# FROM cgr.dev/chainguard/static
FROM scratch
COPY --from=0 /go/src/github.com/kelseyhightower/app/helloworld .
ENTRYPOINT ["/helloworld"]
Size
After building the different variations of the container image, let’s now compare the sizes:
REPOSITORY TAG IMAGE ID CREATED SIZE
helloworld alpine 2c2991efd7cd 7 minutes ago 14.4MB
helloworld distroless 08308f5bc54d 16 minutes ago 9.03MB
helloworld chainguard eaa8a9d18fef 8 seconds ago 7.79MB
helloworld scratch 287ad0140c46 32 minutes ago 7.04MB
They are all very light and small. The alpine
one is bigger, twice the size of the scratch
one, coming with more packages.
Even if this blog post Image sizes miss the point explains the why we shouldn’t focus only on the size. Having smaller container images is often a good sign for less complexity and more security.
To reduce debt, reduce image complexity not size.
Packages
Let’s compare the packages included in these container images once built. Let’s use syft
for that.
alpine
(17 packages):
NAME VERSION TYPE
alpine-baselayout 3.4.3-r2 apk
alpine-baselayout-data 3.4.3-r2 apk
alpine-keys 2.4-r1 apk
apk-tools 2.14.0-r5 apk
busybox 1.36.1-r15 apk
busybox-binsh 1.36.1-r15 apk
ca-certificates-bundle 20230506-r0 apk
helloworld (devel) go-module
libc-utils 0.7.2-r5 apk
libcrypto3 3.1.4-r5 apk
libssl3 3.1.4-r5 apk
musl 1.2.4_git20230717-r4 apk
musl-utils 1.2.4_git20230717-r4 apk
scanelf 1.3.7-r2 apk
ssl_client 1.36.1-r15 apk
stdlib go1.22.2 go-module
zlib 1.3.1-r0 apk
distroless
(5 packages):
NAME VERSION TYPE
base-files 12.4+deb12u5 deb
helloworld (devel) go-module
netbase 6.4 deb
stdlib go1.22.2 go-module
tzdata 2024a-0+deb12u1 deb
chainguard
(7 packages):
NAME VERSION TYPE
alpine-baselayout-data 3.6.4-r0 apk
alpine-keys 2.4-r1 apk
alpine-release 3.20.0_alpha20240329-r0 apk
ca-certificates-bundle 20240226-r0 apk
helloworld (devel) go-module
stdlib go1.22.2 go-module
tzdata 2024a-r1 apk
scratch
(2 packages):
NAME VERSION TYPE
helloworld (devel) go-module
stdlib go1.22.2 go-module
We could now use them wisely, depending on our needs and which programming languages we are using. Really cool and important to know what exactly is in there, right?
CVEs
If you decrease the number of unneeded dependencies and packages, you will reduce your time in maintenance and the associated CVEs updates fatigue.
If we use a tool like trivy
, we could see that only alpine
has CVEs when I wrote this blog post:
helloworld:alpine (alpine 3.19.1)
=================================
Total: 2 (UNKNOWN: 0, LOW: 2, MEDIUM: 0, HIGH: 0, CRITICAL: 0)
┌────────────┬───────────────┬──────────┬────────┬───────────────────┬───────────────┬───────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │
├────────────┼───────────────┼──────────┼────────┼───────────────────┼───────────────┼───────────────────────────────────────────────────────────┤
│ libcrypto3 │ CVE-2024-2511 │ LOW │ fixed │ 3.1.4-r5 │ 3.1.4-r6 │ openssl: Unbounded memory growth with session handling in │
│ │ │ │ │ │ │ TLSv1.3 │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-2511 │
├────────────┤ │ │ │ │ │ │
│ libssl3 │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
└────────────┴───────────────┴──────────┴────────┴───────────────────┴───────────────┴───────────────────────────────────────────────────────────┘
LOW
in this example but still. You will still have to deal with the CVEs of the packages of your own app, so removing the ones from the base image, is always a great win.
More security
Running all of them as unprivileged is no problem with this following command, they all work successfully:
docker run \
-d \
-p 8080:8080 \
--read-only \
--cap-drop=ALL \
--user=65532 \
CONTAINER_IMAGE
If you want to make your container as non-root by default you can use the gcr.io/distroless/static:nonroot
image tag and add a WORKDIR /
instruction in the last part of the Dockerfile
.
Also, the only container image where you can do docker exec
or kubectl exec
is the alpine
one. All the others won’t allow this as they don’t have any shell. On a security standpoint, that’s a really good practice to not have a shell to improve your security posture (again, reducing the surface of attack). Great!
That’s a wrap!
If you have Golang or Rust apps (as mostly-statically compiled), use scratch
. If you need ca-certificates
or tzdata
for examples, go with gcr.io/distroless/static
or cgr.dev/chainguard/static
.
That’s what the Kubernetes project has been doing for 4 years already.
If you need libc
you can use gcr.io/distroless/base-nossl
or gcr.io/distroless/base
(with libssl
).
There are also more container images for other types of apps in Java, .NET, Python, etc.
There are also more initiative around distroless
:
- Chainguard is an important player, they have a lot of
distroless
images. - RedHat has UBI Micro.
- Ubuntu has Chiseled — now included by Microsoft with their
dotnet
container images. As examples, I migrated this container and this other container fromalpine
tochiseled
.
Also, for external containers you deploy in your own clusters, ask for a distroless
flavor. For example, if you use Jib
, it’s now using distroless
by default. For Istio you can optionally use distroless
for the Istio sidecar proxies.
Going distroless
remove the ability to debug into your container (no shell, no package manager, no wget
/ curl
). But at the end of the day, going in debug mode in Production is not a good practice (because you also give more tools to potential hackers). You can use the :debug
tag for the static
, base
and language specific container images if you want. Alternatively, you can also use features like kubernetes debug
or initContainers
instead and if required.
Resources
- scratch for checkout, frontend, productcatalog and shipping by mathieu-benoit · Pull Request #2512 · GoogleCloudPlatform/microservices-demo
- Is Your Container Image Really Distroless?
- Look Docker, No Distro. How to Trim the Fat Off Docker Image
- erickduran/docker-distroless-poc: A simple Proof of Concept of a vulnerable web app using a distroless image and Python
- Chiselled Ubuntu containers: the benefits of combining Distroless and Ubuntu
- Why I Will Never Use Alpine Linux Ever Again | by Martin Heinz | Better Programming