Steady the Ship

The “Develop, Ship and Run” are three building blocks of Containerization. It is Flexible, Efficient, Portable, and Scalable. But there is a big question mark when it comes to security.
In this article, we’ll discuss how to build less vulnerable and secure containers.
Pre-requisites
- Jenkins on Kubernetes cluster
How ToDo

The above diagram demonstrates the approach. We push the code to git, and Jenkins triggers a build. During the Jenkins build the source goes through different stages with the help of different tools.
- Validate Dockerfile (Hadolint)
- Build Docker image (Kaniko)
- Scan Vulnerabilities (Clair)
Let’s have a quick look at these tools.
Hadolint
Hadolint is a newly joined member of the lint family which validates Dockerfile and enforces best practices.
Demo Time
I have this Dockerfile on my repo, I am going to build it now and let’s check the output.
FROM nginx
MAINTAINER "hari.karthigasu@gmail.com"
RUN echo "Building hello docker..."
COPY html/ /usr/share/nginx/html/+ hadolint /home/jenkins/agent/workspace/o-docker_hello-docker_DEV-011119/Dockerfile/home/jenkins/agent/workspace/o-docker_hello-docker_DEV-011119/Dockerfile:1 DL3006 Always tag the version of an image explicitly
/home/jenkins/agent/workspace/o-docker_hello-docker_DEV-011119/Dockerfile:3 DL4000 MAINTAINER is deprecated
The build fails and hadolint is not happy. It’s telling us, MAINTAINER is deprecated and no version in the base image.
Kaniko
Kaniko is an image build tool. It doesn’t depend on Docker daemon. It completely builds the image in userspace. We have security concerns when building Docker images in Jenkins by mounting docker.sock (DooD method). Also, we have to run the container in privileged mode. Kaniko overcomes these issues.
Sample Output
[36mINFO[0m[0005] Downloading base image nginx:1.17.4
[36mINFO[0m[0006] Unpacking rootfs as cmd RUN echo "Building hello docker..." requires it.
[36mINFO[0m[0039] Taking snapshot of full filesystem...
[36mINFO[0m[0049] RUN echo "Building hello docker..."
[36mINFO[0m[0049] cmd: /bin/sh
[36mINFO[0m[0049] args: [-c echo "Building hello docker..."]
Building hello docker...
[36mINFO[0m[0049] Taking snapshot of full filesystem...
[36mINFO[0m[0056] No files were changed, appending empty layer to config. No layer added to image.
[36mINFO[0m[0056] Using files from context: [/home/jenkins/agent/workspace/hello-docker_hello-docker_master/html]
[36mINFO[0m[0056] COPY html/ /usr/share/nginx/html/
[36mINFO[0m[0056] Taking snapshot of files...
2019/11/01 11:23:06 existing blob: sha256:b0bbed1a78ca915b4edb8d4b79354ab619c6a2a90bac9c8e9008bb380bf47c1d
2019/11/01 11:23:06 existing blob: sha256:8d691f585fa8cec0eba196be460cfaffd69939782d6162986c3e0c5225d54f02
2019/11/01 11:23:06 existing blob: sha256:047cb16c0ff61d448f521bcffb2133d6192bbf2e8e725f648f63fc707415cb9b
2019/11/01 11:23:08 pushed blob: sha256:c7c24f5eccafdc466f6fe162202f312751f2cd705a2763f3dc1227c8442f56d5
2019/11/01 11:23:08 pushed blob: sha256:34720cb3ce14f8e2b806e41f9e216564c74ce9a911d439f8ac4a33c73bb26b78
2019/11/01 11:23:09 index.docker.io/****/hello-docker:latest: digest: sha256:418c1c1ca31556e2a78ee559aea93947115acd644a0edbe304d1c0b45d6f8e9c size: 915You can find Kaniko pod template at the Implementation section.
Clair
Clair is an open-source tool for vulnerabilities scanning of container images. It’s using Postgres as backend. Klar is the CLI tool for Clair. You could find Clair helm chart here.
Demo Time
This is the Jenkins pipeline stage for the scan. Vulnerability level Low and Threshold 10 (Number of outputted vulnerabilities Klar can tolerate before returning 1. Default is 0.)
stage("scan") {
steps {
container("klar") {
sh """
CLAIR_ADDR=$CLAIR_ADDRESS \
CLAIR_OUTPUT=Low \
CLAIR_THRESHOLD=10 \
DOCKER_USER=$DOCKER_CREDENTIALS_USR \
DOCKER_PASSWORD=$DOCKER_CREDENTIALS_PSW \
DOCKER_INSECURE=true \
klar $registry:$tag
"""
}
}
}Build fails and the Clair outputs all the vulnerabilities as shown below.
+ CLAIR_ADDR=http://clair-clair.clair.svc:6060 CLAIR_OUTPUT=Low CLAIR_THRESHOLD=10 DOCKER_USER=**** DOCKER_PASSWORD=**** DOCKER_INSECURE=true klar ****/hello-docker:latest
clair timeout 1m0s
docker timeout: 1m0s
no whitelist file
Analysing 4 layersGot results from Clair API v1
Found 72 vulnerabilities
Unknown: 9
Negligible: 48
Low: 15
CVE-2018-7169: [Low]
Found in: shadow [1:4.5-1.1]
Fixed By:
An issue was discovered in shadow 4.5. newgidmap (in shadow-utils) is setuid and allows an unprivileged user to be placed in a user namespace where setgroups(2) is permitted. This allows an attacker to remove themselves from a supplementary group, which may allow access to certain filesystem paths if the administrator has used "group blacklisting" (e.g., chmod g-rwx) to restrict access to paths. This flaw effectively reverts a security feature in the kernel (in particular, the /proc/self/setgroups knob) to prevent this sort of privilege escalation.
https://security-tracker.debian.org/tracker/CVE-2018-7169
-----------------------------------------
CVE-2016-10228: [Low]
Found in: glibc [2.28-10]
Fixed By:
The iconv program in the GNU C Library (aka glibc or libc6) 2.25 and earlier, when invoked with the -c option, enters an infinite loop when processing invalid multi-byte input sequences, leading to a denial of service.
https://security-tracker.debian.org/tracker/CVE-2016-10228
-----------------------------------------
Now we have the ingredients let’s add them together and we’ll cook a secure container.
Implementation
The complete Jenkinsfile.
// hello-docker
String image = env.JOB_NAME.split('/')[1]
String registry = "harik8/$image"
String tag = "latest"
pipeline {
agent {
kubernetes {
cloud 'kubernetes'
label 'hello-docker'
defaultContainer 'jnlp'
yamlFile 'pod.yaml'
}
}
environment {
DOCKER_CREDENTIALS=credentials("DOCKER_CREDENTIALS")
}
stages {
stage("lint") {
steps {
container("lint") {
sh "hadolint $WORKSPACE/Dockerfile"
}
}
}
stage("publish") {
steps {
container(name: 'kaniko', shell: '/busybox/sh') {
withEnv(['PATH+EXTRA=/busybox:/kaniko']) {
sh """#!/busybox/sh
/kaniko/executor --context=$WORKSPACE --destination $registry:$tag
"""
}
}
}
}
stage("scan") {
steps {
container("klar") {
sh """
CLAIR_ADDR=$CLAIR_ADDRESS \
CLAIR_OUTPUT=High \
CLAIR_THRESHOLD=1 \
DOCKER_USER=$DOCKER_CREDENTIALS_USR \
DOCKER_PASSWORD=$DOCKER_CREDENTIALS_PSW \
DOCKER_INSECURE=true \
klar $registry:$tag
"""
}
}
}
}
}Pod template to run the job. Which has jnlp, kaniko, hadolint and klar images.
apiVersion: v1
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: jnlp
image: jenkins/jnlp-slave:3.27-1-alpine
resources:
limits:
cpu: 100m
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
- name: lint
image: harik8/hadolint:v1.0.0
command:
- cat
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
tty: true
- name: kaniko
image: gcr.io/kaniko-project/executor:debug-v0.10.0
command:
- /busybox/cat
resources:
limits:
cpu: 100m
memory: 1Gi
requests:
cpu: 100m
memory: 512Mi
tty: true
volumeMounts:
- name: docker-config
mountPath: /kaniko/.docker/
- name: klar
image: harik8/klar:v1.0.0
command:
- cat
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
tty: true
volumes:
- name: docker-config
configMap:
name: docker-config
restartPolicy: NeverJenkins Console Output of each stage.
+ hadolint /home/jenkins/agent/workspace/hello-docker_hello-docker_master/Dockerfile2019/11/01 11:23:06 existing blob: sha256:b0bbed1a78ca915b4edb8d4b79354ab619c6a2a90bac9c8e9008bb380bf47c1d
2019/11/01 11:23:06 existing blob: sha256:8d691f585fa8cec0eba196be460cfaffd69939782d6162986c3e0c5225d54f02
2019/11/01 11:23:06 existing blob: sha256:047cb16c0ff61d448f521bcffb2133d6192bbf2e8e725f648f63fc707415cb9b
2019/11/01 11:23:08 pushed blob: sha256:c7c24f5eccafdc466f6fe162202f312751f2cd705a2763f3dc1227c8442f56d5
2019/11/01 11:23:08 pushed blob: sha256:34720cb3ce14f8e2b806e41f9e216564c74ce9a911d439f8ac4a33c73bb26b78
2019/11/01 11:23:09 index.docker.io/****/hello-docker:latest: digest: sha256:418c1c1ca31556e2a78ee559aea93947115acd644a0edbe304d1c0b45d6f8e9c size: 915+ CLAIR_ADDR=http://clair-clair.clair.svc:6060 CLAIR_OUTPUT=High CLAIR_THRESHOLD=1 DOCKER_USER=**** DOCKER_PASSWORD=**** DOCKER_INSECURE=true klar ****/hello-docker:latest
clair timeout 1m0s
docker timeout: 1m0s
no whitelist file
Analysing 4 layers
Got results from Clair API v1
Found 72 vulnerabilities
Unknown: 9
Negligible: 48
Low: 15So, in the end, we have a pipeline that validates, builds and scans docker images. Which ensures the secureness of the container.
Sources,
- https://github.com/harik8/hello-docker/
- https://github.com/harik8/images
- https://hub.docker.com/u/harik8
Finally, do you think your container is fully secure now?
Feedbacks are Welcome!
