Scan Docker image vulnerabilities using Clair, Klar, Docker Registry and Traefik

Problem: Need to verify Docker image vulnerabilities

Solution: use the open source tool Clair (https://github.com/quay/clair)

Clair is an open source project for the static analysis of already known vulnerabilities in containers. It pulls the known vulnerabilities from various sources such as:

• Debian Security Bug Tracker
• Red Hat Security Data
• Ubuntu CVE Tracker

In addition I’ll use Klar (https://github.com/optiopay/klar). Klar serves as a client which coordinates the image checks between the Docker registry and Clair.

At this point we need :

• Docker
• Docker-compose
• A Docker registry
• Clair
• Klar
• Traefik v2

Fortunately for us, all of this can be dockerized.

First Step: Create a private Docker Registry

# workdir structure:
…/workdir 
   - acme.json
   - traefik.yml
   - docker-compose-traefik.yml

This step can be bypassed if you already have a Docker Registry or if you’re using Docker Hub to store your images.

If you’re like me and need a private Docker Registry, I’ll use traefik v2 as a reverse proxy to expose the Docker Registry container.

# traefik.yml
entryPoints:
  web:
    address: :80
  websecure:
    address: :443
certificatesResolvers:
  letsencrypt:
    acme:
      email: your@email.com
      storage: acme.json
      httpChallenge:
        entryPoint: web
providers:
  docker:
    endpoint: unix:///var/run/docker.sock
    exposedByDefault: false
        
log:
  level: DEBUG
api:
  dashboard: true

This configuration of traefik is a simple one. Only thing that matters is the certificateResolvers letsencrypt. The actions needed to generate the acme.json file are quite easy: touch acme.json and then chmod 600 acme.json

Let’s take focus on the docker-compose.yml file for traefik v2.

# docker-compose-traefik.yml
version: "3.7"
services:
  traefik:
    image: traefik:latest
    container_name: traefik
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.yml:/traefik.yml:ro
      - ./acme.json:/acme.json

    networks:
      - traefik_network
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.api.rule=Host(`dashboardhost`)'   
      - 'traefik.http.routers.api.entrypoints=web'
      - 'traefik.http.routers.api.service=api@internal'
      - 'traefik.http.routers.api.entrypoints=websecure'
      - 'traefik.http.routers.api.tls=true'
      - 'traefik.http.routers.api.tls.certresolver=letsencrypt'
      
networks:
  traefik_network:
    external: true

Two things to focus on:

• This traefik configuration will deploy traefik’s dashboard (dashboard at true in traefik.yml). Can be bypassed. I use a special network. Be sure to create it before running docker-compose up: docker network create traefik_network
• Let’s Encrypt as a certresolver. To make sure Let’s Encrypt works, it requires the acme.json file. Before starting the container (using docker-compose up), a chmod 600 is needed for this file since letsencrypt will write into that file.

After generating this docker-compose-traefik.yml file let’s do a docker-compose -f docker-compose-traefik.yml up -d.

Be sure to check the logs. Traefik’s dashboard should be available.

Let’s take a look at the registry’s docker-compose.yml

# docker-compose-registry.yml
version: "3.7"
services:
  registry:
    image: registry
    container_name: docker-registry
    networks:
      - traefik_network
    volumes:
      - ./data_registry/:/var/lib/registry
    expose:
      - 5000
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.reg.rule=Host(`registryhost`)'
      - 'traefik.http.routers.reg.entrypoints=websecure'
      - 'traefik.http.routers.reg.tls=true'
      - 'traefik.http.routers.reg.tls.certresolver=letsencrypt'
networks:
  traefik_network:
    external: true
# workdir structure:
…/workdir 
   - data_registry/
   - docker-compose-registry.yml

Obviously if you are just trying to deploy a registry locally , no need for the tls and certresolver. Be sure to use the web entrypoint. Same for traefik’s dashboard.

Again start the container: docker-compose -f docker-compose-registry.yml up -d

curl -X GET -I https://registryhost/v2/ should return an HTTP 200

Second Step: Let’s create the Clair Container

First, the Clair config file. You can get it on the GitHub Repo (https://github.com/quay/clair/blob/master/config.yaml.sample)

# Copyright 2015 clair authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, 
clair:
  database:
    # Database driver
    type: pgsql
    options:

      source: postgresql://postgres:test@clair_postgres:5432?sslmode=disable
# The values specified here are the default values that Clair uses if no configuration file is specified or if the keys are not defined.
clair:
  database:
    type: pgsql
    options:
      source: postgresql://postgres:test@clair_postgres:5432?sslmode=disable      cachesize: 16384
# 32-bit URL-safe base64 key used to encrypt pagination tokens
      # If one is not provided, it will be generated.
      # Multiple clair instances in the same cluster need the same value.
      paginationkey:
# Maximum number of open connections allowed to database
      # If unspecified or <= 0 then no limit is enforced in Clair
      maxopenconnections: 10
api:
    # v3 grpc/RESTful API server address
    addr: "0.0.0.0:6060"
    # Health server address
    # This is an unencrypted endpoint useful for load balancers to check to healthiness of the clair server.
    healthaddr: "0.0.0.0:6061"
# Deadline before an API request will respond with a 503
    timeout: 900s
# Optional PKI configuration
    # If you want to easily generate client certificates and CAs, try the following projects:
    # https://github.com/coreos/etcd-ca
    # https://github.com/cloudflare/cfssl
    servername:
    cafile:
    keyfile:
    certfile:
updater:
    # Frequency the database will be updated with vulnerabilities from the default data sources
    # The value 0 disables the updater entirely.
    interval: 2h
    enabledupdaters:
      - debian
      - ubuntu
      - rhel
      - oracle
      - alpine
      - suse
notifier:
    # Number of attempts before the notification is marked as failed to be sent
    attempts: 3
# Duration before a failed notification is retried
    renotifyinterval: 2h
http:
      # Optional endpoint that will receive notifications via POST requests
      endpoint:
# Optional PKI configuration
      # If you want to easily generate client certificates and CAs, try the following projects:
      # https://github.com/cloudflare/cfssl
      # https://github.com/coreos/etcd-ca
      servername:
      cafile:
      keyfile:
      certfile:
# Optional HTTP Proxy: must be a valid URL (including the scheme).
      proxy:

Nothing crazy here, only a small modification on the postgres connection, source: postgresql://postgres:test@clair_postgres:5432?sslmode=disable instead of source: host=localhost port=5432 user=postgres sslmode=disable statement_timeout=60000.

The mention “clair_postgres” is the postgres container described below.

Then the docker-compose.yml file

# docker-compose-clair.yml
version: "3.7"
services:
  postgres:
    container_name: clair_postgres
    image: postgres:latest
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: test
      POSTGRES_USER: postgres
    volumes:
      - ./postgres/init.sql:/docker-entrypoint-initdb.d/init.sql
      - ./postgres/database-data:/var/lib/postgresql/data
    networks:
      - db
  
  clair:
    container_name: clair_container
    image: quay.io/coreos/clair:v2.1.3
    depends_on: 
      - postgres
    ports:
      - "6060:6060"
      - "6061:6061"
    links:
      - postgres
    volumes:
      - /tmp:/tmp
      - ./clair_config/:/config
    command: [-config, /config/config.yaml]

    networks:
      - web
      - db
networks:
  web:
    external: true
  db:
# workdir structure:
…/workdir 
   -clair_config/
     -- config.yml
   - postgres/
     -- init.sql 
     -- database-data/
   - klar/ 
     -- Dockerfile
   - docker-compose-clair.yml

Points to focus on the:

• To be able to run, Clair uses a postgres database. As you can see in this docker-compose.yml, I mount an init.sql. I only create the clair database and give all the privileges to the user postgres

CREATE DATABASE clair;
GRANT ALL PRIVILEGES ON DATABASE clair to postgres;

• Note that I mount a folder on /var/lib/postgresql/data, it’s intended to make postgres date persistent. Indeed when starting, Clair takes about 10–15 minutes to fill the database with the vulnerabilities.
• I specify the environment variables, to connect to the postgres database, directly into the Dockerfile but it’s better to use a .env file.
• About the Clair container, note that I don’t use the latest image. Indeed, at the time I’m writting, the latest image is far behind the 2.1.3 image (https://quay.io/repository/coreos/clair?tab=tags)

Then only a docker-compose -f docker-compose-clair.yml up -d is necessary to make it run. Check the postgres database or the clair_container logs to know when the vulnerabilities will be available in the database.

When the containers are up and running you can start querying the API. I chose to use Klar to easy things up.

Third Step: Use Klar to query Clair’s API

A possibility is to use Klar’s binary (https://github.com/optiopay/klar/releases/)

Otherwise there is is the Dockerfile

FROM golang:1.9-alpine as builder
RUN apk --update add git;
RUN go get -d github.com/optiopay/klar
RUN go build ./src/github.com/optiopay/klar
FROM alpine:3.8
RUN apk add --no-cache ca-certificates
COPY --from=builder /go/klar /klar
ENTRYPOINT ["/klar"]

Once the image is built you only need to run the container:

docker run CLAIR_ADDR=IPContainterClair --network=web klarimage yourregistry/imagetotest

Again I decided not to store the variables into a .env file. I used localhost but obviously clair_container’s IP would work. You can use different options, you just need to look into Klar’s github repo.

The images you test from your registry need to be pushed to the registry before running Klar.

The output looks like this:

-----------------------------------------
CVE-2019-20388: [Medium] 
Found in: libxml2 [2.9.4+dfsg1-7]
Fixed By: 
xmlSchemaPreRun in xmlschemas.c in libxml2 2.9.10 allows an xmlSchemaValidateStream memory leak.
https://security-tracker.debian.org/tracker/CVE-2019-20388
-----------------------------------------
CVE-2018-14567: [Medium] 
Found in: libxml2 [2.9.4+dfsg1-7]
Fixed By: 
libxml2 2.9.8, if --with-lzma is used, allows remote attackers to cause a denial of service (infinite loop) via a crafted XML file that triggers LZMA_MEMLIMIT_ERROR, as demonstrated by xmllint, a different vulnerability than CVE-2015-8035 and CVE-2018-9251.
https://security-tracker.debian.org/tracker/CVE-2018-14567
-----------------------------------------
CVE-2020-7595: [Medium] 
Found in: libxml2 [2.9.4+dfsg1-7]
Fixed By: 
xmlStringLenDecodeEntities in parser.c in libxml2 2.9.10 has an infinite loop in a certain end-of-file situation.
https://security-tracker.debian.org/tracker/CVE-2020-7595
-----------------------------------------

Now you have a few set of tools that can tell you which are the vulnerabilities and how to fix them. It goes from Negligeable to High.

In order to fix those vulnerabilities, make sur to upgrade the libraries that you use. Otherwise you can check each vulnerability and figure out if they are a threat to your containers.

This way the check of your Docker images is thorough.