How to Red Hat Certify a Docker Image in 8 Steps: Everything You Need To Know

This blog post walks you through the process of building a new image satisfying all the red hat certification requirements. Also recommends few methods that the industry follows to build a smaller and secure container image.

Why should I certify my Docker image?
Certification ensures that the components are from a trusted source. Also ensures they are free from known vulnerabilities. Hence, certifying a docker image gives peace of mind to consumers.

If you are Red hat certifying a Kubernetes operator then you are already one step closer for it. Because, Red hat certifying the images used by an operator is also one of the requirements for it.

Let’s take the Dockerfile of a sample image that needs to be Red hat certified.

FROM node:alpine as dependencies-solver
RUN apk add — no-cache git
COPY package*.json /bats/
WORKDIR /bats
RUN npm install
FROM alpine:3.10
ENV BATS_HELPERS_DIR=/opt/bats-helpers
COPY — from=dependencies-solver /bats/node_modules/bats /opt/bats
COPY — from=dependencies-solver /bats/node_modules/bats-support /opt/bats-helpers/bats-support
COPY — from=dependencies-solver /bats/node_modules/bats-file /opt/bats-helpers/bats-file
COPY — from=dependencies-solver /bats/node_modules/bats-assert /opt/bats-helpers/bats-assert
RUN apk add — no-cache bash coreutils \
 && ln -s /opt/bats/libexec/bats /sbin/bats
WORKDIR /tests
ENTRYPOINT [“/sbin/bats”]
CMD [“-v”]

Step 1: Choose a base image

To Red hat certify a container image, the image needs to use either RHEL or UBI or Scratch as a base.

Using RHEL as a base image gives little hard time to developers as it is hard to redistribute it. Also, to build an image having RHEL as base image, a RHEL base machine with license is needed. To make things easy, Red hat introduced UBI images last year which allows to redistribute to any registry you want. Also to build images using UBI as a base image, a RHEL base machine is not needed.

registry.access.redhat.com/ubi7/ubi
registry.access.redhat.com/ubi7/ubi-minimal
registry.access.redhat.com/ubi8/ubi
registry.access.redhat.com/ubi8/ubi-minimal

UBI images include a subset of packages that are available in RHEL. The UBI image size is smaller than RHEL image. UBI has yum package manager which makes the job easy to install the required packages. Ubi-minimal doesn’t have yum package manager and includes very less number of packages than that are available in UBI. Hence, I would recommend using UBI image as a base image instead of ubi-minimal if your application has more dependency packages.

From registry.access.redhat.com/ubi7/ubi

Step 2: Add Labels and licenses

a) This is an easy step, you can add n number of labels to the image, but need to add below least labels as they will be inspected by Redhat.

LABEL name="<ImageName>" \
      vendor="<CompanyName>" \
      maintainer="<maintainer name>"
      version="<Version>" \
      summary="<Summary>" \
      description="<Description>"

Once the image is build, view the labels by executing the docker inspect command. Labels are included in the Config.Labels section.

docker inspect <imagename:tag> | jq ".[0].Config.Labels"

b) Copy the licenses folder that contains the LICENSE file for your image.

COPY licenses /licenses

Step 3: Identify the list of packages to install

Easiest way to find the list of packages to install, is to look at the Dockerfile and find the packages in the install commands line by line.

The above list of packages might not be the entire list as there may be dependency packages for them or some may come already installed in the base image. In this case, you will hit an error when you build the image or when you test your application on the built image. You will have to come back to this step at that point and include them in your Dockerfile.

Step 4: Find the equivalent packages in UBI

This step is easier if the existing image is using CentOS/Fedora as a base image instead of Alpine/Debian. The reason is CentOS/Fedora uses rpm and yum package managers to install packages and those are available in UBI image as well. Having said that, if ubi repository doesn’t have the packages that are needed, then you will have to identify the source repositories of those packages where they are available and configure them in the ubi image.

UBI Repository
ftp://partners.redhat.com/1c5d859a/UBI-5032b19e400c3605214630e4ba65e14f/7/RHUBI/x86_64/Packages/

On the search for ubi equivalent package, make sure the package is built for the same architecture we are looking for. For Example: amd64, ppc64le, s390x.

Some of the repositories where we can find the the UBI equivalent packages:

1. http://mirror.centos.org/centos-7/7/os/x86_64/
2. http://dl.fedoraproject.org/pub/epel/7/x86_64/
3. https://forensics.cert.org/centos/cert/7/x86_64/
4. https://rpmfind.net/linux/centos/7/os/x86_64/
5. https://www.rpmfind.net/linux/opensuse/update/leap/42.3/oss/
6. https://download1.rpmfusion.org/free/el/updates/7/x86_64/

The above repositories belong to different linux distributions such as CentOS, Fedora, OpenSUSE. Having said that, they all belong to same OS family, so we can install the packages they contain in UBI as well.

To configure a yum repository, use the repository URL like the above ones as the base URL. Place the below contents in a file inside the /etc/yum.repos.d directory.

/etc/yum.repos.d/centos.repo

[centos]
name=centos
baseurl=http://mirror.centos.org/centos-7/7/os/x86_64/
enabled=1
gpgcheck=0

To start pulling the packages from the above repositories that are newly configured, reload all the yum repositories.

yum update -y && yum clean all

The names of the packages in UBI/CentOS/Fedora might differ from Alpine/Debian. Find the ubi equivalent packages using the commands like yum search and yum info.
For example, in Alpine the cryptography library package is installed as:

apt-get install libgcrypt20

If the same exact package name is used in UBI, it might fail:

$ yum install libgcrypt20
Loaded plugins: ovl, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
No package libgcrypt20 available.
Error: Nothing to do

When encountering such cases, use the search command to find the packages that starts with similar name

$ yum search libgcrypt 
Loaded plugins: ovl, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
===================================================================== N/S matched: libgcrypt =====================================================================
libgcrypt-devel.i686 : Development files for the libgcrypt package
libgcrypt-devel.x86_64 : Development files for the libgcrypt package
libgcrypt.i686 : A general-purpose cryptography library
libgcrypt.x86_64 : A general-purpose cryptography library

And then confirm the ubi equivalent package by reading up on the package’s description and version.

# yum info libgcrypt.x86_64
.
.
Installed Packages
Name        : libgcrypt
Arch        : x86_64
Version     : 1.5.3
Release     : 14.el7
Size        : 584 k
Repo        : installed
From repo   : anaconda
Summary     : A general-purpose cryptography library
URL         : http://www.gnupg.org/
License     : LGPLv2+
Description : Libgcrypt is a general purpose crypto library based on the code used
: in GNU Privacy Guard.  This is a development version.
# yum install libgcrypt.x86_64

The important thing to note when installing a package from a non-ubi repository is, Red hat does not provide the security updates and support for it. You may have to depend on the community to fix the vulnerabilities that are found in it. But, when installing a package from ubi repository, Red hat provides support for it. Also provides security fixes whenever a vulnerability is found in it. Note, if a request is placed by a Red hat customer to add a package to the UBI repository from a non-ubi repository, then Red hat might start supporting them. Having said that, there is no guarantee for that and might take their own time.

Step 5: Handle libraries and other commands

The Dockerfile of the image might include copying of libraries from a source. This is dependent on the application that the Docker container will host. Libraries are precompiled routines. They cannot be simply copied to ubi image that Alpine/Debian uses. Libraries need to be compiled on a ubi container first. And then in the Dockerfile copy it from the newly generated source.

Some of the commands or the command’s options offered by Alpine/Debian might not be available in ubi. But, there should be equivalent commands available which we need to look for and replace it.

For example, in Alpine the timeout command looks like

# timeout -s SIGTERM -t 3 sleep 10
Terminated

whereas in UBI, it slightly differs

$ timeout -s SIGTERM 3s sleep 10

Step 6: Run as non-root user

Running a container as root user gives access to the base machine. To avoid that, Red hat inspects whether a container image is configured to run as a non-root user. Configuring a container to run as non-root user is little tricky. Non-root users don’t have same privileges as root user. Permission denied errors occur if appropriate permissions are not set for the required files and directories.

It is tempting to run the container as root user to avoid permission denied errors and keep things easy. But, with little bit of effort the container can be run as a non-root user and can limit the access for that user.

By default, the Docker container is run as root user if the USER instruction is not used in the Dockerfile.

Although the container needs to be run as a non-root user, the user can still belong to the root group. Configuring the image to run as non-root user just before the entry point in the Dockerfile rather than in the beginning of it will be convenient . This way, in the prior stages, can avoid “permission denied” errors while installing the packages as root user. Also while performing other operations such as changing permissions/ownership to files/directories.

FROM registry.access.redhat.com/ubi7/ubi as dependencies-solver
COPY package*.json /bats/
WORKDIR /bats
RUN yum -y update \
 && yum install -y \
 rh-nodejs10 \
 git \
 && scl enable rh-nodejs10 bash
ENV PATH $PATH:/opt/rh/rh-nodejs10/root/usr/bin/
RUN npm install
FROM registry.access.redhat.com/ubi7/ubi
MAINTAINER <Maintainer Name>
### Required Atomic/OpenShift Labels - https://github.com/projectatomic/ContainerApplicationGenericLabels#####
LABEL name="<imagename>" \
      vendor="<vendor>" \
      version="<version>" \
      summary="<summary>" \
      description="<description>"
RUN useradd -r -u 1000 -g root bats
ENV BATS_HELPERS_DIR=/opt/bats-helpers
COPY --from=dependencies-solver --chown=bats:root /bats/node_modules/bats /opt/bats
COPY --from=dependencies-solver --chown=bats:root /bats/node_modules/bats-support /opt/bats-helpers/bats-support
COPY --from=dependencies-solver --chown=bats:root /bats/node_modules/bats-file /opt/bats-helpers/bats-file
COPY --from=dependencies-solver --chown=bats:root /bats/node_modules/bats-assert /opt/bats-helpers/bats-assert
RUN ln -s /opt/bats/libexec/bats /sbin/bats
WORKDIR /tests
USER 1000
ENTRYPOINT ["/sbin/bats"]
CMD ["-v"]

Step 7: Build and test the image

After we have addressed all the red hat certification specific requirements in the Dockerfile, the image needs to be built and tested. This can done by running the container using the newly built image and performing tests against the application that is being hosted in it.

Build the image and start the container:

$ docker build <directory path> -f <filename> -t <imagename:tag>
$ docker run -it <imagename:tag>

This is the step where you actually get to know whether all the dependency packages have been installed and appropriate permissions have been set for the required files and directories. If anything breaks, it needs to be fixed here.

If the image build fails at any step, one way to debug is to run the steps that are failing on a running UBI container. This way, it is easier to identify the issue and find the fix for it, later the fix can be included in the Dockerfile.

Step 8: Scan the image

To Red hat certify an image,
i) Register in Red hat connect portal and login.
ii) Create a product with all the information that needs to show up in the catalog.
iii) A separate project needs to be created for each container image under the above product.
Official instructions can be followed to create the above. Once you have the project created and enter into it, instructions to upload the image for scanning can be seen when clicked on the upload image tab.

After the image is uploaded, you can see the scan status when clicked on the container information tab. Normally it takes few minutes to scan but it may vary depending upon the no. of layers the image has and meanwhile the status would be Scan In-Progress.

If the status is failed, then it means not all of the requirements from the checklist has been met. By clicking on the view results, you can see the reason why it failed.

On the other hand, if the status is passed, then it means the uploaded image has met all the red hat certification requirements. By clicking on the publish button, the image can be published if desired to be published.

If the scan status remains Scan In-Progress for hours and never shows the result, then probably either the uploaded image has huge no. of layers or oversized which is beyond Red hat’s scanning capability. In this cases, the size or no. of layers in the Docker image needs to be reduced.

If all the above steps have been gone through and the scan results shows passed, then Congratulations!!! You have Red hat certified your Docker image successfully. Although the following steps are optional, it is highly recommended go through them.

Step 9: Reduce the size of the image

Reducing the size of the image reduces the time taken for it to upload/ download/ deploy and the red hat scanning process to go smoothly.

There many ways to reduce the size of the image, few important among them are as follows:

i) Reduce the number of layers in an image

The common reason for increase in image size is using separate RUN instructions to install packages or perform any other operations in the Dockerfile. For each RUN instruction used in the Dockerfile, a new layer is created. Can reduce the number of layers in a Docker image by combining the RUN instructions in the Dockerfile together as much as possible.

For example

RUN yum -y update 
RUN yum install -y rh-nodejs10 
RUN yum install -y git 
RUN scl enable rh-nodejs10 bash

can be combined as

RUN yum -y update \
    && yum install -y \
    rh-nodejs10 \
    git \
    && scl enable rh-nodejs10 bash

which reduces four layers to a single layer.

ii) Use ubi-minimal as the base image

The size of ubi-minimal image is smaller than ubi image and it includes only a subset of packages from ubi intentionally to keep its size smaller. To install packages on ubi-minimal, yum package manager is not available but you can use microdnf. Microdnf supports only install, remove and update packages. It doesn’t support everything that is supported by yum.

In order to make both installation easier and reduce the size of the image, the image can be built in multistage. All the packages can be installed in UBI and then copied to the ubi-minimal image like below:

FROM registry.access.redhat.com/ubi7/ubi as dependencies-solver
COPY package*.json /bats/
WORKDIR /bats
RUN yum -y update \
 && yum install -y \
 rh-nodejs10 \
 git \
 && scl enable rh-nodejs10 bash
ENV PATH $PATH:/opt/rh/rh-nodejs10/root/usr/bin/
RUN npm install
FROM registry.access.redhat.com/ubi7/ubi-minimal
RUN microdnf install -y shadow-utils \
 && adduser bats -u 1000 -g 0
ENV BATS_HELPERS_DIR=/opt/bats-helpers
COPY — from=dependencies-solver — chown=bats:root /bats/node_modules/bats /opt/bats
COPY — from=dependencies-solver — chown=bats:root /bats/node_modules/bats-support /opt/bats-helpers/bats-support
COPY — from=dependencies-solver — chown=bats:root /bats/node_modules/bats-file /opt/bats-helpers/bats-file
COPY — from=dependencies-solver — chown=bats:root /bats/node_modules/bats-assert /opt/bats-helpers/bats-assert
RUN ln -s /opt/bats/libexec/bats /sbin/bats
WORKDIR /tests
USER 1000
ENTRYPOINT ["/sbin/bats"]
CMD ["-v"]

Another thing to note here is, the useradd command in ubi is not supported in ubi-minimal. Installing shadow-utils package makes adduser command available in ubi-minimal that is useradd equivalent.

iii) Keep only the essential content and remove rest of them.

When installing packages using package manager, some cache gets created. These cache adds up space in each layer and by removing them after installing the packages, frees up some space. Also if there are any content that are considered as not necessary can be removed so that it frees up space.

yum clean all -y

Step 10: Support running the container as an arbitrarily assigned user ID

By default, Openshift runs the container as an assigned user id. Some of the reasons to run the container as an assigned user are:

i) Providing additional security against processes escaping the container due to a container engine vulnerability and thereby achieving escalated permissions on the host node.

ii) Providing an extra layer of separation between applications run by different users, or which are different parts of a complex system which is deployed across multiple projects and which should have limited visibility of other parts.

In an effort to support them, the major difference between run as non-root user and run as an assigned user is:

To run the container as a non-root user, it is enough if that user has appropriate read, write and executable permissions for the required files and directories. That user can belong to any group, the permissions are not needed to be assigned in the owner group level.

But, to support running the container as an arbitrarily assigned user id, the root group needs to have appropriate read, write and executable permissions for the required files and directories. The user id is dynamically generated and belongs to root user group.

Conclusion

To summarize all the steps:

1. choose a base image
2. Add labels and licenses
3. Identify the list of packages to install
4. Find the equivalent packages in ubi
5. Handle libraries and other commands
6. Run as non-root user
7. Build and test the image
8. Scan the image
9. Reduce the size of the image
10. Support running the container as an arbitrarily assigned user ID.

Have you struggled with any step in the process of Red hat certifying a Docker image ?

Please let me know your thoughts and comments. You can drop a mail at harinarayananmohan@gmail.com