Securing Containers Using Red Hat Quay and Clair — Part II

In our previous post, we discussed the issues administrators face when it comes to the security of outside container content in both development and runtime environments.

Open source components are a significant weak link in container environments, and the layered aspect of container images means careful review is required before deployment into a runtime environment. Accomplishing this without diminishing agility or speed requires tools created specifically for integration and implementation.

Red Hat Quay is the container and application registry which allows developers to securely build, analyze, and distribute container images. Quay also provides a Docker registry service, and then seamlessly integrates with Clair to implement scanning of container images for security vulnerabilities.

The Clair project is an open source engine that powers Red Hat Quay Security Scanner to detect vulnerabilities in all images within Red Hat Quay, then notify developers as those issues are discovered.

This article assumes you have a functioning Quay environment. Our article is based on having Red Hat Quay 3 installed with Red Hat OpenShift 3.11.98 and Ansible 2.6. This also assumes that you have a supported Object Oriented Storage location established and is already being used by Quay.

CONFIGURE QUAY

Quay needs to be configured to communicate with Clair. Configuration must be done using a Quay user account with Superuser permissions.

1. 	Login to the Quay web UI using a Superuser type user account
2. 	Navigate to:
Super User Admin Panel --> Registry Settings --> Security Scanner
3. 	Select checkbox next to "Enable Security Scanning"
4. 	Set Security Scanner Endpoint: http://clair-app-api.quay-enterprise.svc.cluster.local:6060
5. 	Create Authentication Key (record Key ID and private key):
Generate Key -> Generate Shared Key -> Continue -> Generate Key
6. 	Write private key to openshift_certs/security_scanner.pem
7. 	Click “Save Configuration Changes” button at the bottom of the page.

STAGE CONTAINER IMAGES

Depending on the organization and its security policies, Direct internet access may be allowed and the OCP infrastructure servers may be configured to download directly from Red Hat or Docker. If this is the case, the next section can be skipped.

However; if an on-Prem registry is already in place and is going to be the preferred location for image storage and retrieval. This will need to be done if the OCP servers will NOT be allowed to download directly from Red Hat or Docker.

$ docker pull quay.io/coreos/clair-jwt:v2.0.8
$ docker tag quay.io/coreos/clair-jwt:v2.0.8 quay.io/coreos/clair-jwt:latest
$ docker tag quay.io/coreos/clair-jwt:latest \
registry.apps.ocp-dev01-bna.ssc.tsc/quay-enterprise/clair-jwt
$ docker push registry.apps.ocp-dev01-bna.ssc.tsc/quay-enterprise/clair-jwt

CREATE DATABASES

The Clair tool requires that it installs a small version of the Postgres database. This database is used for CVE (Common Vulnerabilities and Exposures)files that are downloaded from Red Hat and Docker. These files are needed for the scanning of images that will be used in the OCP environment.

$ oc rsh <postgresql pod name>
sh-4.2$ createdb clair -O quayuser
sh-4.2$ createdb quayuser -O quayuser

CONFIGURE AUTHENTICATION

Configure credentials to enable Clair to communicate with the Postgresql database and Quay.

This communication is vital for the in depth scanning process that Clair will perform and compare an imported image from the developers against the CVE information housed in the Postgres database. Since this information contains sensitive information and may expose certain vulnerabilities, allowing a “service” account to perform the tasks without human intervention is necessary for the automation process. Once that process is complete, this information is then taken by Quay to show results and reports of what it has found.

# Set Postgresql password in config/config.yml, then create secret.
# Important - URL encode any special characters in password (see https://developer.mozilla.org/en-US/docs/Glossary/percent-encoding)
$ vi config/config.yml
clair:
database:
type: pgsql
options:
# A PostgreSQL Connection string pointing to the Clair Postgres database.
# Documentation on the format can be found at: http://www.postgresql.org/docs/9.4/static/libpq-connect.html
source: postgres://quayuser:<POSTGRESQL_PASSWORD>@postgresql.quay-enterprise.svc.cluster.local:5432/clair?sslmode=disable
cachesize: 16384
<snip>
$ oc create configmap clair-app --from-file=config/ -n quay-enterprise
# ca.crt and security_scanner.pem should be located in openshift_certs directory
$ cd ../openshift_certs
$ oc create secret generic ca-crt --from-file=ca.crt -n quay-enterprise
$ oc create secret generic securityscanner \
--from-file=security_scanner.pem -n quay-enterprise
$ cd ../quaydeploy/

DEPLOY CLAIR

Deploying Clair is similar to when we deployed Quay, OCP treats this just like an internal service and therefore should be deployed as projects that will in turn create containers to run the service and start up all of the necessary processes , opening of the published ports needed for network communication and all the required internal certificate needed for secure communication. This also allows for the service to be monitored for health checks and self-healing by the OCP platform.

$ oc create -f clair-service.yml -n quay-enterprise
$ oc create -f clair-deployment.yml -n quay-enterprise

USING CLAIR

Quay will automatically start using Clair to scan images as they are pushed to Quay once this implementation process has been completed. Clair scanning results will be displayed in the “SECURITY SCAN” column on the “Repository Tags” page for each Repository in the Quay web UI. Click the scan result for additional details such as the number of vulnerabilities, vulnerability criticality, links to advisories, etc.

APPLICATION DEPLOYMENT

This section shows how to deploy an application container using Quay as the container image registry. The source image is pulled from the Quay registry. Then the image resulting from the application build process is pushed in the Quay registry. Next, Quay scans the new image using Clair. Finally, the application container is pulled from the Quay registry so that it can be deployed by OpenShift. In this example, we’ll call this application “HelloWorld”. Our application is going to deploy and Apache Web Server with a greeting page.

*NOTE* Our example is based on the assumption that you have been able to successfully import an image into your own environment or you have access to a registry that already has test images that you would like to use. Some information may need to be filled in by you the reader for functionality and to be relative to your environment.

CREATE APPLICATION DEPLOYMENT FILES

Create a directory to hold the application deployment template and subdirectory for application content. Then create the template and application content.

$ mkdir -p helloworld-app-demo/build_content
$ cd helloworld-app-demo/

httpd-helloworld-demo.yml

Create httpd-helloworld-demo.yml in the helloworld-app-demo/ directory.

apiVersion: template.openshift.io/v1
kind: Template
labels:
app: httpd-helloworld-demo
template: httpd-helloworld-demo
message: |-
The following service(s) have been created in your project: $.
For more information about using this template, including OpenShift considerations, see https://github.com/openshift/httpd-ex/blob/master/README.md.
metadata:
annotations:
description: An example Apache HTTP Server (httpd) application that serves static
content. For more information about using this template, including OpenShift
considerations, see https://github.com/openshift/httpd-ex/blob/master/README.md.
iconClass: icon-apache
openshift.io/display-name: Apache HTTP Server
openshift.io/documentation-url: https://github.com/openshift/httpd-ex
openshift.io/long-description: This template defines resources needed to develop
a static application served by Apache HTTP Server (httpd), including a build
configuration and application deployment configuration.
openshift.io/provider-display-name: Red Hat, Inc.
openshift.io/support-url: https://access.redhat.com
tags: quickstart,httpd
template.openshift.io/bindable: "false"
creationTimestamp: 2019-04-17T03:35:08Z
name: httpd-helloworld-demo
namespace: openshift
resourceVersion: "1726"
selfLink: /apis/template.openshift.io/v1/namespaces/openshift/templates/httpd-helloworld-demo
uid: <<find the UID of the image template from the above directory.copy and paste it here>>
objects:
- apiVersion: v1
kind: Service
metadata:
annotations:
description: Exposes and load balances the application pods
name: $
spec:
ports:
- name: web
port: 8080
targetPort: 8080
selector:
name: $
- apiVersion: v1
kind: Route
metadata:
name: $
spec:
host: $
tls:
termination: edge
to:
kind: Service
name: $
- apiVersion: v1
kind: ImageStream
metadata:
annotations:
description: Keeps track of changes in the application image
name: $
spec:
lookupPolicy:
local: false
tags:
- annotations: null
from:
kind: DockerImage
name:</Provide/an/image/to/be/used/here/$:latest
generation: 1
importPolicy: {}
name: latest
referencePolicy:
type: Source
- apiVersion: v1
kind: BuildConfig
metadata:
annotations:
description: Defines how to build the application
template.alpha.openshift.io/wait-for-ready: "true"
name: $
spec:
output:
to:
kind: DockerImage
name: </Provide/an/image/to/be/used/here/$:latest
pushSecret:
name: "quay-helloworld-pull-secret"(This needs to be created)
source:
contextDir: $
git:
ref: $
uri: $
type: Git
strategy:
sourceStrategy:
from:
kind: DockerImage
name: </Provide/an/image/to/be/used/here/$/httpd-24-rhel7:latest
forcePull: true
type: Source
triggers:
- type: ImageChange
- type: ConfigChange
- gitlab:
secret: $
type: GitLab
- generic:
secret: $
type: Generic
- apiVersion: v1
kind: DeploymentConfig
metadata:
annotations:
description: Defines how to deploy the application server
template.alpha.openshift.io/wait-for-ready: "true"
name: $
spec:
replicas: 1
selector:
name: $
strategy:
type: Rolling
template:
metadata:
labels:
name: $
name: $
spec:
containers:
- env: []
image: ' '
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 30
timeoutSeconds: 3
name: httpd-quay-demo
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 3
timeoutSeconds: 3
resources:
limits:
memory: $
triggers:
- imageChangeParams:
automatic: true
containerNames:
- httpd-quay-demo
from:
kind: ImageStreamTag
name: $:latest
type: ImageChange
- type: ConfigChange
parameters:
- description: The name assigned to all of the frontend objects defined in this template.
displayName: Name
name: NAME
required: true
value: httpd-helloworld-demo
- description: The OpenShift Namespace where the ImageStream resides.
displayName: Namespace
name: NAMESPACE
required: true
value: quay-helloworld-test
- description: Maximum amount of memory the container can use.
displayName: Memory Limit
name: MEMORY_LIMIT
required: true
value: 512Mi
- description: The URL of the repository with your application source code.
displayName: Git Repository URL
name: SOURCE_REPOSITORY_URL
required: true
value: https://github.com/openshift/httpd-ex.git
- description: Set this to a branch name, tag or other ref of your repository if you
are not using the default branch.
displayName: Git Reference
name: SOURCE_REPOSITORY_REF
- description: Set this to the relative path to your project if it is not in the root
of your repository.
displayName: Context Directory
name: CONTEXT_DIR
- description: The exposed hostname that will route to the httpd service, if left
blank a value will be defaulted.
displayName: Application Hostname
name: APPLICATION_DOMAIN
- description: GitLab trigger secret.  A difficult to guess string encoded as part
of the webhook URL.  Not encrypted.
displayName: GitLab Webhook Secret
from: '[a-zA-Z0-9]'
generate: expression
name: GITLAB_WEBHOOK_SECRET
- description: A secret string used to configure the Generic webhook.
displayName: Generic Webhook Secret
from: '[a-zA-Z0-9]'
generate: expression
name: GENERIC_WEBHOOK_SECRET

build_content/index.html

Create index.html in the build_content/ directory. We want a custom index.html page to show that we were successful in our deployment. This Page is what will be displayed when we access our Apache Web server after its been deployed

<html>
<header><title>Quay App Demo</title></header>
<body>
Hello <Your name>!
Let's test Quay.
</body>
</html>

CREATE GIT REPO

Create a new repo in Github using your own Git instance. Open a web browser and login to your Git repo using your credentials. Open the “New” menu at the top of the page, then click “New Project”. Set Git repo access to private.

COMMIT FILES TO GIT

Commit application template and content to Git. This is an optional step but one that is used to demonstrate having and using a Git repo as part of the CI/CD chain.

Note — In this example, the new Git repo is named “quay-helloworld-demo”.

$ git init
$ git add .
$ git commit -a
$ git remote add origin git@gitprd.<your.git.repo>:<username>/quay-helloworld-demo.git
$ git push --set-upstream origin master

CREATE QUAY REPOSITORY

Create a new Repository and robot account in Quay for this application. The robot account’s credentials will be used by OpenShift to pull/push images to the application’s private repository in Quay. This is for an automated CI/CD pipeline update and publishing using Github. Treat this as a type of Service account.

1. 	Quay web UI → Create New Repository → Enter “quay-app-demo” as the repository name → Leave Repository Visibility set to Private → Create Private Repository
2. 	Settings → User and Robot Permissions → Expand drop down menu → Create New Robot Account → Enter name for robot account → Create Robot Account
3. 	Toggle permissions for robot account to “Write” → Add Permission
4. 	Click Robot account name to view Docker login info → Docker Configuration Download ***-auth.json file and rename to config.json (place outside application template/content directory)

Pull secrets

Create secret using robot account’s Docker credentials and link to service accounts.

Note — The “ — for=pull” option should not be used when linking the secret to the builder service account.

$ oc create secret generic quay-pull-secret \
--from-file=".dockerconfigjson=config.json" \
--type='kubernetes.io/dockerconfigjson' -n quay-app-test
$ oc secrets link default quay-pull-secret --for=pull -n quay-app-test
$ oc secrets link builder quay-pull-secret -n quay-app-test

Git secret

Create secret for accessing GitLab with SSH private key. Add annotation to secret so that the secret will be used automatically by OpenShift when a build in this project references ssh://gitprd.ssc.tsc/*

Important — Modify path to ssh-privatekey as required so that it references private key used to access Git repo.

$ oc create secret generic gitlab-auth \
--from-file=ssh-privatekey=~/.ssh/id_rsa \
--type=kubernetes.io/ssh-auth \
-n quay-app-test
$ oc annotate secret gitlab-auth \
'build.openshift.io/source-secret-match-uri-1=\
ssh://gitprd.ssc.tsc/*' --overwrite=true -n quay-app-test

Deploy application

# Generate 2 different long random string of characters using the openssl command.
$ openssl rand -base64 40
$ openssl rand -base64 40
$ oc process -f httpd-quay-demo.yml \
-p=SOURCE_REPOSITORY_URL=ssh://git@gitprd.ssc.tsc/<repo/project>/quay-app-demo.git \
-p=GITLAB_WEBHOOK_SECRET=<1st long random string of characters> \
-p=GENERIC_WEBHOOK_SECRET=<2nd long random string of characters> \
-p=CONTEXT_DIR="build_content" \
| oc create -n quay-app-test -f -

Configure webhook in Gitlab

Configure webhook integration for the application repo in the Gitlab web UI. Value of GITLAB_WEBHOOK_SECRET must match value used when deploying the application in the previous step.

GitLab Repo --> Settings --> Integrations
Set URL = https://master.ocp-dev01-bna.ssc.tsc:443/apis/build.openshift.io/v1/namespaces/\
quay-app-test/buildconfigs/httpd-quay-demo/webhooks/<GITLAB_WEBHOOK_SECRET>/gitlab
Secret Token = (leave blank)
Trigger: Push events
Uncheck "Enable SSL verification"
Click "Add webhook"

APPLICATION BUILDS

Manually start build

A build can be manually initiated using the c start-build command.

$ oc start-build httpd-quay-demo -n quay-app-test

Rebuild application image

OpenShift will automatically rebuild the application container image anytime a new commit and push is made to the application’s Git repo.

# Update content
$ vi quay-app-demo/build_content/index.html
(update and save file)

# Note: If files are added, renamed, or moved, run “git add .”, then commit.

$ git commit -a
$ git push

Deploy new application image

Assuming that a new application container image has been built, it can be deployed by using oc import-image

$ oc import-image httpd-quay-demo -n quay-app-test

CONCLUSION

Congratulations on your deployment of the Red Hat Quay servers using Clair for image scanning. This software was designed to be integrated into the OpenShift platform. By following some of our examples and suggestions, you have experienced what resources are required to deploy Quay. You have learned how we deploy Quay and Clair as projects in the OCP environment. You have also been given the challenge of deploying a “HelloWorld” application using your Quay environment. While there are other ways of deploying Quay, hopefully this exercise gave you some insight for planning your supported Enterprise solution.

As containers pave the way forward for ever faster and more productive DevOps, more refined and comprehensive toolkits are required to keep environments secure. Lack of container image security leaves vulnerabilities which can be exploited, but failing to implement containers can significantly cripple an organization’s ability to move forward with high speed development and rollouts.

According to the Tripwire’s 2019 The State of Container Security Report, 42% of respondents said they were delaying container adoption due to security concerns, and 82% said they were considering restructuring security responsibilities and controls to prepare for container adoption or address existing issues with container security.

However, with a combination of Red Hat Quay and Clair, you can effectively secure outside container content and prevent security vulnerabilities while driving development forward in an agile environment.

ABOUT THE AUTHOR

John Lucas is a Senior Red Hat OpenShift Accredited Consultant for Stone Door Group, a Cloud and DevOps consulting company that helps enterprises successfully complete digital transformation projects. Stone Door Group offers rapid adoption of Red Hat cloud technologies with their OpenShift Container Platform Accelerator. To speak to John, drop us a line at letsdothis@stonedoorgroup.com.

References:

https://containerjournal.com/2019/03/22/the-4-most-vulnerable-areas-of-container-security-in-2019/

https://www.gartner.com/smarterwithgartner/gartner-top-10-security-projects-for-2019/

https://access.redhat.com/documentation/en-us/red_hat_quay/2.9/html-single/deploy_red_hat_quay_on_openshift/index

https://access.redhat.com/solutions/3533201

https://github.com/VeerMuchandi/QuayOnOpenShift

https://access.redhat.com/RegistryAuthentication

https://www.tripwire.com/state-of-security/devops/organizations-container-security-incident/