Automating container security in GKE with Binary Authorization and Circle CI

Published in

Google Cloud - Community

7 min readDec 20, 2018

How do we ensure that containers deployed to Google Kubernetes Engine (GKE) are from a trusted source and how can we automate this critical security check with our CI/CD tools?

Note: This article assumes familiarity with Google Cloud Platform, Kubernetes and related tools (gcloud, kubectl etc.). Familiarity with PGP key generation processes is also helpful.

Making a case for container security

Let’s take the example of a Healthcare application that is deployed in GKE. This app is used to manage very sensitive patient healthcare data.

In the figure below, you can see the standard CI/CD processes happening:

The code is checked into GitHub
A CI/CD tool (CircleCI) picks it up from GitHub, runs all the required tests and creates a Docker image and pushes it to the Google Container Registry (GCR).
It then deploys this image to GKE.

Now, let’s say that a privileged user within the organization decides to create another version of the application and deploys it directly to the GKE cluster without going through the CI/CD pipeline. This could range from something as innocent as testing out a new feature without impacting the production application to something more dangerous where the code is trying to access sensitive data for malicious purposes.

In any case, since this user already has access to the Kubernetes cluster, it is quite easy to do a local build of the application with the new code and deploy it to GKE and bypass the entire CI/CD process.

Problem: How do we ensure that the containers in our cluster are from a trusted source and have passed all the necessary checks including security vulnerability checks, etc.?

Solution: Binary Authorization by Google. It is a service that allows only “attested” images to be deployed to the cluster. An attested image is one that has been verified or guaranteed by an “attestor”. Any unauthorized images that do not match the Binary Authorization policy are rejected as shown in the figure below.

Terminology

Binary Authorization is a deploy time security service provided by Google that ensures that only trusted containers are deployed in our GKE cluster. It uses a policy driven model that allows us to configure security policies. Behind the scenes, this service talks to the Container Analysis service.
Container Analysis is an API that is used to store trusted metadata about our software artifacts and is used during the Binary Authorization process
Attestor is a person or process that attests to the authenticity of the image
Note is a piece of metadata in Container Analysis storage that is associated with an Attestor
Attestation is a statement from the Attestor that an image is ready to be deployed. In our case we will use an attestation that refers to the signing of our image

Incorporating Binary Authorization into our pipeline

To achieve this, there are 2 sets of steps that we need to follow. The first one is a one-time manual setup process and the second one can be automated (via your favorite CI/CD tool). I will show this using CircleCI later in this article.

It looks like a lot of work (and it is) but the good news is that much of it can be scripted and automated.

One-time manual setup: We will create a GCP project, enable the required APIs, create a Kubernetes cluster that has Binary Authorization enabled, set up a Note, generate the PGP keys and create an Attestor. For the purpose of this demo, I am using a barebones node.js Hello World app for deployment to GKE.

Let’s start with the manual setup:

Create a Kubernetes cluster and enable Binary Authorization

Enable the Container Registry, Container Analysis and Binary Authorization APIs for your project
The default Binary Authorization policy allows all images. You can access it by going to Security -> Binary Authorization in the GCP console

Note: The rest of the steps below are available in the script onetime_setup.sh

Create the Note payload. This note will be needed when we create an attestor. The note id should not contain spaces.
Send the payload to the the Binary Authorization service
Generate the PGP key pair (public and private keys) using gpg. The public key will be associated with the attestor and the private key will be used to sign the image. Note: gpg requires entropy (randomness) for generating the keys. As I did this on my Mac, I didn’t need to run a separate program like rngd to generate entropy. Also, gpg requires a passphrase during key generation. I chose a blank passphrase and accepted the warning.
The public key is exported to a file and saved for later use
Create an Attestor specifying the current project and the above Note. Associate the public key with the attestor. In my case the attestor is called “binauth-attestor”. There should be no spaces in the attestor name
Create a Binary Authorization policy (See sec-policy.yml). In this policy we will restrict all project level images by default and only enable images attested by our attestor (projects/nmallyatestproject/attestors/binauth-attestor) for our cluster called cd-cluster. Nice and restrictive!

After you run onetime_setup.sh, if everything goes well, you should be able to see that the binauth-attestor has been created and has 1 public key associated as shown below.

The Binary Authorization policy has also been changed to reflect our new restrictions as shown below:

Let’s move on to Automating the second part of the job: Signing images before deploying to GKE.

Automated C I /CD process: In this process, we will execute the standard tasks of building our app, testing it, dockerizing it and deploying it to GCR. Then we will create an image digest from the image, sign it with the private key from the above PGP key pair, create an attestation and deploy it to Binary Authorization.
Important Note: In order for CircleC I to be able to sign our image digest, it needs to be able to access the keys generated earlier. One way to achieve this is to base64 encode the private key and save it in a CircleCI environment variable for our project. For the purpose of this demo, we will be taking that approach. A CircleCI environment variable called BINAUTH_PRIVATE_KEY will contain this value. Please note, there might be other, possibly better solutions to do this.
You can find the key extraction logic in https://github.com/nmallya/containerdemo/blob/master/kube/extract_private_key.sh

The rest of the steps below are available in the script generate_signature.sh and will be automatically executed by CircleCI for every new build

Build/Test/Dockerize image/Push to GCR
Get the image path and digest from GCR and create a signature payload
Extract the private key from the Circle C I environment variable and store locally
Extract the PGP fingerprint from the public key
Sign the payload with a signature generated by gpg using the above inputs and for our attestor
Create an attestation with all the above information and submit to Binary Authorization
Wait for a short while (30 secs to 1 minute) for the changes to take effect in Binary Authorization
Deploy the app to GKE using the kubectl create -f command and passing in the yml for our containerdemo pod

cat << EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: containerdemo
spec:
  containers:
  - name: containerdemo
    image: "${IMAGE_PATH}@${IMAGE_DIGEST}"
    ports:
    - containerPort: 3000
EOF

Check the pod deployment status. If all goes well, you should see the output below. Congratulations!

Causing trouble: Let’s create a new image of this app without the signature part and see what happens when we try to deploy it.

Build, tag and push a new image to GCR

$ docker build -t containerdemo -f ./Dockerfile .
$ docker tag containerdemo gcr.io/nmallyatestproject/containerdemo:badimagev1$ gcloud docker -- push gcr.io/nmallyatestproject/containerdemo:badimagev1

Deploy to GKE — you will see an error below indicating that this deployment was “Denied by Attestor”. There were no attestations found that were valid and signed by a key trusted by the attestor.

Customizing deployment rules with other techniques:

What we’ve seen thus far is a very advanced way of ensuring trust (by disallowing all images at the project level and only allowing attested ones within a specific cluster) when deploying containers. We can also try other techniques that are very easy to achieve such as:

Whitelisting image paths in GCR. Any images present in this path will not be subject to the authorization check via deployment rules
Allowing / Disallowing all images for a given cluster etc

Resources

GitHub repository for this article. I’ll clean it up soon.
Binary Authorization official documentation from Google
Thanks to Brad Geesaman for his tutorial on Binary Authorization which was extremely helpful and provided a lot of the details that I’ve used in my article