A work around for leaky build secrets.

avi

Follow

Feb 16, 2018 · 5 min read

Managing build time secrets can be a huge pain with docker. If you’ve built a docker image with private repositories, you might benefit from reading this.

Skip to the meat

There are multiple solutions floating around the internet for this problem. We’ll start with the obviously bad ones and work our way up.

Copy the SSH key

This is a really bad idea. Any secret keys in the final Docker image. You might delete the secret after using it and think you are safe.

The secret used becomes the part of the build history.

Pass token as a build time arg

This is not a secure solution either. As the official docker docs say:

Warning: It is not recommended to use build-time variables for passing secrets like github keys, user credentials etc. Build-time variable values are visible to any user of the image with the docker history command.

It is not possible to pass secrets securely, cleanly to docker builds. This issue filed on Github goes more into the details why existing solutions (and hacks) are bad.

Multi stage builds

As this article suggests, you can use multi stage builds and use the intermediate build stage to pull all dependencies and pass them to your final image.

The intermediate build stage exists in the system long after you have left.

This solution does not work well and is riddled with traps if your private dependency has some setup work. You could easily miss copying a python/node package’s dependency.

Subtle differences in the build environment and running environment can result in failed builds and hours of frustration.

State of things at Verloop

We use Container Builder to build docker images. GCP does not provide any solution for this problem.

We could build the Go binary and move it to the docker image. But that just solves it for one language on the happy path.

This solution would not work for our python and node servers.

pip install and npm install need access to private repos from within the container.

We needed something that would work always, with all current and future languages we dive into.

Invert the problem

When really backed against the wall, one exercise we often try is to invert the problem on it’s head.

What if leaking the secret was not a big deal?

As soon as we asked this question, an old idea from our documentation server came to mind.

Back when we needed to centralise documentation of all our services, we needed to pull code from multiple Github repos in every build.

Maintaining these many deploy keys wasn’t ideal. A little known feature of github integrations had come to our rescue.

Short lived access tokens!

Fidelius

In under 100 lines of code, Fidelius completely does away with the problem of having leaked access tokens. It produces a token that becomes safe to leak after an hour!

If the tokens mean nothing, you can leak all of them!

Fidelius shares a secret with you that you can’t share with others, because sharing it is useless.

How Fidelius works?

Fidelius uses a companion Github Integration. It fetches a temporary Github access token. This token expires after one hour.

We use that token in the Cloud Builder or within the Docker build.

Fidelius can also generate a valid .gitconfig file which you can use or update an existing .gitconfig.

The build history and the final docker image will have an expired token.

Since the process gives the app read access to your repository, you should create your own app.

DIY steps

Create Github App

Create a new Github App with permission Repository Contentsset to Read Only.

You can check an example app at Fidelius Charm

On page https://github.com/settings/apps/<your-app>, scroll down to the About section.

Note the Github app ID, this will be useful later.

Generate Private key

Scroll further below and click Generate private key

Install the app

Scroll back up and click installations.
You should now be on the page https://github.com/settings/apps/<your-app>/installations

On the next screen, select the repositories you’ll need in your build and click install.

Copy the installation ID

On the next screen, you will see your installation id as part of the URL

https://github.com/organizations/<org>/settings/installations/88254

Here 88254 is the installation id.

You can also go back to your app page and check it in the webhooks github tried to deliver under “Advanced”

Using Fidelius

Make Fidelius part of your build step, either generate the access token or place the .gitconfig at an appropriate place.

If you want to pass the token to docker build, you can do something like this:

$ fidelius --gh-integration-id=9045 --gh-installation-id=88254 --gh-private-key=private_key.pemv1.20dfff65bb3cff4cbe2330ef10f41d43b441a772

Yes that’s a real token that gives access to all of Verloop’s secrets!

If you pass a path a --git-config-out argument, it will write appropriate git config to that file:

$ fidelius --gh-integration-id=9045 --gh-installation-id=88254 --gh-private-key=private_key.pem --git-config-out=/root/.gitconfig

Disclaimer

If you are going to release the image to the public, use a trigger that is set to fire at least an hour after the image is built.

But my code is not on Github!

If your code is not on Github, you can still use this design with a reverse proxy server to give out timed tokens.

The idea of using short lived tokens to make password leaks irrelevant is liberating!