Secret Diver — Searching for deeply hidden secrets

We are glad to introduce Secret Diver, a tool that allows you to search inside Docker layers for secrets.

TL;DR:

Docker images are composed of layers. These containers, even after modifications and updates, may have secrets hiding in previous layers. One often overlooked but vital practice should be to check and verify that these layers don’t expose your secrets.
We have built a tool that searches the different layers in a fun and easy way.

Docker images are actually OCI images that are comprised of multiple layers. When docker builds an image, a new layer is created for each command in the docker file. The end result (docker image) is a tar file that consists of all the layers in tar file format and an additional manifest.

You can read more about how Docker layers work in this excellent article.

A very useful utility to analyze the different layers of a docker image is “Dive” which allows to analyze container structures

Short example:

Shown below is an example analysis of a docker image consisting of 3 layers:

• The first layer imports the base image (in this case alpine)
• The second layer creates a .pypirc file with our secret inside
• This layer with the secret is needed only when building the image
• In a real-life scenario, between the second and third layers, you would typically find the logic supporting the building of the image, such as fetching internal Python packages, running installation scripts, and so on.
• The third layer deletes the .pypirc file

dive cidersecurity/secret-deleted-in-container

Running dive on a container

The final image built is uploaded to the container registry and becomes available for download. Once someone downloads the image and runs it in a docker environment, only the squashed file system is visible (without the secret in layer 2).

docker run -it --entrypoint=/bin/sh \
               cidersecurity/secret-deleted-in-container

Opening shell inside the container

So a question we often ask ourselves, when we are trying to be security-minded and ensure there are no overlooked exploitable oversights, is whether someone can still obtain our secret, despite it being deleted.

Regretfully, no surprises here. The answer is YES!!

We’ll now demonstrate how you too can replicate the method to obtaining deleted secret.

If we take the sample scenario above, there are essentially two ways to do so:

1. If you look closely at the “dive” above, we can view the commands that were run and, because we wrote the password as one of the commands, it is written to the manifest. The manifest contains all the commands and all the arguments that were used in each stage.
2. As mentioned above, the final image contains all the layers squashed together. An advanced user can extract any layer they wish from the image and read the file system contents of the layer extracted.

Let’s get our hands dirty:

1. We pull and save the container image by saving it as a tar file and then extracting the layers.

docker pull cidersecurity/secret-deleted-in-container
docker save cidersecurity/secret-deleted-in-container -o \
            secret-container.tar
tar xvf secret-container.tar

Extracting the image locally

tar xvf 7fb8df49962686...06d733/layer.tar
ls

Extracting the layer

3. When we view the contents, we see the deleted .pypirc we were looking for. Great Success!!

ls -alh tmp

Password found in file

Now the fun part begins — how do we now hack this PoC for scale and repeatability?

This got us thinking about automation. What if we could do this automatically and combine a secret scanner to scan each layer independently for secrets?

We are Proud to Introduce Secret Diver

We developed the Secret Diver tool, specifically to prevent the scenario described above. We often forget that when we delete a secret from the repository, it still exists in the log history of the repository. But hackers don’t forget that — it’s the first thing they check for. That’s why we have you covered with Secret Diver.

How does it work?

Steps to use the tool:

Manually

1. Install secret-diver from here — https://github.com/cider-rnd/secret-diver

go install github.com/cider-rnd/secret-diver@v0.0.1

2. Run secret-diver in human mode (machine mode is the default)

secret-diver -image=cidersecurity/secret-deleted-in-container -human

Running secret diver in human mode

Inside your CI

Run Secret Diver as part of the build process so that newly created images are scanned immediately after creation:

docker build -t <image-tag> .
secret-diver -image=<image-tag>

It is recommended to use the default machine mode which outputs to friendly SARIF output

2. All done, you are good to go. Pass this sarif file to your favourite issue collection system or just view it here

P.S. Going forward we recommend, from the security perspective, using multi-stage docker files which will upload only the final stage and automatically squash the previous stages into the base stage.

Have fun with the tool and do no harm!!!