Secrets management with Kapitan

Efficiently managing secrets is an essential part of any configuration management tool that cannot be left to the last minute. When evaluating a solution for Kapitan, we immediately looked into plug-in solutions like the amazing git-crypt which was at the time one of the suggested approaches for Helm.

There were a couple of things that we didn’t like about this approach, which made us think about the top requirements for the perfect solution:

Requirements

Encrypted secrets while you work: With other approaches, while the data is encrypted in the git-repository, it will be most likely unencrypted on your workspace while you are working with it. We wanted the secrets to be always encrypted on disk.

Granular permissions: While with git-crypt and similar you can encrypt different files with different keys, for the very nature of how Kapitan work this would not be possible. On compilation, all users would need to be given access to all files. We wanted to be able to give different permissions to different sets of users.

Seamless operations for unprivileged users: While we wanted owners of a specific environment to be able to access the secrets, kapitan should still work even for people with no access to such secrets. Privilege to encrypt/decrypt is only needed on creation/rotation/reveal of a secret.

Easily detect if secrets have changed: Other solutions do not give the user the ability to detect that a secret has changed. We wanted to empower the user with that knowledge, to prevent mistakes or abuses.

GitOps based: Secrets should live as files so that they can be handled as easily using a git based approach. Peer review and merge you secret changes! Declarative secrets :)

Easy secrets rotation: Rotating a secret should be super simple and its impact should be easy to track.

Offline operations: With the GPG backend, we wanted to offer an offline alternative that would not depend on solutions like Vault or KMS. While Vault and KMS are amazing, sometimes you do not want to maintain that solution or need offline operations. Also, this backup alternative offers you a way to store secrets to unseal that Vault of yours.

Secret auto-generation: We wanted to make it easy to generate the right kind of secret and avoid having to resort to having to cut and paste passwords or generate them manually.

Kapitan Secrets Walk-Through

In my previous post, looking through the code, it is already possible to see that we were already silently using the secret backend.

So let’s start from there. Head to my previous post and load up the environment.

Introduction to Kapitan

Let’s follow the example from the git repo. The katacoda example can come handy to follow on this post as well

Introduction to Kapitan by ademaria | Katacoda

Let’s start by compiling your configs, to make sure nothing has changed.

$ kapitan compile
Compiled prod-sea (0.44s)
Compiled dev-sea (0.45s)
$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean

Let’s go ahead and inspect the inventory/classes/components/tuna.yml

cat inventory/classes/components/tuna.yml
parameters:
  releases:
    tuna: latest
  tuna:
    image: alledm/tuna:${tuna:release}
    release: ${releases:tuna}
    replicas: ${replicas}
    args:
    - --verbose=${verbose}
    - --secret=?{plain:targets/${target}/tuna|randomstr}
kapitan:
    mains:
      - components/tuna/main.jsonnet
    docs:
      - components/tuna/docs/tuna.md

Before we dig deeper into what that secret tag means, there are a few more things I want you to see first.

First of all, let’s check the compiled manifests

$ cat compiled/prod-sea/manifests/tuna-deployment.yml
[cut]
spec:
   containers:
    - args:
        - --verbose=False
        - --secret=?{plain:targets/prod-sea/tuna:7d3096e2}

Notice that the secret in the manifests has the actual target name, and it has a hash at the end. More on this later.

We then see in the refs subfolder that a file structure mimics the name and secret name specified in the secrets tag:

$ tree refs/ # the `tree` command is not installed by default.
refs/
`-- targets
    |-- dev-sea
    |   `-- tuna
    `-- prod-sea
        `-- tuna

Let’s see what’s inside those files

$ cat refs/targets/prod-sea/tuna
data: cbA4OQFhKbVu2MJUf9ksn_HJ2gd6OXN3dT6MhKQPHPc
encoding: original
type: plain

I can hear you screaming. Whaaat? How is that a secret!

Kapitan secrets explained well

First off, Kapitan secrets are a “kapitan” feature. Meaning that do not come from either reclass, jinja or jsonnet. Because they appear as nothing but “text tags”, secrets are suitable to be used in other kapitan components like the inventory.

Kapitan supports (at the time of writing) 4 different secrets backends:

• GPG
• Google KMS
• AWS KMS
• Vault
• plain (or “plain text secrets”, see this issue for context before you laugh :))
• base64

In the documentation, you can find how to make use of all the different backends. I will cover the benefits of using them later on in this post. For now, grant me the license to use the `plain` backend so that I can explain all the functionalities in a way that you can easily reproduce without further setup. Check out the “advanced backends” for details on the capabilities offered by the other backends

A kapitan secret looks like this:

?{backend:path||functions}

The backend part can be plain, base64, gkms, awskms, vault or gpg

We recommend the gkms or the awskms because they are the easiest to work with. gpg unfortunately requires the user to manage their keys and can become a bit difficult to use and to get it to work, but it is useful in offline operations.

plain and base64 can be used for 2 purposes:

• Testing or demoing your work with other users
• To refer to non secret values that you might want to “pass in” from the command line. i.e. you can have your CI to inject the commit-id so that it can be used by kapitan as a reference.

The path part is simply the relative location within the refs directory. The path can be enhanced with reclass variables so that you can easy create per-target secrets. i.e. ?{gpg:targets/${target}/my_secret}

This is an absolutely *fundamental* feature whose usefulness might a first escape you. Perhaps I’ll touch on this later on.

As for the functions part, well you have already discovered the randomstrfunction. As the name implies, this function injects on creation a random string to the newly generated secret.

Let’s see that in action. Jump again to your katacoda session (or your local copy) and let’s play with secrets a bit more.

Rotating secrets

Rotating secrets with Kapitan is easy when using functions like randomstr!
Let’s start by nuking the secrets subdirectory and compile again.

$ rm -fr refs/
$ kapitan compile
Compiled prod-sea (0.33s)
Compiled dev-sea (0.38s)
$ git status
On branch master
Your branch is up to date with 'origin/master'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
modified:   compiled/dev-sea/manifests/tuna-deployment.yml
modified:   compiled/prod-sea/manifests/tuna-deployment.yml
modified:   refs/targets/dev-sea/tuna
modified:   refs/targets/prod-sea/tuna
no changes added to commit (use "git add" and/or "git commit -a")

With little surprise, Kapitan has recreated the secrets for both the environments. A closer look shows you also what this meant for the manifests:

git diff compiled/dev-sea/manifests/tuna-deployment.yml
diff --git a/example-3/compiled/dev-sea/manifests/tuna-deployment.yml b/example-3/compiled/dev-sea/manifests/tuna-deployment.yml
index a6f93bd..97e61b8 100644
--- a/example-3/compiled/dev-sea/manifests/tuna-deployment.yml
+++ b/example-3/compiled/dev-sea/manifests/tuna-deployment.yml
@@ -13,7 +13,7 @@ spec:
       containers:
         - args:
             - --verbose=True
-            - --secret=?{plain:targets/dev-sea/tuna:ee958e1c}
+            - --secret=?{plain:targets/dev-sea/tuna:2bb05264}
             - --brine
             - --canned
           image: alledm/tuna:v2.0.0

The hash for the secret has changed. This is helpful once again to evaluate not only the fact that a secret has changed, but also to understand the effects it created.

A similar behaviour happen when, instead of rotating and existing secret, we are creating a new environment which does not have a secret yet.

Let’s create a new demo-sea to see what happens:

$ cp inventory/targets/dev-sea.yml inventory/targets/demo-sea.yml
$ sed -i 's/dev-sea/demo-sea/g' inventory/targets/demo-sea.yml
$ kapitan compile
Compiled demo-sea (0.37s)
Compiled prod-sea (0.37s)
Compiled dev-sea (0.37s)
$ git status
On branch master
Your branch is up to date with 'origin/master'.
[cut]
Untracked files:
  (use "git add <file>..." to include in what will be committed)
        compiled/demo-sea/
        inventory/targets/demo-sea.yml
        refs/targets/demo-sea/
no changes added to commit (use "git add" and/or "git commit -a")

As you can see, kapitan takes care of creating the new secret file and injecting it with a randomly generated new string.

Important: When using other backends which support users and encryption, a user only needs the permission to use the secret to create, reveal or rotate secrets. This means that all your users can continue using Kapitan even if they will not have the ability to alter or read secrets!

Automatically generating secrets

Kapitan supports the following functions, and most can be combined:

• randomstr: i.e. randomstr:32 to generate a 32 characters secrets
• base64: i.e. randomstr|base64 to base64 encode a secret
• sha256: i.e. randmostr|sha256:${target} to sha256 with target name as seed (optional)
• reveal: i.e. |reveal:path/to/secret|base64
• rsa: i.e. |rsa:2048
• rsapublic: i.e. |reveal:path/to/encrypted_private_key|rsapublic

The functions are extremely versatile and easy to create. If you think of something new please send us a merge request!

Manually creating secrets

Sometimes you want to create a secret manually or from a pre-existing source, for instance if you want to create it from a given SSL certificate key, or other source. You can easily do this with kapitan refs --write:

kapitan refs --write gkms:shared/certificates/star_domain_dev_private -t demo-sea -f certificate.key

Revealing Secrets

Kapitan offers some other useful command line options to manage secrets. The most useful one is surely the kapitan refs --reveal option that allows you to actually see the content of your secrets.

$ kapitan refs --reveal -f compiled/demo-sea/manifests/tuna-deployment.yml
apiVersion: apps/v1beta1
[cut]
    spec:
      containers:
        - args:
            - --verbose=True
            - --secret=WFScxDXCdaVRs35UYc-xob0nhDCqxl7iy1XKt_Tq6EI
            - --brine
            - --canned
          image: alledm/tuna:v2.0.0
          name: tuna
          volumeMounts:
            - mountPath: /tmp
              name: data
              readOnly: true
      volumes:
        - configMap:
            name: readme
          name: data

Notice how until now, we never had to reveal a secret in order to operate kapitan. Your developers won’t even need to have permissions to read the secrets unless they need to create a new environment or rotate secrets. In that case, we suggest you implement CI/CD so that the CI user can create secrets for them.

The refs --reveal command is particularly helpful especially when deploying to Kubernetes. In fact, in the apply.sh canned script that we often use, we reveal the secrets from the manifests using kapitan before passing them to kubectl.

cat compiled/demo-sea/scripts/apply.sh
#!/bin/bash -e
DIR=$(dirname ${BASH_SOURCE[0]})
# Create namespace before anything else
${DIR}/kubectl.sh apply -f ${DIR}/../manifests/namespace.yml
for SECTION in manifests
do
  echo "## run kubectl apply for ${SECTION}"
  kapitan refs --reveal -f ${DIR}/../${SECTION}/ | ${DIR}/kubectl.sh apply -f - | column -t
done

Notice how no secrets are ever stored on disk in plain text (other than the plain and base64backend as explained)

Advanced Backends and user management

GPG backend

The GPG backend is the most ancient backend in Kapitan, but it is also very flexible and offers you offline operations.

Mind you, setting up the python environment on both Mac and Linux to make use of GPG properly might be a daunting task.

While I invite you to read the documentation about all the features, I want to point out that by setting the “recipients” list it is very easy to give permissions on a per-target level.

kapitan:
    secrets:
      gpg:
        recipients:
          - name: example@kapitan.dev
            fingerprint: D9234C61F58BEB3ED8552A57E28DC07A3CBFAE7C

As an example you can be to use different keys for different CI/CD agents (prod/dev/demo) so that one agent cannot apply by mistake (or read secrets) for the other.

*KMS backends

Both KMS implementations depend on your local environment being setup to use the appropriate KMS from your cloud CLI. Kapitan will transparently make use of it once the appropriate key path references are configured. See the documentation for more details.

parameters:
  kapitan:
    vars:
      target: ${target_name}
      namespace: ${target_name}
    secrets:
      gpg:
        recipients:
          - name: example@kapitan.dev
            fingerprint: D9234C61F58BEB3ED8552A57E28DC07A3CBFAE7C
      gkms:
        key: ${key_path}
      awskms:
        key: ${key_alias}

Alessandro De Maria - Medium

Read my other posts!

Let kapitan take the helm of Kubernetes

Introduction to Kapitan