Declarative secret management for GitOps with Kapitan

Image for post
Image for post

In this post I would like to compare different ways to manage secrets in code, especially relevant to Kubernetes but also in general with a more broad reach.

I have already introduced with “Managing secrets with Kapitan” the approach we promote, but with the release of Tesoro, our new “secrets” admission controller, it’s time to review the different approaches and show why we prefer our way.

When talking about approaches for managing secrets, the solutions that normally come to mind are the excellent Mozilla Sops and Bitnami Sealed Secrets (I will get to Vault in a second).

For Mozilla Sops, an excellent introduction is

For Bitnami Sealed Secrets, you can have a look here:

However, while they are excellent for handling already existing secrets, they add nothing to the process of creating the secrets or rather, capturing the definition of what the secret content looks like.

I’ll use an example.

With both tools, you need an already have the secret at hand, and then you can protect/secure/encrypt it using:

Let me now ask you some questions:

  • How do you know how to create the secret?
  • Was it a password? A jwt? a service account json file? A token?
  • Imagine it has expired. How do you renew it? How do you rotate it?
  • Can you easily re-use the secret from a script than needs it?

I also leave it entirely as an exercise to the reader to understand the implications around how the secret was created, given it had to be somehow manually crafted with plain text, copied in clipboard, etc etc before it was handed to the magic of encryption.

So from my point of view, here is where these approach leave you high and dry: they do not help you to manage the entire lifecycle of a given piece of secret, but focus on the secure storage part of the problem.

For some, the limitations of the above approaches have to do with the design ideal intention of keeping the above tools generic and isolated from the concerns of where the secrets come from. Fine, I like that. But don’t we still lack an overarching tool that can glue all of that together?

Popular Kubernetes tools like Helm and Kustomize come short of offering anything suitable other than offering very primitive approaches (see example), effectively passing again the buck to other (non-existent) tools.

Now I can hear you all screaming “Vault! Vault! Vault!” and indeed the mighty Hashicorp Vault does enable some patterns that allow you to handle both storing secrets and also define how they are generated. As an example, the excellent Database Backend support, Certificate generation, and overall the ability to generate random strings/passwords.

It is also true, however, that Vault does introduce a great deal of complexity to the solution, and while for some it is the perfect compromise of a complex solution for a complex problem, for the vast majority of smaller setups Vault adds a burden that some people just do not want to deal with.

Let alone that as soon as you add Vault, IMHO you can kiss goodbye to your GitOps approach because you are handling the responsibility around secrets to a runtime event/service/side-container or whatever. (And this may as well be perfectly fine, I am just making a point for the fellow GitOps fundamentalists like me)

I have already introduced how Kapitan manages secrets in the above blog post, but I will go through some of the concepts and contextualise it to the topic we are discussing.

Important note:
In the Kapitan lingo, we to use the more generic word reference instead of secret because a reference can also be valuable even in cases where it is not encrypted and used as plain text.

Brief explanation of how Kapitan references work

A kapitan reference looks like this:

The backend part can be plain, base64, gkms, awskms, vault or gpg

The path part is essentially the way we refer to it, which maps to a ref file on disk.

The functions part allows us to define how to define or transform the secret.

An example reference definition could be:

Which translates in plain english to:

Using Google KMS for encryption, make this a reference to a payload stored on disk at the following location:targets/${target_name}/password.
Should the reference not already exist, generate a random 32 characters string and store it in base64 format, otherwise use the existing one.

Although Kapitan is currently needed to make full use of references, we are also working on a smaller tool so that you can use Kapitan secrets in non Kapitan setups.

When we compile a kapitan target (e.g. the dev target) that has such reference in one of its components, the resulting compiled file will contain the actual compiled tag, for instance ?{gkms:targets/dev/password:abcd123}

The hash at the end of the tag allows you to detect whether the payload of the secret has changed.

A file usually in refs/targets/dev/password will instead contain all the actual secret payload and metadata:

The behaviour changes if we decide to run kapitan with the--embed-refs option, which will inline the encrypted secret payload directly into the manifest. We will discuss later when we talk about Tesoro.

Putting kapitan secrets into action

Now that I have explained the basics, let’s dive into the important aspects of this solution.

#1 — A reference tag refers to the payload of a single “item”.

As a comparison, with Sealed Secret and SOPS we saw that we encrypt the file containing the secret, but we have no way to then refer to that specific content.
For instance, once I encrypt a mysql-secret.yml file with either tools, I do not have a way easily refer to its content, let alone a single key.

With Kapitan, ?{gkms:target/dev/mysql_password} can instead be referred to individually and used from within a script or another component.

#2 — A reference can be referred to multiple times.

If you have a secret to be shared across multiple deployments (e.g. a reference to a shared Github token ?{gkms:shared/github_token_ro}), you can go ahead and add it to all those at once.

As an added bonus, if you update the one github token reference, all the affected compiled target files will update accordingly.

#3 — A reference can be create/updated or read from the command line

You can inject a value into a secret using

You can read the content of a secret using

You can also reveal the content of a file/manifest that contains multiple secrets by simply running

#4 — You only need secret access to reveal or create, not to use Kapitan

All of the above, CLEARLY, works provided you have the right access to the backend used (in the example, gkms)

However, the magic is that you do not need to be granted GKMS access for any non-secrets related operation. So that any developer/user can use kapitan, compile, make changes to components and targets without any need to be exposed or allowed to secrets.

Practical examples

Learning from the above 3 points together, and using kapitan and its ability to template anything, it become super easy to create helper scripts that can simplify the creation of secrets beyond the native functions provided by kapitan (i.e. randomstr and others)

In the future, reference functions will be extended to allow to run external script, so to remove the need to manually invoke helper scripts.
Instead, you will have something like

Why scripts? Because scripts are the simplest way to capture all the complexity of how the secret is actually defined, and will be checked in in git alongside the actual secret payload it generates. You can comment them, you can review them, you can add them to your playbooks.

Writing references from scripts

For instance, here is an simple example how I generate a self signed certificate to be used by tesoro: (

The script is automatically generated for each target containing the tesoro component from a template.

Every time I run it for a given target, my secrets gets correctly generated and added to the references so that the tesoro component can refer to them.

Note that by using templating, I do not need to pass any parameter to it, because everything is discovered directly from the inventory, making it super simple to use.

This avoids us from having to run things like:

Which could simply again fail to capture the full aspect of the secret creation, leaving space for mistakes and errors.

Another example shows how I can load a GCP service account into a secret ( Note (again) that by using templating, I do not need to pass any parameter to it, making it super simple to use.

Using references from scripts

You can also create scripts that can read from references and use them to access their content. For instance, we have a tiny wrapper script that allows developers to easily connect to a database simply running the following script:

Tesoro — Kapitan Secrets Controller for Kubernetes

We have discussed how we create, store and use secrets. But how do we securely use then within a Kubernetes setup?

Until recently, the only way to apply secrets to make use of a wrapper around kubectl, that would essentially run the following

this was wrapped into a convenient templated script like

However this approach required for the Kubernetes operator (be it human or CI) to have the keys to decrypt the secrets.

Tesoro Admission Controller

So taking inspiration from Sealed Secrets, we developed Tesoro, a Kubernetes webhook admission controller responsible for securely decrypting secrets in-cluster.

Unlike Sealed Secrets, Tesoro supports all backends already supported by Kapitan: Google KMS, AWS KMS, vault, gpg to name a few.

Tesoro works by intercepting the manifest creation and automatically and transparently extracting the secret payload from a standard Kapitan reference and injecting it into the Kubernetes resource.

For Tesoro to work, the manifest needs to embed the encrypted secret payload, that can be achieved by running kapitan with the --embed-refs option.

An example of an embedded secret is here.

A very interesting aspect is that Tesoro supports injecting secrets into any type of Kubernetes resources, and so not only in Secrets objects.
This means that you could inject a secret directly into a ConfigMap, making it possible to do things other solutions simply cannot.

Objects managed by Tesoro will show an annotation with details of what Tesoro did:

Using Tesoro, the “reveal” part of Kapitan is not needed anymore, and any user/CI can deploy to a Tesoro enabled Kubernetes cluster without the user being given access to the secret key to decrypt secrets.

Final words

If you have had the chance to read the blog post and the other posts available in the Kapitan blog, you will probably be appreciating the features and the flexibility of Kapitan in any aspect of configuration management.

The secrets management alone sets it apart from any other tool out there.

If you are interested in learning more, please read

Kapitan Blog

Kapitan Community Blog

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store