Keep your ship together with Kapitan

NEW: Katacoda scenario!

UPDATE: Generators have been ported from jsonnet to kadet (python)

Manage complexity with Kapitan

We open sourced Kapitan back in October 2017.

In the 12 months before that date, Kapitan helped us revolutionise the way we were running things at DeepMind Health, and allowed us to keep control over many heterogeneous systems: Kubernetes, Terraform, Documentations, Scripts, Playbooks, Grafana Dashboards, Prometheus rules: everything was kept under control from one single point of truth.

I am not afraid to say it loud: there is nothing out there which is as versatile and powerful as Kapitan for managing configurations of complex systems. There.. I said it. Prove me wrong :)

Having a product so radically different than anything else out there, obviously also meant that we had to learn and discover how to use it: Patterns, Best Practices.. We had to recognise and discover them as they surfaced while refactoring the mess we made with our initial rollout. Fortunately one of Kapitan’s strength is to make refactoring a joy, and so we did, and did it again, until we came out with a nice set of best practices.

What we didn’t do, was to make them available to others… until now. Spoiler!

Joining Synthace last year as Head of SRE also meant I had a chance to apply those best practices and approaches to a fresh new environment. This gave me the opportunity to test them and improve them much faster than what I would have been able to do previously, due to the much faster iterations we have there. The results were spectacular, allowing me to bring control in a place where manifests were generated using unmaintainable go code, secrets were managed manually, and each Kubernetes cluster was a snowflake.

I introduced successfully Kapitan, but I was still working way too much with jsonnet, and needed to have a jsonnet file for each of the (almost identical) 20 microservices we had.

So I had a though: what if I could replicate the full setup without touching any code at all? Introducing Kapitan Generators

Kapitan Generator Libraries

Today I will give you a preview of how we use Kapitan internally at Synthace.

I am also pleased to announce that we have released some of our internal kadet (ported from jsonnet) libraries we developed at Synthace as open source!


In particular, we will release:

  • [RELEASED] A kadet manifest generator library to quickly create Kubernetes “workloads” manifests by simply defining them in the inventory. Get started with something as simple as:
  • A kadet pipelines generatorlibrary to quickly create Spinnaker pipelines for the above defined workloads
  • A kadet terraform generator library to create Terraform configurations.
  • A set of helper scripts which will make easy to get up and running with Kapitan.

To set expectations right, these libraries will be released in a form that will probably require some refinements, but it should hopefully allow you to get started and inspire you to contribute with your libraries or implement the same approach to manage your system of choice.

These generators are a huge step forward from our approach where you would need to create a jsonnet/kadet component file for each service you wanted to manage with Kapitan. The ambition is to allow you to get started and generate configuration for you 80% of cases, and at the same time enforce some sane best practices along the way. Of course, you can still extend them or write your own generator using jsonnet/kadet if you need something fancier or want to have full control on a specific component.

A sneak peek — generating manifests

So let’s say that you want to get started with Kapitan. Until now the steps to get you started where quite a few, often cryptic and not well documented.

We have releasing a “kapitan reference” repository ( with all the batteries included. I will run this session assuming you have this already.


  • docker
  • gcloud (the example is on GCP)
  • kapitan
  • yq
  • kapitan generators (released)

Suggested read:

Your first target file

Note: Since we have released the Manifest Generator, you will now be able to follow these steps

From your kapitan-reference repository (, go on and create a first devtarget file: inventory/targets/dev.yml. For simplicity, we won’t be creating inventory classes right now, so we are going to edit the target file directly.

Make sure it has the following content:

Now run: kapitan compile --fetch

The --fetch command will make kapitan download the latest libraries and supports scripts that we package. Some of the third-party libraries we use are kube.libsonnet and spinnaker/sponnet.

If you now check your git repository, you will find that Kapitan has generated for you some files:

Let’s look at what we have! Have a look at echo-server-bundle.yml

Admit it. That was quick!

The manifest generator loops through the keys of thecomponents inventory hash, and generates a new set of config for each one it finds, in this case, echo-server (from:

As anticipated, the generator library tries to be smart and adds some best practices that you may or may not like for all your services.

Why would you want a Deployment without podAntiAffinity? I’m sure there are valid reasons, but let’s make it a default, shall we?

Exposing the service

The deployment looks good, but it is missing some essential parts. Ehm.. we need a service! Right, let’s do that and recompile with kapitan compile

Adding the port definition will produce the following:

It will add the port definition to the container

And it will create a new service definition

Are we there yet? Not quite:

  • The service assumes that the echo-server runs on port 80. From the documentation, it looks as if the service is actually running on port 8080 instead.
  • We would want the service to be exposed using a LoadBalancer service, so let’s change that.
  • We would like a readiness probe

Have a look at the bundle again:


Adding Environment Variables

What else could we do? Well, from the echo-server docker page it looks as if we can play with a few parameters to change its configuration. Let’s add some env variables.

As expected, the changes are reflected in the manifest:

Adding secrets

Just for the sake of testing, let’s also add a secret to the setup, even if the component won’t be using it.

Let’s break this down:

  • ?{plain:targets/${target_name}/echo_server_password||randomstr} will create a random string, and store it in on git. Because we have used the plain backend, we will be storing it in cleartext. User gkms or other secrets backend if you care about your secrets
  • The SECRET_PASSWORD env variable will have the content of the generated password. Now because we have decided to test the plain backend, you will see it in clear text in the manifest. Otherwise if would be encrypted and you would only see a secure tag.
  • The items instruction will also mount the secret as a volume, and only exposed the selected item. This means you will be also able to access the content of the secret using /opt/secrets/echo_server_password

The result of the compilation adds a new file:

Also notice that the files are all nicely and consistently named after the service.

Lastly, we can move the component definition in its own class file: inventory/classes/components/echo-server.yml

And then we can simplify the target to reference the component

This way we can now reuse that component across other targets, like for instance inventory/targets/production.yml

Running kapitan compile again will effortlessly generate the new files for the new target prod:

which produced:

Final words

We are aiming to release the kapitan reference repository with all the goodies within the next few weeks. Please contact us directly on the #kapitan kubernetes slack if you want to help us to work on releasing it, and if you want to take part of an initial alpha release.

Kapitan Blog

Kapitan Community Blog

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store