From Python to Golang: New Connaisseur Release

Connaisseur 3.4.0: Minor Release, Major Change

From Python to Golang: Learn how and why this change improved security, performance and compatibility in Connaisseur’s latest release.

Connaisseur released version 3.4.0 and even though this is a minor release, there has been a major change: the programming language switched from Python to Go. This resulted in a multitude of improvements:

• Better security thanks to the use of FROM SCRATCH base image, getting rid of many dependencies
• Native integration of validation frameworks like Cosign, Notary and future ones, yet to come
• Smaller image size due to the removal of the Python interpreter and Cosign binary
• Increased performance with new Redis-based caching mechanism
• More stability through more robust goroutines

To better understand why we made this change, let me embark on a little story.

Created with DALL-E 3

Tough decisions to make

In software development it is not uncommon to start out simple — you have a specific problem and create a small proof-of-concept for solving it, trying to validate your initial ideas and then continue forward from there. Some ideas get fleshed out and new features are added over time, continuously building on the current state and organically growing the whole project into a full-fledged piece of software. Eventually, you might encounter problems that weren’t part of your initial assessment and whose solutions are difficult to incorporate into your existing code base, at which point you have a decision to make: find a quick but maybe hacky way of solving it, or rethink and rewrite most of your previous solutions so everything fits well together (or there may be something in between).

The right decision here is not obvious at all and is dependent on many factors, mostly on the time and resources that are available, compared to the benefits gained from investing them. So even if a “clean“ rewrite of your solution may feel like the right decision, it is not worth anything should you never have the time to finish it. In contrast, always choosing the quick way will bloat your code with technical debt that becomes increasingly hard to manage.

Those are the kinds of decisions we also had to make while developing the Connaisseur project, the Kubernetes admission controller for verifying container image signatures. While doing so, at times we did go for the quick and hacky way, while at others we re-evaluated our solutions and made major rewrites of our code base. This time, we took several steps back and re-evaluated one of the very first decisions we made back when the project started: which programming language should we use?

From Python to Go in 5 seconds (or more)

Back then, Python was the obvious choice as a programming language since everyone within the team knew it, but over time it became clear that we had to make compromises because of that choice.

Connaisseur was created with Notary as the only validation framework in mind, until Cosign entered the scene. Our integration of Cosign works by including its binary inside our container, since we didn’t see value in writing our own Cosign Python library. A choice that led to quick results, but made maintaining it cumbersome in the long run. Speaking of the Connaisseur container — inside you will find a full Python interpreter, whose only purpose is to run Connaisseur, although it could do much more. That means we leave a very powerful tool inside our container, just so we can conveniently run our application.

Now let’s look ahead to what the future of supply chain security brings us 🔮. I can see something! It’s Notation that also wants to be integrated into Connaisseur and doesn’t have a Python library as well. Looks like we are out of luck.

Created with DALL-E 3

As you can see, many reasons ranging from security to maintainability issues piled up over time that made Python a questionable choice. As a replacement, one language stood out from the rest: we wanted to go with Go!

Rewriting Connaisseur in Go was very tempting for several reasons:

• Compilability: Golang is compilable and having a compiled Connaisseur binary in the end would allow us to get rid of a great number of dependencies in the Connaisseur image. We hoped to be able to drastically decrease the image size and, more importantly, reduce the attack surface to an absolute minimum by making use of the possibility of building our image from scratch — and this dream came true: The Connaisseur image size went down from 193MB to 92,5MB and now only includes the new Connaisseur binary. No more Python interpreter or OS packages that can potentially be abused for attacks.
• Integrability: Along these lines, we were especially looking forward to eliminating the need to use the Cosign binary, embracing the native integration with the Go-written Cosign package offering access to many more functionalities bearing advantages for both end users but also us maintainers. However, during implementation, we noticed that not only Cosign but also the Notary integration benefits from having direct access to the package’s Go API. One example: Suddenly, it was a no-brainer to broaden the supported key types to cover all TUF-compliant key types. We haven’t even come to make use of all the new possibilities yet, so stay tuned: More features are on the way!
  After all, our GOal to GO with the GOrgeous Go-GOverned Kubernetes eGOsystem (eh.. ecosystem of course) really paid out.
• Robustness and performance: Sporadically, Connaisseur was facing stability issues for two different reasons:
  Once, there were issues with the event loop used for parallelization that we put quite some effort into, but although we were able to solve these issues in the cases we didn’t fully eradicate them. Parallelization based on Goroutines worked out of the box and like a charm (at least up to now).
  Secondly, Connaisseur needed its time. Despite several performance improvements, in some rare cases, Conaisseur still exceeded the maximum waiting time of Kubernetes admission controllers. The performance of the new Connaisseur release improved significantly, not only tackling the aforementioned timeout issues but also smoothening the user experience in general. First quick measurements testing a sample deployment covering both, containers as well as initContainers based on 10 unique images show a performance gain by a factor of 5.

In the end we committed to the Go switch, knowing it would take a lot of time and are very happy with the result!

Created with DALL-E 3

But wait, there is more

Besides the advantages we gained from the language switch, we didn’t pass up the chance to implement some extra features, two of which had been on our list for quite some time.

• We introduced a Redis-based caching mechanism that allows storing digests of validated images for 30 seconds. Before, deployments of workload objects with multiple container instances of the same image entailed multiple network calls to the registry or Notary retrieving the exact same signature data. Admitting an image within 30 seconds of the initial validation allowed a huge performance gain for these kinds of deployments without compromising security with regard to rollback attacks.
• It was brought to our attention by several users (🙏) that Connaisseur was quite difficult to integrate with GitOps solutions. This is due to Connaisseur being a mutating admission controller, mutating the image references at all places by replacing the human-readable image tag with the SHA256 content hash, the image digest. (Why Connaisseur does this is explained here.) As a consequence, the workload object description inside the cluster differs from the workload description from the git repository that has originally been applied even though the referenced workload object itself is exactly the same. However, the core job of GitOps solutions is to detect exactly those diffs and to sync the workload objects from the git repository to the cluster. The result was an infinite loop of GitOps reapplying content from the git repository and Connaisseur changing it.

The old Connaisseur applied this behavior to all workload objects, be it Deployments, ReplicaSets, Jobs, Pods… With the new release, Connaisseur now offers a mode to only mutate Pod resources. This mirrors the behavior of other admission controllers e.g. the Kubernetes native PSA (Pod Security Admission). As for every workload object the actual core consists of one or several underlying Pods, this does not come with any decrease in security.

However, in terms of usability, one could argue for one or the other. With Pod-only mutation, Connaisseur will officially admit the parent workload object, say a deployment even though the Pod resources inside the deployment are being denied, so the deployment is being admitted while, at the same time being rendered dysfunctional, due to the denial of the actual Pod resource doing the work. For human users, this won’t be a problem, because there is a warning being issued, but it might not be the best solution for CI/CDs only watching for the status code. This is why we made the pod-only mutation optional, being configurable in the values.yaml configuration file.

Clara Futura Est

As you can see, a lot has changed with release 3.4.0 of Connaisseur, but we are not done yet! There are many changes yet to come:

• First of all we want to make the Cosign integration feature complete, meaning we finally want to implement keyless signature verification for Cosign. This will be another step towards a more seamless integration of supply chain security inside your organization, as you no longer need to worry about key management.
• Similarly we want to look into OpenPubKey, as it will be the way to go for verifying Docker Official Images, giving you more baseline security without the need for any configuration.
• As already mentioned, Notation is yet another signature verification framework that we aim to support, giving you more choices on signature solutions you want to use, or even mix and mesh.
• We are also looking into streamlining some of our configuration options that might have been confusing to use (looking at you alerting), but that will be part of a future 4.0.0 release since this will incorporate breaking changes.

In the meantime, we are happy for any feedback from the community on how the Golang switch turned out for them and what other features they would like to see implemented. If you are interested in helping us but haven’t dared so far because you don’t feel comfortable with Python — now is your chance to Go!

Created with DALL-E 3

Hopefully you could take away some insights from our little journey from Python to Go, perhaps looking at your own projects and taking a few steps back, reconsidering the decisions you have made and why you made them. It doesn’t need to be as drastic as we did, by switching out the entire underlying programming language, but reworking smaller parts of existing solutions may improve your overall project bit by bit.

Take care and cheers.