Integrating Helm into the sigstore Project

Helm has simplified how applications are deployed and managed in Kubernetes. As the de facto package manager for Kubernetes, Helm enables the orchestration of complex deployments by running only a few commands. But, given that Helm is yet another component of the software supply chain when deploying applications to Kubernetes, it is imperative that steps be taken to assess the resources being deployed and ensure they have not been tampered with. In this post, we will introduce a set of tools from the sigstore project — and when combined with existing features of the Helm project — improve the security posture of Helm charts which can then be integrated into any software security toolchain.

Helm Provenance

Asserting the integrity of a piece of software is one of the most common tasks when securing the software supply chain. This process typically involves the verification of a digital signature that has been placed on the software asset itself or alongside. Helm has long since provided support for signing and verifying Helm charts through its set of provenance tools. At the time of chart packaging, a separate provenance file is created alongside the package containing several pieces of information — including a PGP based digital signature that can be used to assess the validity of the chart.

All that is needed to sign a Helm chart is a valid PGP keypair, and of course, the Helm CLI. On the consumption side, similar to verifying any other type of software that has been signed, only the public key associated with the private key is needed.

But, if signing and verifying can be accomplished by creating a PGP keypair and running a few commands, can it not be integrated into an existing software supply chain?

It can, but we can do better.

One of the inherent challenges when operating in a public/private key infrastructure is that the consumer places a level of trust on the validity of the public key. Several high profile security incidents, including the SolarWinds security breach (known as Sunburst), were partially attributed to a misconfigured PKI infrastructure. In the case of SolarWinds breach, the attackers gained access to the code servers and distributed software that had been signed with a compromised PKI. Since the consumers trusted software signed with these keys, they willingly installed the malicious software, unbeknownst to them.

So, what can we do to reduce this type of situation from occurring with our Helm charts? This is where the sigstore project can help.

What is sigstore and how can it help?

Sigstore is a Linux Foundation project which contains tooling that is focused on making it safer to package and distribute open source software and can be used by anyone participating in the software development and release process. Included in the sigstore ecosystem is Rekor, a transparency log and timestamping service that enables software publishers a location to publish signed metadata produced by software maintainers. This immutable tamper resistant ledger provides a mechanism that allows consumers to make informed choices in the software they are about to use.

A publicly hosted Rekor instance is available for anyone to use at https://rekor.sigstore.dev. In addition, you can also deploy a self managed instance of your own instance in your environment (A Helm chart is available and the steps on how it can be deployed to a Kubernetes cluster is described here).

Integrating Helm with sigstore

Entries within the Rekor transparency log consist of one of the available types. One of the benefits of Rekor is that it employs a pluggable model for types which allows anyone to define their own extension points and extend the existing set of types. Rekord is the default type, but there is a growing list of types supported by Rekor. Given the fact that it is relatively easy to integrate with the types found in Rekor, a new type supporting Helm was introduced.

All Rekor types extend the default base schema (also called a ProposedEntry) and one of the primary goals when implementing the schema for Helm support was for it to be modeled using the same provenance schematics that are provided by Helm natively. This included signature support using PGP and capturing assets that are essential to the Helm signing and verification process. The result was an entry consisting of the following:

• The provenance file for the signed Helm chart (captured but not stored)
• The detached signature included in the provenance file
• Public key associated with the private key used to sign the chart
• The checksum of the packaged chart

Each of these data points are readily available and familiar to anyone who has worked with signed Helm charts. The somewhat challenging part is to apply the appropriate format to comply with the Helm schema. Rekor does provide a command line interface to interact with the exposed API which can be used for submission, but for custom types, like Helm, there is an assumption that entries are properly formatted beforehand. In order to streamline the interaction from an end user perspective and to simplify the integration between Helm and sigstore, a purpose built tool would need to be created.

Introducing the Helm sigstore Plugin

Helm has the capacity for access tools from the Helm CLI that is located outside of the core codebase through the use of Plugins. The plugins themselves are ordinary executables, but the plugin system provides the capability to not only invoke them from within Helm, but also manage their lifecycle, including the installation and update process.

The Helm sigstore plugin is a Helm plugin that brings the capabilities provided by the sigstore project (specifically Rekor) to the Helm ecosystem. Not only can it aid in the publishing of Helm chart content to a Rekor server, it can also be used to verify previously published resources.

Like any other Helm plugin, installation is a breeze. Assuming you have Helm previously installed and configured, the Helm Sigstore plugin can be installed by executing the following command:

$ helm plugin install https://github.com/sigstore/helm-sigstore

Confirm the plugin was installed correctly by running helm sigstore which will display all of the available subcommands. The capabilities provided by each subcommand is as follows:

• upload — Upload an existing signed packaged Helm chart to Rekor
• verify — Assert the validity of a signed packaged Helm chart based on previously published content from Rekor
• search — Determine whether a signed packaged Helm chart is found in the Rekor server

To demonstrate the full capabilities provided by the Helm sigstore plugin, let’s walk through how one would make use of the plugin — either as a Helm chart producer or consumer.

Using the Helm sigstore Plugin

Multiple personas can play a role when using the Helm sigstore plugin. Helm chart producers would use the upload capability once the chart has been created, packaged and signed. Helm chart consumers would then be able to determine whether the chart they are about to install is located in the Rekor server and then subsequently verify the content of the chart.

First, acting as a chart producer, create a new Helm chart called “helm-sigstore-example” which will be used to demonstrate how Helm charts can be uploaded to Rekor using the Helm Sigstore plugin:

$ helm create helm-sigstore-example

Feel free to modify the contents of the chart created by the prior invocation as you see fit. The next step is to package and sign the chart. Be sure that you have a PGP keypair available along with any of the other prerequisites necessary for signing Helm charts.

Sign and package the Helm chart:

$ helm package --sign --key <key> --keyring <keyring> helm-sigstore-example

The result is the packaged Helm chart and the associated provenance file. Both files (similar to Helm’s provenance itself) are needed in order to publish the chart to Rekor.

Since all of the necessary resources are now available, we can now use the helm sigstore upload command provided by the Helm sigstore plugin to publish the chart to Rekor. The only required argument is the path to the packaged chart. The remaining options, such as the PGP keyring and public can, can be inferred either using the default values or from the content of the provenance file. If the --keyring option was used to sign the Helm chart, it will need to be provided as the default location is identical to that of the helm package command.

The final option to consider is the destination for which the chart will be published. By default, charts are published to the publicly hosted Rekor instance. However, the plugin can make use of a self managed Rekor instance as well by specifying the --rekor-server parameter or through the REKOR_SERVER environment variable. Steps for deploying your own instance were provided earlier.

Now, upload the Helm chart to Rekor

$ helm sigstore upload --keyring <keyring> helm-sigstore-example-0.1.0.tgz
Created Helm entry at index 688276, available at: https://rekor.sigstore.dev/api/v1/log/entries/f5d7359c0c32bba544e9258a40c64537be773d4bb65d1fd11e32d945927bf88d

Congratulations, you just published your first Helm chart to a Rekor instance using the Helm sigstore plugin!

With the chart successfully published to Rekor, we can now change personas to a role that represents the majority in the Helm community: the consumer.

As a consumer of Helm charts, or any software, you have the opportunity (and responsibility) for asserting the content prior to use. The Helm sigstore plugin provides several options that consumers can use to manage Helm charts.

First, you can determine whether a chart has been previously published to Rekor. The mere fact that a publisher has taken the time to not only sign a chart, but publish its content, is a good sign they have security in mind. The search subcommand of the Helm sigstore plugin queries whether a chart has been published to the targeted Rekor instance.

Let’s use the previously published helm-sigstore-example chart to demonstrate searching for charts in Rekor.

$ helm sigstore search helm-sigstore-example-0.1.0.tgz
The Following Records were Found
Rekor Server: https://rekor.sigstore.dev
Rekor UUID: f5d7359c0c32bba544e9258a40c64537be773d4bb65d1fd11e32d945927bf88d

The Rekor UUID values along with the server are displayed if the chart was located. To see what would happen when a chart could not be located, feel free to create and sign a new Helm chart and then use the helm sigstore search command once again pointing to the new chart.

While just determining whether a chart has been published to Rekor or not is a good first step, it is important that the content be verified as to ensure that it has not been tampered with since it was published. Use the helm sigstore verify subcommand to confirm the signed and packaged chart matches the content in the Rekor server. The public key that was used to sign the chart must be available either as a standalone file or imported into a PGP keyring. Verify the chart by executing the following command:

$ helm sigstore verify --keyring <keyring> helm-sigstore-example-0.1.0.tgz
Chart Verified Successfully From Helm entry:
Rekor Server: https://rekor.sigstore.dev
Rekor Index: 688276
Rekor UUID: f5d7359c0c32bba544e9258a40c64537be773d4bb65d1fd11e32d945927bf88d

Great! The chart was verified successfully. So, what does the verification process entail?

• A match between the public key being specified and the entry in Rekor
• A match between the digest included in the provenance file and the entry in Rekor
• PGP verification of the provenance file (similar to the default verification Helm performs when installing signed charts)

At this point, given that the chart completed the verification process, you should have the confidence knowing that no malicious action or corruption occurred and can make informed choices on its usage.

We’re Just Getting Started

Very much like the sigstore project itself, the integration with the Helm ecosystem is still in its infancy. There are a lot of ideas and opportunities in the works. One of the areas of future investigation and collaboration with the upstream Helm community is around the inclusion of additional methods for signing charts beyond PGP. Signed Helm chart adoption still remains relatively low, and this statistic can be partially attributed to the complexities and management of PGP keys. By lowering the barrier of entry and enabling signing of Helm charts with ease, the benefits can be realized by both chart producers and consumers.

Of course none of this would be possible without contributions from the community. Does the integration of Helm with the sigstore project sound interesting to you? To get involved, check out the sigstore website, GitHub organization or join the sigstore workspace on Slack.