Demystifying Vault’s Secrets Management Solutions in Kubernetes

Nicola Kabar

Published in

HashiCorp Solutions Engineering Blog

10 min readFeb 1, 2022

This blog was originally posted on the HashiCorp Blog.

Overview
Intro to Secrets & Secret Management
Four Modern Patterns for Managing Secrets in Kubernetes
A Tale of Two Vault & Kubernetes Integrations
Common Design Considerations
Similarities & Differences between Sidecar & CSI Solutions
Going Beyond the Native Kubernetes Secrets
Summary
Additional Resources

Overview

In this post, I will explore the different methods integrating HashiCorp Vault with Kubernetes and provide you with practical guidance to help you understand and choose the best solution for your use case. This post is not intended to be a product documentation or step-by-step implementation guide. It is intended for DevOps practitioners familiar with HashiCorp Vault and Kubernetes who also have a basic understanding of secret management concepts.

What are Secrets and Why You Should Care

Secrets are anything that you want to tightly control access to such as: key/value pairs containing sensitive data, tokens, passwords, TLS keys and certs, API/Cloud keys, and database credentials. Unlike application configurations, secrets can be sensitive, dynamic, time-scoped and require access control and a detailed audit log. They can also have a different life-cycle than the applications that consume them. For these reasons, secret management solutions, like Vault, have emerged to address the complexity of storing different types of secrets securely and provide a centralized, API-driven workflow to manage them across different environments.

Four Modern Patterns for Managing Secrets in Kubernetes

Traditionally, applications are composed of the business logic source code, some dependencies, backing services, configurations and secrets. Legacy applications packaged all of these components as part of the release and deployment processes effectively creating a monolith. Although this approach ensured all dependencies of the applications are included in the deployment unit, it soon introduced security, operational, and scale challenges. In contrast, modern application patterns (including microservices and 12-factor apps) shifted the structure of the applications to smaller autonomous microservices with externalized & decoupled backing services, configurations, and secrets. Although these newer patterns enhanced the overall application delivery process, they have also introduced architectural complexity and operational challenges especially with the rise of cloud. Kubernetes (k8s) took aim at solving these challenges and emerged as the de-facto platform to run applications at scale.

Organizations are now rapidly adopting Kubernetes and leveraging this momentum to adopt modern and secure patterns for managing secrets in Kubernetes.

The first pattern we are seeing these organizations adopt is decoupling secrets from the application lifecycle by leveraging automation and CI tooling to create separate workflows for creating and managing secrets.

Second, they are ensuring secrets never make their way into version control systems typically by using scanning tools.

Third, they are leveraging native k8s constructs to define and manage application secrets similar to how they define and manage application deployments. This is typically done by leveraging the native k8s secrets objects. Kubernetes has native support for secrets and utilizes etcd to store them and manage their access policies similar to other k8s objects. Compared with traditional patterns of embedding secrets alongside other application deployment and configuration assets, leveraging k8s secrets is a significant security advancement because k8s now encrypts these secrets at rest in etcd (if enabled) and automatically presents them to the applications that have access to them.

The fourth pattern is integrating a secrets management solution, namely Vault, with Kubernetes combining the power of using a full-fledged, feature-rich, and centralized secrets management solution with the ease of application deployment orchestration in Kubernetes. This approach goes beyond what the native K8s secrets offer and addresses a larger set of advanced applications and use-cases. Let’s dive into this specific pattern to understand its advantages and compare available methods to adopt it.

A Tale of Two Vault & Kubernetes Integrations

There are a few open-source projects out there that integrate Vault with Kubernetes. I will focus on two open-source solutions which are developed, maintained, and supported by HashiCorp:

Vault Sidecar Agent Injector leverages the sidecar pattern to alter pod specifications to include a Vault agent container that renders Vault secrets to a shared memory volume. By rendering secrets to a shared volume, containers within the pod can consume Vault secrets without being Vault aware. The injector is a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request. This functionality is provided by the vault-k8s project and can be automatically installed and configured using the Vault Helm chart.

2. Vault CSI Driver allows pods to render Vault secrets by employing ephemeral volumes using the CSI Secrets Store.

At a high level, the CSI Secrets Store driver allows users to create SecretProviderClass objects. This object defines which secret provider to use and what secrets to retrieve. When pods requesting CSI volumes are created, the CSI Secrets Store driver will send the request to the Vault CSI Provider if the provider is vault. The Vault CSI Provider will then use Secret Provider Class specified and the pod’s service account to retrieve the secrets from Vault, and mount them into the pod’s CSI volume. The secret is retrieved from Vault and populated to the CSI secrets store volume during the ContainerCreation phase. This means that pods will be blocked from starting until the secrets have been read from Vault and written to the volume.

There are some similarities and differences between the two solutions that you would need to consider when designing and implementing your secret management strategy. Before I walk you through them, let’s go through common design considerations to keep in mind.

Common Design Considerations

Secret Projections Every application requires secrets to be presented to it in a specific way. Typically, applications expect secrets to be either exported as environment variables or written to a file that the application can read on startup. Keep that in mind as you’re deciding on the right method to use to ensure that it meets your application requirements.
Secret Scope Some applications are deployed across multiple k8s environments (e.g dev, qa, prod) across your datacenter, edge or public clouds. Some services run outside of k8s on VMs, serverless, or other cloud-managed services. You may face scenarios where these applications need to share some sets of secrets across these heterogeneous environments without the need to replicate the secrets. Scoping the secrets correctly to be either local to the k8s environment or global across different environment will ensure that each application can easily and securely access its own set of secrets within the environment it is deployed in.
Secret Types Secrets can be text files, binary files, tokens, certs. They can be statically or dynamically generated. They can be valid permanently or time-scoped. They also vary is size. You need to consider which secret types your application requires and how they’re projected into the application.
Secret Definition You would also need to consider how each secret is defined, created, updated, removed and the tooling associated with that process.
Encryption Encrypting secrets both at rest and in-transit is a critical requirement for many enterprise organizations.
Governance Applications and secrets can have a many-to-many relationship which would require careful considerations when it comes to granting access for applications to retrieve their respective secrets. As the number of applications and secrets scales so does the challenge of managing their access policies.
Secrets Updates & Rotation Secrets can be leased, time-scoped, or automatically rotated and in each of these scenarios there needs to be a programmatic process to ensure the new secret is properly propagated to the application pods.
Secret Caching In certain K8s environments (e.g edge or retail) there is a potential need for secret caching in the case of communication or network failures between the environment and the secret storage.
Auditability Keeping a secret access audit log detailing all secret access information is critical to ensure traceability of secret access events.

Keeping these design considerations in mind, let’s go over some of the similarities and differences between the two integration solutions.

Similarities & Differences between the Sidecar and CSI Solutions

Let’s start with some of the similarities between the two solutions. Both CSI and Sidecar solutions:

Simplify retrieving different types of secrets stored in Vault and expose them to the target pod running on K8s without it being aware of the not-so-trivial Vault processes. It’s important to note that there is no need to make any changes in the application logic or code in order for it to use these solutions, making it easier to migrate brownfield applications into Kubernetes. Developers working on greenfield applications are leveraging the Vault SDKs to directly integrate with Vault.
Support all types of Vault Secret Engines. This means that you can leverage the extensive set of secret types ranging from static KV secrets to dynamically generated database credentials and TLS certs with customized TTL.
Leverage the application’s k8s pod service account token as Secret Zero to authenticate with Vault via the Kubernetes auth method. This means that there is no need to manage yet another separate identity to identify the application pods when authenticating to Vault.

4. Require the desired secrets to exist within Vault before deploying the application.

5. Require the pods service account to be bound to a Vault role with a policy enabling access to desired secrets (e.g k8s RBAC isn’t used to authorize access to secrets)

6. Deployable via Helm.

7. Require successfully retrieving secrets from Vault before the pods are started.

8. Rely on user-defined pod annotations to retrieve the required secrets from Vault.

Let’s go over how the two solution are different:

The Sidecar Agent Injector solution is composed of two elements. Sidecar Service Injector is deployed as a cluster service and is responsible for intercepting K8s apiserver pod events and mutating pod specs to add required sidecar containers. Vault Sidecar Container is deployed alongside each application pod and is responsible for authenticating into Vault, retrieving secrets from Vault, and rendering secrets for the application to consume. In contrast, the Vault CSI Driver is deployed as a daemonset on every node in the k8s cluster and uses Secret Provider Class specified and the pod’s service account to retrieve the secrets from Vault, and mount them into the pod’s CSI volume.
The Sidecar Agent Injector supports all Vault auto-auth methods. The Sidecar CSI driver only supports Vault’s Kubernetes auth method.
The sidecar container launched with every application pod uses Vault Agent which provides a powerful set of capabilities such as auto-auth, templating and caching. The CSI driver does not use the Vault Agent and therefore lacks these functionalities.
The Vault CSI Driver supports rendering Vault secrets into both K8s secrets and environment variables. Sidecar Injector Service does not support rendering secrets into k8s secrets however there are ways using agent templating to render secrets into environment variables.
The CSI Driver uses hostPath to mount ephemeral volumes into the pods which some container platforms (e.g OpenShift) disable by default. On the other hand, Sidecar Agent Service uses in-memory tmpfs volumes instead.
Sidecar Injector Service automatically renews, rotates, and fetches secrets/tokens. The CSI Driver does not support that.

The table below provides a high-level comparison between the two solutions:

* achieved through Agent Templating

Going Beyond the Native Kubernetes Secrets

Although from the surface k8s native secrets might seem quite similar to the two approaches presented above, there are major differences between them and I will highlight a few of them.

First, Kubernetes is NOT a secret management solution. It does have native support for secrets, but that is quite different from an enterprise secret management solution. Kubernetes secrets are scoped to the cluster only and many applications will have some services running outside k8s or in different k8s clusters. Therefore, considering the secret scope as part of the design process is critical. Having these applications use k8s secrets from outside a k8s environment will be cumbersome and introduce authentication & authorization challenges.

Second, k8s secrets are static in nature. You can define secrets by using kubectl or the k8s API but once they are defined they are stored in etcd and only presented to pods during pod creation. This can create scenarios where secrets get stale, outdated, or expired, requiring additional workflows to update and rotate the secrets and re-deploying the application to use the new version of the secrets. This can cause added complexity and wasted time. So make sure you consider any requirement for secret freshness, updates, and rotation as part of your design process.

Third, the security model of secret access management is tied with the k8s RBAC model. This can be challenging to adopt for users that are either not familiar or do not have the experience working with kubernetes. Adopting a security governance model that is platform-agnostic can enable you to adopt workflows for applications regardless of how and where they are running.

Summary

Designing for secret management in kubernetes is no easy task. There are multiple approaches each with its own set of pros & cons. I highly recommend exploring the different options presented in this blog and understand their internals to decide on the best option for you.