Vault multi-tenancy strategies

Comparisons between a secret consumer and a secret producer lead strategies

A multi-tenancy strategy can help consolidate your Vault installation, improving the user experience and enhancing security posture. As a result, there is a reduced risk of exposure and lower overall maintenance costs and time.

As a HashiCorp Solutions Engineer, I have seen multiple ways of implementing multi-tenancy in HashiCorp Vault, which I’ve boiled down to two main strategies: Consumer-Focused and Producer-Focused.

I’ll do my best to explain both strategies and compare them at the end, to help you decide which one to choose for your organization.

Solutions for Vault multi-tenancy

Before diving into the explanation of the two strategies, it is important to understand some Vault concepts that can be used for multi-tenancy as well as their current limits.

• Namespaces — creates a space that isolates a set of resources, such as entities, groups, mounts, policies, and tokens. (Limit: ~7k)
• Entities (Limit: ~480k)
• Groups (Limit: ~480k)
• How Group Policies are applied
• ACL Policy Path Templating (No limit)
• Sentinel EGP / RGP (No limit, but may slow down requests if misused)

How a Vault token gets a policy

To see the capabilities of your token, you can query the Vault API. However, understanding how policies are applied to your token can be complex.

Consider the following scenario: an application or user authenticates to the authz mount point using authZ_role.

The resulting token from authZ_role will have the following policies:

• policy_L from the authentication role (token role)
• policy_K attached to the groupZ
• policy_H attached to groupA
• policy_G attached to entity_id

Now that we understand all the available tools and their impact, we can proceed to examine the two strategies.

Consumer-focused strategy

When the multi-tenancy strategy is focused on the consumer, each consumer, such as an application, usually has its own segregated space in Vault, where it expects to store all its secrets.

This strategy can simplify policy writing and maintenance, because the consumer will only have access to its namespace or specific templated path.

All secrets management functions, such as lifecycle, quality, and access management, are done by an external third-party system that sits outside the control of Vault. This external system will write one or more secrets into each consumer enclave, which usually results in some duplication that needs to be externally managed.

Given the explanation above, Vault is being used as a centralized secrets caching system, with mostly a lot of static KV stores. Sometimes, this pattern grows into leveraging Vault for configuration management.

In this strategy, the secret is known by 3 parties:

• The 3rd party system
• Vault
• The application

To onboard an application or consumer, it is necessary to update, align, and audit the information on all three systems for risk and compliance.

Here are some example scenarios where this strategy can be used:

• An external process updates an API key, which is used by all or some consumers.
• An external team manages public CA keys that need to be used by some consumers.

Producer-focused strategy

When the multi-tenancy strategy is focused on the producer, each producer, such as a team or workflow, usually has its own segregated space in Vault, where it manages the secrets engine configuration and access.

The external system configures a secrets engine and decides the access level allowed to the consumers. This can be done with identity groups, ACL policy path templating, and cross-namespace secret sharing.

Vault is then responsible for the creation, update, and quality of the secret that is used by the consumer.

In this strategy, the secret is generated dynamically, so it’s only known by the consumer because Vault does not need to store dynamic secrets.

Vault natively supports many dynamic secret engines. If your use case is not covered, consider writing a Vault plugin for secrets management instead of creating a third-party process that needs to be operational to push secrets into a KV store.

This strategy can be more complex for the consumer or application because it may need to access more than one Vault path, so it needs to be integrated into the application onboarding process.

This strategy can be used, for example, by a development team that has provisioned a database, owns the data inside it, and wants to allow another application to read and/or write data. They can achieve this by running a Terraform module that creates a Vault database role, policy, and group, and adds the entity ID of the application to the group.

Which one to choose?

As usual, it depends…

It depends on your company structure, team structure, the workflows you are able to control, and your Vault consumption model. (see Landing Zones blog post)

You can use both strategies for different use cases.

Going back to the basics:

• A secret is something that only you know and own.
• You are very selective with who you share that secret.
• In the case of IT, it may provide access to another system or data.

In other words, the team that owns the data should be the one deciding which system or teams have access to that data, and that decision-making process needs to be delegated in a controlled manner.

Consider which systems your team is managing in the onboarding process. Does it also cover any external third-party systems?

Regardless of the strategy, decentralize the Vault configuration, but still control it through a centralized and secure process, such as Terraform modules and workspaces.

A Terraform module allows you to define some inputs (e.g. the ones you require in a ticket), and then it will configure Vault the way you intend.

A Terraform workspace allows you to secure the workflow and enforce those rules of engagement.

My final words of advice are:

• Be opinionated and own the approved code modules that configure Vault.
• Aim to address 80% of the use cases. If you try to cover every little edge case you’ll end up in design hell.
• At any point in the journey, your decision can change as your organization grows or your Vault usage matures.