Secret management in multi-tenant environments

Jeremy Krach | Software engineer, Infrastructure Security

Knox, Pinterest’s open-source secret management service, now supports SPIFFE x509 identity documents as an authentication method. This update enables Knox to manage secrets in multi-tenant environments like Kubernetes.

Knox is widely used across Pinterest’s infrastructure to provision secrets into production systems. In an environment where one host machine corresponds to one “service” identity, Knox ACLs are completely expressive. However, microservice infrastructures today leverage tools like Docker and Kubernetes to deploy multiple containerized services onto a single multi-tenant host. In this world, Knox needs to re-evaluate its concept of a service.

Fortunately, the folks at SPIFFE have done plenty of work in defining a schema for providing a service with an identity (detailed spec here). The main idea — in this context — is that a URI Subject Alternative Name can be added to a typical x509 certificate to provide a service with a unique identity. The general format looks something like this: spiffe://trust-domain/arbitrary-path

Knox currently authenticates machines by examining the hostname of a standard x509 client certificate. With this update, Knox can authenticate any service with a SPIFFE identity by inspecting the URI SANs in the certificate. Below is a sample certificate containing a SPIFFE identity:

If a client were to authenticate to Knox using this certificate, Knox would store its identity as spiffe://pinterest.com/service_a.

We’ve also added the ability to specify these service identities in ACLs through the Knox client. The new entry type can be specified using the -S argument and providing a SPIFFE identity, as shown in the below example.

By specifying KNOX_SERVICE_AUTH instead of KNOX_MACHINE_AUTH in your environment, Knox will know to use the SPIFFE identifier rather than the hostname:

Using the certificate above, which has the spiffe://example.com/service_a identity, this request would be granted. If instead we ran the example below, we wouldn’t be able to get the key with our example certificate.

However, service_b (potentially on the same system) could use its own certificate with the proper identity to get this key.

In order to use this in production, you’ll need a mechanism to provision certificates to each individual service running on your multi-tenant system. SPIRE, the reference implementation for SPIFFE, is a good place to start.

At Pinterest, we already have an internal PKI that provisions certificates to host machines using AWS to attest identity. By extending this PKI with a Kubernetes attestation layer, individual pods are now able to request certificates that include an embedded SPIFFE identity.

Check out these changes and any new ones to come by starring Knox on GitHub.