Cloud computing, secrets, and adventures in open source
Today we’re announcing that we’re open-sourcing Pentagon for synchronizing Vault and Kubernetes secrets. But what is Pentagon? What are these secrets? The following information is for your eyes only.
Vimeo heavily leverages cloud computing for everything from video transcoding to traditional web serving. Recently, we’ve adopted containers and Kubernetes to manage new components that we add to the architecture. While this has served our new projects well, we ran into a case where we wanted to access sensitive configuration data like certificates and encryption keys stored in other systems and make them conveniently available to certain Kubernetes pods.
Vimeo uses Vault to store sensitive configuration data, but we don’t currently run it in a highly available setup. For this reason, Vault isn’t a hard dependency of any component of our systems. Since Kubernetes automatically manages the life cycle of pods and we were planning on heavily leveraging autoscaling, we didn’t want any temporary Vault unavailability to cause Kubernetes to be unable to scale up an application that read secrets from Vault on startup.
Kubernetes already provides secrets as a mechanism for storing sensitive data (though there are some serious caveats that we’ll discuss later). From our new component’s perspective, the most convenient thing would be for the required keys to be available as secrets and simply mounted into the container. That said, we didn’t want to introduce a second source of truth for this data, creating a management headache in the future.
Discussions between Vimeo’s Core Services and SRE teams resulted in a proposal: build a small application that runs periodically to keep configured secrets in Vault synchronized with Kubernetes secrets. The components in Kubernetes could simply mount the secrets and use them without being aware of Vault, and Vault wouldn’t have any additional hard dependencies. Effectively, the Kubernetes secrets would function as a cache for Vault.
Pentagon
Pentagon was born as the application to handle these synchronization tasks. Its own configuration is essentially a list of mappings from Vault paths to names of Kubernetes secrets along with a Kubernetes namespace. Given the availability of convenient client APIs for both Vault and Kubernetes, we chose to write Pentagon in Go.
We decided to use the credentials available using Google Cloud Platform service accounts to authenticate to Vault as this would eliminate having to manage another set of credentials just for Pentagon. We also created a special service account for Pentagon that had special access to create and update secrets.
The high-level business logic is as follows:
- Get the service account credentials with the help of the metadata API.
- Use the credentials to authenticate to Vault via the Vault API.
- For each configured path, retrieve the secrets from Vault.
- For each secret retrieved from Vault, create or update a Kubernetes secret with the same data.
We also decided to introduce a reconciliation concept to clean up any keys that no longer required synchronization. Reconciliation uses labels, which are part of most Kubernetes objects, including secrets. When the secret is created, Pentagon attaches a label indicating that it’s responsible for the management of that secret. Then, once secrets are updated, Pentagon checks to see if there are any existing secrets that no longer have mappings in the configuration and removes them.
Configuration
The following is a sample configuration for Pentagon. We typically store this in a Kubernetes ConfigMap and provide it as the first and only argument to the Pentagon application.
Testing
While Pentagon’s application logic isn’t terribly complicated, we wanted to ensure that we had adequate test coverage. Initially we went down the path of an integration test based on Kind, a stripped-down version of Kubernetes that runs in its own Docker container. While this worked in principle, we found the bootstrapping to be fairly mysteriously error-prone (Docker on macOS can be flaky) and time-intensive.
Complications with Kind led us to the Kubernetes fake library, which worked quite nicely in providing an API-compatible mock that we could use to test the core business logic of Pentagon. Vault had no such fake available, but the simplicity of the Vault API made it easy enough to create a simple interface that wrapped the relevant methods and implement a mock in short order.
Secrets
Kubernetes secrets on their own are simply stored as Base64-encoded text, making them merely obfuscated and not truly protected against anything but the most casual snooping. Recently, Google Cloud has introduced application-layer secrets encryption for Kubernetes that uses keys managed in Google KMS to encrypt and decrypt secrets.
We didn’t want the cached secrets created by Pentagon to be less secure than their sources in Vault, so this feature seemed particularly applicable to our use case.
Not So Fast
Vimeo uses Terraform to manage our infrastructure, and we had used it to create and manage many Kubernetes clusters successfully. Technically, application-level encryption is a beta feature, so this would have been part of the Google Beta Terraform Provider; but, given that application-level encryption is a relatively new feature, support wasn’t yet available.
The provider was open source, though, so we looked through issues and pull requests and found that one other person had requested access to the encryption option in question, but no attempt had been made on implementing it. We then figured it would be a good excuse to fork the beta provider and attempt to add support ourselves. We didn’t have a sense of how long it would take from submission to being merged, but we figured it was the right thing to do as a good open-source citizen.
The Terraform provider was nicely organized with plenty of test helpers and precedent for how other features with similar structures were implemented. In a day or so we had a PR ready and submitted it. Working with the maintainers was very easy, and in less than one week, the changes were accepted without much modification (mostly some misunderstanding of how some of the test helpers worked on our part). Overall, it was a great open-source interaction.
Putting It All Together
Once all the pieces were in place, we were able to build a Kubernetes cluster with database encryption using Terraform. We then used CronJobs as a convenient way to deploy Pentagon so that it runs periodically to handle the synchronization of secrets.
Join Us!
Does this kind of stuff sound like fun? Join us; we’re hiring!
