Production Secret Management at Airbnb
Our philosophy and approach to production secret management
Airbnb is a global community built on trust. The Security team helps to build trust by maintaining security standards to store, manage and access sensitive information assets. These include secrets, such as API keys and database credentials. Applications use secrets to provide everyday site features, and those secrets used to access production resources are particularly important to protect. This is why we built an internal system we call Bagpiper to securely manage all production secrets.
Bagpiper is a collection of tools and framework components Airbnb uses in all aspects of production secret management. This includes storage, rotation and access. More importantly, Bagpiper provides a safe, repeatable pattern that can be applied across Engineering. It is designed to be language and environment agnostic, and supports our evolving production infrastructure. To better understand what Bagpiper is, it is worth understanding our design considerations.
Design Goals
A few years ago, the majority of Airbnb’s application configurations were managed using Chef. These configurations included secrets, which were stored in Chef encrypted databags. Chef helped us to progress towards having infrastructure as code, but led to usability problems for our engineers because applications were deployed using a separate system. As Airbnb grew, these challenges became significant opportunities to improve our operational efficiency.
To scale with Airbnb’s growth we needed to decouple secret management from Chef, so we built Bagpiper to provide strong security and operational excellence. We aimed to achieve these goals:
- Provide a least-privileged access pattern
- Allow secrets to be encrypted at rest
- Support applications across different languages and environments
- Manage secrets for periodic rotation
- Align with Airbnb’s engineering patterns
- Scale with Airbnb’s evolving infrastructure
Segmentation and Access-Control
Bagpiper creates segmented access by asymmetrically encrypting secrets with service-specific keys. At Airbnb, services are created with basic scaffolds to support common features such as inter-service communication. At this time, an unique IAM role and a public/private key-pair is also created. The private key is encrypted by AWS’ key management service (KMS) using envelope encryption. The encrypted content is tied to the service’s IAM role, so none other than the service itself can decrypt it.
Selected services access each secret through a per-secret keychain. Bagpiper encrypts a secret with each of the public keys found on the keychain. Only those services that possess the corresponding private keys can decrypt the secret. Envelope encryption is used to encrypt the secret but it is made transparent to the user. In our deployment, a production application will first invoke KMS to decrypt the private key, and then use it to decrypt any number of secrets that it is allowed to access. Since most of the operations happen offline, it is scalable and cost effective to deploy.
Encrypted at Rest and Decrypted on Use
Bagpiper allows engineers to add, remove and rotate secrets, or to make them available to selected production systems. Bagpiper translates this work to file changes and writes them to disk as databag files. Databags are JSON formatted files with similar structures as Chef encrypted databags. Databag files, along with the encrypted key files and code are checked into applications’ Git repositories.
Changes to secrets go through the same change-release process as code. This includes peer review, continuous integration (CI) and deployment. During CI, databags and key files are incorporated into application build artifacts. On deployment, applications use the Bagpiper client library to read secrets from databags. With Bagpiper, applications are able to access secrets securely. At the same time, deploying secrets together with code changes simplifies the release process. Application states from forward deployments and rollbacks are predictable and repeatable. We are able to progress towards having infrastructure as code with improved operational safety.
Application Support and Integration
Many technologies we use today did not exist at Airbnb a few years ago, and many more are yet to be built. To build for the future of Airbnb, Bagpiper must be able to support a variety of applications across different languages and environments. We achieved this goal with the Bagpiper runtime-image, a cross-platform executable that abstracts parsing and decrypting databags. It is written in Java, and then built into platform-specific runtime-images using the jlink tool. Bagpiper runtime-image runs on Linux, OSX and Windows systems without a Java runtime environment (JRE). It is installed across different application environments and found in the file systems.
Bagpiper can support different types of applications cost-effectively. Today, majority of Airbnb’s applications are written in Java and Ruby. Therefore, we created lightweight client libraries for Java and Ruby applications to securely access secrets. Underneath, these client libraries read databags and key files from the build artifact and directly invoke the Bagpiper runtime-image from the file system. Client libraries receive the secrets and make them available to the application idiomatically.
Secret Rotation on a Continual Basis
Best practice is to rotate secrets periodically. Bagpiper helps to enforce a secret rotation policy with per-secret annotated data. Secret annotations are key-value pairs that can specify when a secret was created or last rotated, and when it should be rotated again. Annotated data is encoded and stored alongside the encrypted secret inside databag files. Though it is unencrypted, it is cryptographically tied to the secret, so tampering is not possible.
Today, a small but growing number of secrets are annotated for the purpose of managed rotation. A scheduled job periodically scans application Git repositories to find secrets that are eligible to be rotated. These secrets are tracked through an internal vulnerability management system, so that application owners are held accountable. Sometimes it is possible to generate secrets, for example MySQL database credentials. If this is described in the annotations, a pull request that contains the file changes for the rotated secret is created for the application owner to review and deploy.
Putting it All Together
We looked at Bagpiper from several different angles. It is a secret management solution that uses a non-centralized, offline architecture that is different from many of the alternative products. Bagpiper leverages Airbnb’s infrastructure as much as possible and uses a Git-based change-release pattern that Airbnb is already familiar with. This approach has helped us to meet our secret management needs across different application environments. To help put everything into perspective, see an architectural diagram below:
Concluding Thoughts
A secure and repeatable secret management practice is critical to a company that is undergoing rapid growth and transformation. It also presents many unique technical challenges and tradeoffs. We hope sharing Airbnb’s philosophy and approach is helpful to those that face similar challenges. As we remain focused on building technology to protect our community, we are grateful for our leaders within Airbnb for their continued investment, commitment and support.
Want to help protect our community? The Security team is always looking for talented people to join our team!
Many thanks to Paul Youn, Jon Tai, Bruce Sherrod, Lifeng Sang and Anthony Lugo for reviewing this blog post before it was published
