Delivering Secret Zero: Vault AppRole with Scalr and Chef
The USA TODAY NETWORK is comprised of thousands of virtual machines spread across multiple cloud providers and regions. These VMs are orchestrated by Scalr and configured using Chef. To increase the security and flexibility of our current configuration process, we wanted to give our development teams complete control over their secrets without relying on systems administrators. To help facilitate that, our team chose a product by HashiCorp called Vault. Here is how we set out to increase security and flexibility by integrating Scalr webhooks with Vault to deliver access personalized for every machine based on the requirements defined by the development team.
It all starts with a GitHub pull request to our central vault policies repository managed with terraform. For example, a fictitious web team wants to give their HAProxy instances access to the SSL certificates stored in their vault space:
This single file defines a policy with the exact scope of access the web team’s HAProxy instances will need. It also defines a Vault AppRole with that same policy attached. The last resource configures our internal api to provision an AppRole token when a server owned by webteam is initialized in Scalr.
Scalr webhooks gives us a secure way to notify our internal services when various events occur within our infrastructure. Scalr adds a cryptographic signature to each notification so that we can verify the notification is genuine and not an attacker’s forgery. Events range from BeforeInstanceLaunch to HostDown and everything in-between. For our purposes we configure Scalr to send us a webhook request on the HostInit event, so that the notification fires right when the managed instance finishes booting. Here is a condensed version with only the attributes we care about:
The AppRole API will verify that the SCALR_FARM_TEAM attribute matches the one we configured in the pull request. The SCALR_FARM_TEAM variable matches the team of the Scalr user that created the farm, thus ensuring the policy and AppRole are only available to that team. If the APPROLE_NAME and SCALR_FARM_TEAM match, then our API creates a limited use token that matches the SCALR_SERVER_ID. This ID has a limit of two uses, and one of the uses is used by the API to write a wrapped AppRole login response to cubbyhole/token. Now it is up to the server and Chef to retrieve that token.
With the help of the Vault ruby gem, we created a Chef recipe that will read the wrapped token stored in the SCALR_SERVER_ID and store it on the servers’ local file system. Once the token is stored, the recipe will attempt to renew the token as long as the server is active, and chef-client is running. That recipe looks something like this:
After our webteam uses the recipe above in their Chef run, they provide a path to the vault_secret resource so that Chef can read the secrets. The following Chef code will take care of retrieving the secrets and creating a file containing the certificates:
Chef will now pull these secrets and reconfigure the server about once every 30 minutes. This is an easy and convenient way for our teams to manage and rotate their SSL certs, and has been extended to manage certificates and credentials for all kind of applications.
By using this approach, teams no longer have to compromise their secrets or go through a painful process to update their systems since they are the only ones who have access to the secrets and their servers will reconfigure themselves if the secrets ever change.
