Dynamic Secrets Retrieval in Microsoft Azure App Service with HashiCorp Vault

Please click for an animated flow diagram of the demo

In a recent blog post, my colleague Kapil Arora demonstrated how you can use HashiCorp Vault from an Azure VM using system managed machine identity and Vault Agent. But what if your application, or a component of your application, runs as an Azure App Service (PaaS) or an Azure Function (FaaS)?

In this blog post we will learn how you can authenticate and retrieve secrets from HashiCorp Vault from within an Azure App Service. This method can also be used from an Azure Function. I also delivered a webinar on this topic and it is available here.

Demo Application

To demonstrate this functionality we’ll be using a simple CRUD front end application written in Python that connects to a back end MySQL database. There will also be a Vault server providing database credentials and performing encryption and decryption of application data. The front end application will be run on Azure App Service, and the Vault server and MySQL database will be hosted on an Azure VM. If you’d like to skip right to the demo, the code repository is located here.

HashiCorp Vault

HashiCorp Vault is an open source tool for securely accessing secrets. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log. If you need a secrets management solution with advanced features and capabilities, require a higher uptime SLA than the cloud native tools offer, and/or if your organization is hybrid cloud or multi-cloud, these are some of the reasons you may want to consider (or may already be using) Vault.

Flexible Methods

One of the benefits of open source is the vibrant and strong community that can develop around a tool, and Vault is no exception. HashiCorp and the open source community provide a variety of flexible methods for interacting with Vault. A list of libraries is compiled here, but that list is by no means exhaustive. For example, I recently learned about the gcp-vault Go library developed by the New York Times engineering team, which reduces Vault authentication and secrets retrieval from App Engine to a single function call. Very cool! In this demo I am using the Python HVAC library, but the concepts apply equally to any language or framework.

Secret Zero Challenge: Identity

There are many tools that attempt to solve the secrets distribution challenge. What makes Vault unique is that it also aims to solve the secret zero challenge; which is to say, how do I securely deliver that initial secret to my application, and in a way that is automated and highly scalable? Vault addresses this problem by providing an array of authentication methods that give applications different ways to identify themselves to Vault, and give Vault ways to verify those application identities.

Identity in Azure

Vault provides an array of authentication methods, but before we can choose the right method for the job, let’s take a look at how identity works in Azure:

Managed identities for Azure resources is a feature of Azure Active Directory. … The feature provides Azure services with an automatically managed identity in Azure AD. The managed identities for Azure resources feature is free with Azure AD for Azure subscriptions. There’s no additional cost.

And:

Azure AD returns a JSON Web Token (JWT) access token.

Perfect!

(For a quick primer on JWTs, I found this guide helpful.)

JWT Auth Method

It just so happens that Vault comes with an authentication method called the JWT auth method. The jwt auth method can be used to authenticate to Vault by providing a JWT. Vault will cryptographically verify the provided JWT against public keys from the issuer. The issuer public keys can be stored locally in Vault, or, if configured, a JWKS or an OIDC Discovery service can be used to fetch the appropriate keys during authentication. When OIDC Discovery is used, which we’ll be using in this example, OIDC validation criteria (e.g. iss, aud, etc.) will be applied. This gives us the fine grained controls necessary to verify that the client is truly who it claims to be. (ha ha ha, get it?)

On the auth method I will configure the below parameters as follows:

bound_issuer: $ARM_TENANT_ID,
oidc_discovery_url: $ARM_TENANT_ID

Together these parameters ensure that during authentication Vault will fetch the appropriate public keys for JWT verification from my Azure tenant, as well as that only JWTs from my Azure tenant will be allowed.

Bound Audience, Bound Subject, Bound Claims

Within an auth method we can configure any number of roles. The client will specify the role as part of their login request. In my demo the web app will be using webapp-role, and on webapp-role I will configure the below parameters as follows:

bound_audiences: “https://management.azure.com/", (Note: This can be configured in Azure to the address of your Vault cluster, but for now I am using the Azure default.)
bound_subject: “<the webapp Principal ID>”,
bound_claims: { (Note: These are entirely optional. Claim definitions per this documentation.)
  idp: “https://sts.windows.net/$ARM_TENANT_ID/",
  oid: “<the webapp Principal ID>”,
  tid: “$ARM_TENANT_ID”
},

Together these parameters allow Vault to authenticate only JWTs issued to this web app in our Azure tenant.

Claims as Metadata

Another unique aspect of Vault is its inherent understanding of identity. Vault internally maintains a record of the clients who are recognized by Vault. Each client is internally termed as an entity. When the web app authenticates to Vault for the first time, Vault creates a new entity representing the web app. The entity identifier will be tied to the authenticated token. When such tokens are put to use, their entity identifiers are audit logged, marking a trail of actions performed by specific applications, services, or users. We can enrich the identifiers in Vault’s identity system and audit logs by ingesting claims from the JWT and attaching them to the Vault entity as metadata.

user_claim: “sub”,
claim_mappings: {
  “appid”: “application_id”,
  “xms_mirid”: “resource_id”
},

There is also a groups_claim, but I am not using it in this demo. Together, these parameters attach the Azure application ID as the entity alias user identifier, and the Azure resource ID as metadata in Vault’s identity system.

Token Settings

The last bit of configuration we’ll do on our webapp-role is set a few configuration elements specific to the tokens that will be generated from successful authentication requests against this role. The first is configuring a token type, which we’re setting to batch for this role. (Read up on the differences between service and batch tokens here). The next is TTL (time-to-live). Every non-root token in Vault has a lifespan associated with it. After the current TTL is up, the token will no longer function — it, and its associated leases, are revoked. Finally, we can limit the IP address ranges that will be allowed to use this role by setting the token_bound_cidrs parameter.

token_type: “batch”,
token_ttl: “768h”,
token_bound_cidrs: [“10.0.2.254/32”],

Together, these parameters configure Vault to issue batch tokens with a TTL of 768 hours for this role, and restrict usage of this role to clients with the IP address of our web app. (You can find more details on Azure’s App Service VNet Integration feature here.) You may also configure parameters like token_num_uses, which specify how many times a token is allowed to be used. Last but not least, parameters that aren’t explicitly set will inherit from auth method and system defaults.

Vault ACL Policy

Now that we have our role configured, we’ll also need to create an ACL policy within Vault that defines the scope of access allowed. If you’re unfamiliar with Vault ACLs, this guide on the HashiCorp Learn site is a great place to start.

This is the web app policy used in this demo:

Demo

Thanks for following along this far, now let’s put this all together!

Provision the Demo

To deploy the sample code,

1. Make a copy of the provided terraform.tfvars.example
2. Rename it with a .tfvars extension
3. Configure it with values for your environment.
4. Run: terraform init; terraform plan; terraform apply
5. That’s it!

JWT Auth Method Configuration

The demo code automatically applies this configuration for you, but for reference here’s how we’re enabling and configuring the JWT auth method:

Vault Authentication in Python

The web app we’re using for this demo is written in Python, so I’m using the HVAC library to authenticate to Vault. Here’s how we’re retrieving an access token (JWT) from Azure, and using it to log in to Vault:

Note: For now I am intentionally using the Kubernetes auth wrapper in the HVAC library since the JWT method has not been completed, per this GitHub issue. Nonetheless I’m directing the login request to the webapp-role of auth/jwt/login.

Run The Demo

Please see the Terraform outputs to find the address of your web app and Vault server. My terraform output looked like this:

Outputs:

vault_https_addr = 
    Please note Vault configuration will take a couple minutes to complete.
    Connect to your virtual machine via HTTPS:

    "https://52.156.128.141:8200"

vault_ssh_addr = 
    Connect to your virtual machine via SSH:

    $ ssh azureuser@52.156.128.141

webapp_url = 
    Please note the first time you try to view the web app, it may take
    a couple minutes for the container to boot.

    "https://assareh-hashidemos.azurewebsites.net"

When you load the web app you should see a page that looks like this:

Screenshot of the demo web app

Super! The web app authenticated itself to Vault using its Azure managed identity. You’re welcome to browse the app and observe how Vault’s Transit engine enables seamless encryption and decryption of application data. We’re also using the Transform engine to perform format-preserving encryption on the Social Security numbers, and one-way masking of credit card numbers. The web app also retrieved a dynamic database credential from Vault’s Database secrets engine.

Let’s take a look at the Vault audit logs. Note the metadata key- we’ve attached specific identifying information about the web app from Azure to our logs.

Vault audit log entry for jwt login request showing metadata from Azure in Vault’s identity system

Conclusion

In this post we saw how we can leverage managed identities for Azure resources to authenticate to HashiCorp Vault with the JWT auth method and retrieve dynamic secrets. By using this approach, we can overcome the secret zero challenge, and allow an Azure Function or App Service to access Vault with a cryptographically signed identity document provided by Azure. This same pattern can be used for other cloud platforms such as AWS or GCP. I hope you found this post useful!