Automatic Dynamic Secrets Retrieval in Microsoft Azure VMs with HashiCorp Vault

If you already know HashiCorp Vault and Microsoft Azure well and you have some Terraform experience and would like to jump to the code directly, be my guest :) Here is the GitHub Repo.

Introduction

Applications running inside Azure VMs need access to various secrets. e.g. API keys, database credentials, certificates, cloud credentials etc. Although traditionally secrets are provided manually in application configuration or environment variables, many new cloud applications store these secrets in Azure KeyVault.

There are many advantages of using Azure KeyVault, e.g.

• Secrets are decoupled from the application
• Secrets can be managed and revoked centrally and with APIs
• Secrets are saved more securely and can be accessed programmatically
• Secret sprawl can be avoided
• Access keys can be rotated

Even though Azure KeyVault solves many problems for you, there are still multiple challenges that you need to address to secure your application and automate secret access and retrieval. These are some of the additional challenges you may consider:

• How do you authenticate and provide secrets to applications that don’t use Azure AD?
• How do you integrate legacy applications without code change?
• How do you make your application cloud agnostic?
• How do you rotate database credentials?
• How do you provide credentials in other data-centers and clouds?

In this blog post we will learn how you can use HashiCorp Vault and Vault Agent to dynamically fetch secrets and configure your application inside an Azure VM without any code change using system managed machine identity, Azure auto-auth and templating powered by Vault Agent. Vault ensures tighter security, end to end automation, and avoids credential sprawl.

HashiCorp Vault

HashiCorp Vault is an API-driven, cloud-agnostic, secrets management platform. It allows you to safely store and manage sensitive data in hybrid and multi-cloud environments. You can use Vault to generate dynamic short-lived credentials, and encrypt application data on the fly.

Vault is currently trusted by many established financial institutions and large enterprise organizations across the world. You can learn more about it here.

Vault and Azure Active Directory Identity Management

Today we will focus on Vault’s ability to integrate with Azure Active Directory and managed identities and also use this authentication method to access dynamic short-lived secrets for a MySQL database.

Vault supports a long list of authentication methods apart from Azure Active Directory showcased here and dynamic credentials for a lot many systems and databases apart from the MySQL database used in this blog.

Hence, this workflow and pattern can be applied to multiple environments with different authentication methods, databases, systems and clouds.

Vault Server Installation

Vault can be downloaded here. Even though Vault can easily be started in Dev mode, for today’s exercise I recommend running it as a service in your Azure VM or VM Scale Sets. I used the following script as part of my Terraform code to deploy Vault in an Azure VM. The only thing you need to change where required, is the correct URL for downloading Vault and the Azure credentials.

You can find general Vault installation documentation here.

For production installations please refer to the Vault operations section of the Vault learn website.

Vault Agent

Vault Agent is a client daemon that provides the following features:

• Auto-Auth — Automatically authenticate to Vault and manage the token renewal process for locally-retrieved dynamic secrets.
• Caching — Allows client-side caching of responses containing newly created tokens and responses containing leased secrets generated off of these newly created tokens.
• Templating — Allows rendering of user supplied templates by Vault Agent, using the token generated by the Auto-Auth step.

In our setup we will run the Vault Agent inside an Azure VM, use Azure auto-auth method with managed identities to authenticate, and use templating to configure an application with dynamic secrets.

Setup

Assuming that HashiCorp Vault is already installed, MySQL database and Application VM are already provisioned in your Azure environment, this scenario can be setup as follows:

Vault Setup

1. Enable the database secrets engine.
2. Configure database secrets engine with MySQL credentials.
3. Create a role for the dynamic database credentials with a TTL and user creation statement.
4. Define a policy to allow access to the database credentials.
5. Enable the Azure auth method.
6. Configure the Azure auth method.
7. Create an Azure auth role and assign policy created in step 4 and bind it to your subscription and resource group.

Azure VM Setup

1. Create Vault Agent configuration.
2. Define application configuration template for rendering.
3. Install Vault Agent.

Once both Vault and Vault Agent are setup Vault Agent will automatically fetch the database credentials and render the application configuration based on the template.

Vault Setup

Let’s take a look at the Vault configuration in more detail.

Dynamic Database Credentials for MySQL

1. Enable the database secret engine.
2. Configure it with MySQL details and credentials and associate Vault role definitions.
3. Rotate root password, this ensures no human or machine has seen the root credentials and Vault is managing credential creation for MySQL now.
4. Create roles with creation statement and required TTL.
5. Use Vault read command to test if the setup and connection with MySQL is successful.

Vault ACL Policy

Create a Vault ACL policy that allows access to the MySQL database credentials.

Azure Auth Method Setup

1. Enable Vault Azure auth method.
2. Configure Azure auth method with your service principal Azure credentials.
3. Create a role, associate the policy we create in the previous step to allow access to MySQL database credentials and bind the role to your subscription and resource group.

Azure VM Setup

Let’s take a look at the Azure VM configuration in more detail.

First of all you have to make sure you have configured your Azure VM to use system-assigned managed identities.

Vault binary to run the Agent can be downloaded from here.

Define a template for your application’s configuration, e.g.

Create the Vault Agent configuration with Azure auto_auth method, specifying the Azure auth role that we created earlier, path to the template file and destination path where the file should be saved for the application after it is rendered. It is also possible to provide a sink file to write the Vault token in case the application needs to use the token to make Vault API calls.

Here is some sample code to configure and start Vault Agent as a service :

Once the Vault Agent is successfully configured and started you will see that your application configuration is rendered in the template destination location specified in the Vault Agent configuration.

That’s it. Now Vault will dynamically update your configuration whenever there is a change in the database credentials.

This one-time configuration allows you to avoid sprawl or leakage of credentials and use Azure system-assigned managed identities for authentication.

Vault Agent Templating

As you can see in this example, Vault Agent can be used to render templates with credentials or any kind of secrets including dynamic credentials for systems like databases.

This is a simple yet very powerful tool to help you automate secret retrieval and configure applications in run time without any manual intervention.

Demo

As stated in the beginning of the blog post, if you would like to test or demo this functionality I have created a Terraform project on GitHub. This will build your environment from scratch and will deploy the following:

1. Azure MySQL database
2. Azure VM with Vault Server installation and setup
3. Azure VM with Vault Agent install, setup and a python webapp

Here is the link to the GitHub repository:

https://github.com/kaparora/vault-azure-python-mysql-webapp

Summary

With HashiCorp Vault and Vault Agent you can fully automate the configuration of your applications running inside an Azure VM with dynamic secrets and database credentials using system-assigned managed identities for Azure VMs. This enables you to secure your applications and truly implement DevSecOps.