Onboarding the Azure Secrets Engine for Vault

Note: Please use the HashiCorp learn guide for the latest steps: https://developer.hashicorp.com/vault/tutorials/secrets-management/azure-secrets

The Azure AD APIs have changed and some of the API calls from scripts in this blog post are no longer supported.

Securely distributing cloud credentials at scale is a core challenge that HashiCorp Vault addresses. In this post we will look at how to onboard the Azure Secrets Engine that distributes least privileged and time bound Azure credentials. I’ll be delivering a webinar on this topic on July 21, 2020. You can register here.

Some Vault Authentication Methods and Secret Engines

We will provide workflows to enable the following tasks:

• Dynamically creating new Azure service principals in one or more subscriptions
• Generating new passwords for an existing service principal
• Revoking a dynamically created credential
• Rotating the password for the trusted user in Vault

Finally, we will demonstrate a use case in which the Azure Secrets Engine provides Azure credentials to HashiCorp Terraform that does Infrastructure-as-Code (IaC) provisioning. We will also provide a full CI/CD pipeline example with GitLab CI/CD.

This post uses CLI-based workflows with the hope that it will be useful for automating or scripting onboarding tasks. For a GUI-based approach using the Azure Portal, or if you just want to verify your Azure setup on the Portal, please view the Azure Secrets Engine Learn guide.

Prerequisites

• A HashiCorp Vault Server: You can deploy one in minutes using our Getting Started guide. Your Vault token should have an admin policy which can create and configure secrets engines (see example-vault-admin-policy.hcl). The code snippets assume VAULT_ADDR and VAULT_TOKEN environment variables are set properly. You can validate this using the Vault CLI as below.

# Vault endpoint and token
$ export VAULT_ADDR=https://vault.example.org:8200
$ export VAULT_TOKEN=<admin-or-root-token>
# Vault CLI commands (outputs omitted):
$ vault status
$ vault token lookup
$ vault secrets list

• Azure Account: You need Azure credentials with the Owner Role in at least one subscription. If you are behind a corporate Firewall, you will also need outbound network access to the following endpoints: https://management.azure.com/ and https://login.microsoftonline.com/.
• CLI Tools: Please ensure that you have access to a command line shell with the Vault binary in your PATH, along withgit, make and the Auzre CLI: az. Links to obtain these tools are listed here: pre-reqs.md.
• (Optional) A CI/CD tool and theterraform binary: These tools are needed if you are interested in seeing how the Azure Secrets Engine can help with IaC use-case. In this example we will first use terraform locally, then show GitLab CI/CD.

Getting Started

To get started we will login to Azure using the az login command and clone the demo repository. If you do not have the az tool installed, please view Install the Azure CLI documentation from Microsoft.

Please enter the following commands in a terminal to login to Azure. Then set the ARM_SUBSCRIPTION_ID environment variable, and set the active Subscription ID (note that your Subscription ID will be different).

$ az login
$ az account list
{
    "cloudName": "AzureCloud",
    "homeTenantId": "vault:fpe:yu6m6hww-cvzi-vdfm-x73p-u2zhqjfkq3ck",
    "id": "vault:fpe:8a4kij3a-jvcx-vin6-pbm0-ho9gexwdv6qg",
    "isDefault": true,
    "managedByTenants": [
      {
        "tenantId": "vault:fpe:m7qemial-wm8r-jcy2-6xqc-vbvile14cpck"
      },
      {
        "tenantId": "vault:fpe:sfy66yab-0zcm-5ro5-d779-ujmq3y27hdnj"
      }
    ],
    "name": "Engineers",
    "state": "Enabled",
    "tenantId": "vault:fpe:yu6m6hww-cvzi-vdfm-x73p-u2zhqjfkq3ck",
    "user": {
      "name": "user@company.com",
      "type": "user"
    }
  }
$ export ARM_SUBSCRIPTION_ID=8a4kij3a-jvcx-vin6-pbm0-ho9gexwdv6qg
$ az account set -s $ARM_SUBSCRIPTION_ID

• Note: all sensitive values in this post such as Azure subscription ID, client ID, and client secret have been encoded using Vault’s Transform Secrets Engine and are unusable without decoding. These values have a vault:fpe prefix which can be ignored. Please view the transform.py repo if you are interested in how this was done.

Next, please clone the vault-azure-demo repo using the following command and navigate into the vault-azure-demo directory.

git clone https://gitlab.com/kawsark/vault-azure-demo.git
cd vault-azure-demo/

1. Create and Configure a Privileged Service Account for Vault

Azure credentials must be provided to Vault for authentication. In this section we will set up a new privileged service principal for Vault and provide the client ID and secret. If Vault is running in Azure, the Managed Identity of the Azure VM can be used instead of the client ID and secret. Please refer to the Authentication section of the Azure Secrets Engine documentation.

Please run the make 1_create_vault_demo command to create the Vault-Demo service principal. We have redirected the output to the “vault-demo.json” file.

$ make 1_create_vault_demo > vault-demo.json
$ cat vault-demo.json
{
  "appId": "vault:fpe:zcnzbxhh-iym4-ubrm-q4ud-abhskg94qrlw",
  "displayName": "Vault-Demo",
  "name": "http://Vault-Demo",
  "password": "vault:fpe:bf49j9ao-wndo-4xgm-7nsn-u61dlqrbskjm",
  "tenant": "vault:fpe:wpwoy5ok-45a7-qx20-ryld-3uwkqc2ipaqx"
}

The 1_create_vault_demo target is specified within the Makefile in the root directory of the repository. It calls the az ad sp create-for-rbac command. In general, each target in the Makefile calls a set of commands. Please feel free to edit any of the commands in the Makefile or invoke the commands directly.

Next, please run source scripts/set_vars.sh to export three additional environment variables: ARM_CLIENT_ID, ARM_TENANT_ID, and ARM_CLIENT_SECRET. We will reference these variables in the rest of this post. The “set_vars.sh” script also exports some other variables used to configure the Azure secrets engine.

It’s important that we use source to run the set_vars.sh script and not run the script directly since doing that will only export the variables within the script itself. Please also run make 1_validate to ensure that all four ARM_* variables have been set to non-empty values.

$ source scripts/set_vars.sh
$ make 1_validate
OK: All variables are set

2. Assign Permissions to the Service Principal

The following Azure roles and Azure Active Directory (AAD) permissions are required for the privileged Vault-Demo service principal.

> “Owner” role for the subscription scope: The Owner role is needed for any Azure subscriptions in which Vault will create new credentials. This permission was assigned already as part of the --role Owner parameter in the az ad sp create-for-rbac command run by the 1_create_vault_demo target.

> Azure Active Directory permissions:

• Delegated permission: User.Read.
• Application permission: Application.ReadWrite.All
• Application permission: Directory.ReadWrite.All.

Please run the following command to set up the above permissions. It will run the script assign_permissions.sh in the scripts directory.

$ make 2_assign_permissions

To set up these permissions using the Azure Portal, please follow step 1 on the learn guide: step-1-create-an-azure-service-principal.

To verify permissions in the Azure Portal, please click on the “Vault-Demo” account in the Azure Portal: https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade.

3. Create and Configure the Azure Secrets Engine

We are now ready to enable the Azure Secrets Engine and configure it. We can mount the Azure Secret Engine at any path; for this guide we will use theazure-demo path, which was set in the set-vars.sh script. Please modify AZ_SECRET_PATH environment variable to mount the Azure Secrets Engine on a different path.

# Optionally set the path where Azure secrets engine will be mounted
$ export AZ_SECRET_PATH="azure-demo"
$ make 3_azure_secrets_engine

Using the Azure Secrets Engine

In this section we will start using Azure Secrets Engine.

4. Create a New Azure Service Principal

In this scenario we want Vault to create a new service principal to manage an Azure resource group. Please run the make 4_azure_role command to create a role for the Azure secrets engine named rg-alice-demoapp-dev-role. The naming convention for the rolesis rg-<owner>-<app>-<env>-role and was previously set by the set_vars.sh script as an environment variable named RESOURCE_GROUP. Please feel free to modify this environment variable if needed.

# Optionally modify the RESOURCE_GROUP variable
$ export RESOURCE_GROUP="rg-alice-demoapp-dev-role"
$ make 4_azure_role
scripts/create_role.sh
Success! Data written to: azure-demo/roles/rg-alice-demoapp-dev-role

Note that the 4_azure_role target invokes the create_role.sh script which contains the role creation command as follows:

vault write ${AZ_SECRET_PATH}/roles/${RESOURCE_GROUP}-role ttl=5m azure_roles=-<<EOF
[ {
    "role_name": "Contributor",
    "scope":  "/subscriptions/${ARM_SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}"
        }]
EOF

Now we should be able to create a dynamic credential using the vault read command. Note that the resource group needs to exist, so we will create it first using the az group create command.

# Create the resource group
$ az group create -l westus -n ${RESOURCE_GROUP}
$ vault read "$AZ_SECRET_PATH/creds/$RESOURCE_GROUP-role"
Key                Value
---                -----
lease_id           azure-demo/creds/rg-alice-demoapp-dev-role/Se3nOuKabz053BylMEqFV0AP
lease_duration     5m
lease_renewable    true
client_id          vault:fpe:92cfe7wz-c2us-gwcn-mw3f-gdhbxjlmbipp
client_secret      vault:fpe:0olvt9jj-a15e-hv6s-41wq-ja8vyg8ejfui

4.1 Revoking Credentials

In the previous snippet, note the lease_duration of 5 minutes which was specified in the role. After this duration, Vault will revoke this lease and delete the credential. To revoke the lease before 5 minutes, we can use the vault lease revoke command as shown below.

# Export the lease_id from above
$ lease_id="azure-demo/creds/rg-alice-demoapp-dev-role/Se3nOuKabz053BylMEqFV0AP"
$ vault lease revoke ${lease_id}
All revocation operations queued successfully!

To lookup previous leases, you can use the Access >> Leases tab in the Vault UI, or send a request to the List leases API endpoint. Below is an example using curl.

# Vault Lease lookup example
$ curl -s --header "X-Vault-Token: $VAULT_ADDR" \
  --request LIST \
${VAULT_ADDR}/v1/sys/leases/lookup/$AZ_SECRET_PATH/creds/$RESOURCE_GROUP-role

Alternatively, we can destroy all outstanding credentials based on the auth method or role path.

# Destroy all credentials issued to the Auth method
$ vault lease revoke -prefix $AZURE_SECRET_PATH
# Destroy all credentials issued to the Role
$ vault lease revoke -prefix \
    $AZURE_SECRET_PATH/creds/$RESOURCE_GROUP-role

4.2 Create Credential in Another Subscription

In the scenario where we want to create credentials with permissions in another Azure subscription (or in multiple subscriptions) we will need to perform two additional steps:

• Assign Owner role to the Vault demo user in the new subscription(s)
• Create Vault roles to reference the new subscription ID(s)

If you are interested in this scenario, Please use the commands below to perform these steps. We will create a newmulti-subs-role that has Contributor rights to the original subscription ID and an additional one whose ID is set in the $ARM_SUBSCRIPTION_ID2 environment variable.

# Export the new Subscription ID and Role name as variables
$ export ARM_SUBSCRIPTION_ID2=<new-subscription-id>
$ export ROLE="multi-subs-role"
# Assign owner permission
$ az role assignment create --assignee $ARM_CLIENT_ID \
    --role "Owner" \
    --subscription ${ARM_SUBSCRIPTION_ID2}
# Create Role
$ vault write $AZURE_SECRET_PATH/roles/$ROLE ttl=5m azure_roles=-<<EOF 
[{ "role_name": "Contributor", 
"scope":  "/subscriptions/${ARM_SUBSCRIPTION_ID}/resourceGroups/$RESOURCE_GROUP"
},{
"role_name": "Contributor",
"scope":
"/subscriptions/$ARM_SUBSCRIPTION_ID2/resourceGroups/$RESOURCE_GROUP"
}]
EOF

Now we should be able to create the dynamic credential for this new role as before using the command: vault read $AZURE_SECRET_PATH/creds/multi-subs-role.

5. Create Password for an Existing Service Principal

In this section we will address a “password rotation” use-case by having Vault create new passwords for existing service principals. Vault doesn’t actually rotate the password, rather it creates new passwords for the service principal, and deletes them after the lease TTL has elapsed. Azure allows us to create multiple passwords per service principal, although there is an upper limit. To rotate the password, we just need to create a new password before the previous TTL expires.

As an example, assume that we have a cron job whose password needs to be rotated every day. Please use the commands below to create a new service principal using theaz create-for-rbac command. Then we will look up its object ID and create a role in Vault with a TTL of 25 hours. Then we can generate a new password using the Vault role at the same time every day. There will be one hour buffer time where both the previous and the new passwords will be valid.

# Create the demo credential
$ make 5_cron_job_cred
(output partially omitted ...)
Looking up Object ID:
az ad app list --filter "displayname eq 'cron-job-demo'" | jq -r '.[].objectId'
vault:fpe:67bf217w-0cvu-xpol-h737-nv6yxxzbt3tn
# Export above Object ID:
$ export OBJECT_ID="vault:fpe:67bf217w-0cvu-xpol-h737-nv6yxxzbt3tn"
# Create the Vault role
$ make 5_cron_job_role

Now we can create passwords for this new account as before using the command: vault read $AZURE_SECRET_PATH/creds/cron-job-role as shown below. If you re-run the command, you will notice that the client_id does not
change, but the client_secret does.

$ vault read $AZ_SECRET_PATH/creds/cron-job-role
Key                Value
---                -----
lease_id           azure-demo/creds/cron-job-role/gXNonNyy8wREpayqqP5wNNNM
lease_duration     25h
lease_renewable    true
client_id          vault:fpe:99vgbzj8-r60f-07cq-ew8f-ycuxudmylyah
client_secret      vault:fpe:v9xxpdnx-3xv6-b00q-yp6e-aahtkkzkr1u4
# Identical client_id but different client_secret
$ vault read $AZ_SECRET_PATH/creds/cron-job-role
Key                Value
---                -----
lease_id           azure-demo/creds/cron-job-role/lhk47cUvQc29qzL8XHJD9Qfs
lease_duration     25h
lease_renewable    true
client_id          vault:fpe:99vgbzj8-r60f-07cq-ew8f-ycuxudmylyah
client_secret      vault:fpe:gtbflnq3-s9gc-48dl-17cr-58qwh7tdlmtg

To verify that multiple passwords exist in Azure, you can run the az ad sp credential list --id $client_id command.

6. Rotating the Vault Demo Credential

We may want to rotate the password for the Vault-demo principal that we created in Section 1. This is a special use case of the scenario in section 5. We will follow the same steps, except at the end we will update the client_secret parameter for Vault’s Azure Secrets Engine configuration.

Please run the make 6_rotate_demo command to perform these steps. The 6_rotate_demo executes the rotate_demo.sh script which will perform three actions:

• Obtain the Object ID of the Vault-Demo principal.
• Create a rotate-demo role in Vault and generate a new password.
• Call vault write $AZ_SECRET_PATH/config client_secret=<new-password>

$ make 6_rotate_demo

If you ran into an error, you can reinstate the original Vault-demo password for the secret engine configuration using the following command: vault write ${AZ_SECRET_PATH}/config client_secret=${ARM_CLIENT_SECRET}.

Since the rotate_demo role in Vault has a lease_ttl of 73 hours, we should perform this process every 72 hours. Similar to section 5, there will be a one hour overlap with the old and new passwords.

A few more items to note:

• Similar to section 5, you can use az ad sp credential list --id $ARM_CLIENT_ID to list passwords associated with the Vault-demo principal.
• You might notice the original “RBAC” credential we created before. It is now safe to delete this credential using the CLI command below:

$ az ad sp credential delete \
   --key-id  <rbac-password-id> --id $ARM_CLIENT_ID

7. CI/CD Example: Using Dynamic Azure credentials with Terraform

In this section we will review an example use-case for the Azure Secrets Engine as part of an Infrastructure-as-Code (IaC) pipeline. We will use Terraform to provision a few simple resources in Azure.

There are a few ways for Terraform to authenticate as listed in the Authenticating to Azure documentation. We will use the “Service Principal with a Client Secret” method and set four environment variables that will be read by Terraform: ARM_CLIENT_ID, ARM_CLIENT_SECRET, ARM_SUBSCRIPTION_ID, and ARM_TENANT_ID (reference documentation).

Please see the example commands below to generate a dynamic Azure credential as before. First we will create a Vault policy called “terraform” that allows us to generate credentials for Terraform from Vault and then create a Vault token that is assigned this policy. We can supply this token to our CI pipeline, or configure an authentication method to reference this policy.

$ make 7_create_tf_policy
(output partially omitted)
path "azure-demo/creds/rg-alice-demoapp-dev-role" {
  capabilities = ["read"]
}
# Test creating Azure credential with token
$ VAULT_TOKEN=s.f62bzNoF30XF8NSusH2ivD4T vault read azure-demo/creds/rg-alice-demoapp-dev-role

Please run the commands below to save the credentials in a file and export the above ARM_* environment variables.

$ VAULT_TOKEN=s.f62bzNoF30XF8NSusH2ivD4T vault read -format=json "$AZURE_SECRET_PATH/creds/$RESOURCE_GROUP-role" > vault_app_creds.json
$ source scripts/set_tf_creds.sh

Finally, cdinto the terraform directory and run terraform init, plan and apply commands as shown below. The main.tf file creates two resources in the resource group: a storage account and a storage container.

$ cd terraform
$ terraform init
$ terraform plan
$ terraform apply

For an automated IaC pipeline implementation of the above process, please see the .gitlab-ci.yml file in the repository root.

8. Cleanup

Please run the make clean command to delete any *.json files, the resource group we created, the cron job and the Vault-demo principal. Note: if you get errors, please view the clean: target in Makefile and run the commands individually.

Conclusion

In this post we learned how to generate dynamic credentials for existing Azure service principals using Vault’s Azure Secrets Engine. We provided an example use case with a Terraform IaC pipeline. We hope you found this post useful! Please leave any feedback for improvements. If you would like to learn more, please register for my July 21, 2020 webinar here.