Authenticating & Connecting to Azure Key Vault Programmatically
After completing my first Medium post about using environment variables in Python, I promised that I would follow up with an article for connecting to the Azure cloud and its’ resources programmatically (aka through code). In proper fashion, it’s important to start with what cloud resources are and why we would want to do this.
The what…
In short, cloud resources are tools or technology that are offered by someone other than your organization’s IT team. For example, they provide the means to host infrastructure, build data workflows, fancy dashboards, etc., etc. For the most part Amazon (AWS), Microsoft (Azure), or Google (GCP) dominate the market. I’m going to be pulling a secret from Azure Key Vault using Azure Identity python libraries but you could very well be connecting to SQL databases, logic apps, or other Azure resources using the same mechanism.
The why…
In order to interact with Azure resources, especially via code, you need to establish and authenticate an identity using credentials. Microsoft discusses different authentication approaches for Python applications. I chose to work with the Azure Key Vault because it’s a simple place to reference and store valuable information like tokens, certificates, keys, connection strings, and configuration files. Without this you would be managing and hardcoding objects manually or relying on the console which gets mundane.
Alrighty, time to get the show on the road.
The prerequisites:
- At least Windows 10 operating system
- A Microsoft Azure account, an active subscription, & resource group — to create a service principal, resources, and key vault objects
- VS Studio Code — intuitive code editor
- The Python extension for VS Code
- Python downloaded and added to PATH variable on your computer
- Understanding of environment variables from my previous article
Python Libraries & Functions
Step 0: Setting up your environment
First things first, follow steps 1–3 in my previous post. This will get you up to speed before we go creating a key vault, a key vault secret, and a service principal in Azure. You will also need to install the two additional libraries above, not used in the previous post, via terminal commands:
pip3 install azure-identityand
pip3 install azure-keyvault-secretsStep 1: Create an Azure key vault & key vault secrets
On the home page of the portal, select + Create a resource.
Type in Key Vault and hit Enter or choose it from the list of common resources below.
On the Create a key vault blade, choose the appropriate subscription and resource group for your key vault and give it a name. I am going to choose ‘im-eda-dev-platforms-kv’ for mine. Follow a proper naming convention if you are using CICD, that way you can easily reference these objects in your pipelines. After, make sure to choose the region closest to you or your customers, use the rest of the default options (we will make changes later, and hit Review + create.
Once the key vault has been deployed navigate to the resource. Copy the Vault URI down from the overview page and put it in a notepad. We will be referencing this later as an environment variable. After, under the Objects blade select Secrets.
On the Secrets blade select + Generate/Import. Your secrets will show up here after they have been created.
Add a secret value by giving it a name to reference, then enter the actual value you want to call, set an expiration date, and hit Create. In real-world scenarios, this is a good place to add configurations or connection parameters. Another tip from experience, is not adding environment prefixes or suffixes to secret names (i.e. ‘medium-token-dev’) as this will create extra work for you in the CICD process.
You can view your secret details by selecting the newly created secret back on the key vault secrets blade. I chose ‘This is a test!’ for the secret value and ‘medium-token’ for the name, which we will be referencing later.
For context, if you are only using the Interactive Browser Credential (browser pop-up and login) you can skip to Step 5. Otherwise, buckle up as I start with the requirements for service principals, followed by what’s needed for managed identities. Also, managed identities are a type of service principal but are less complex to manage.
Step 2 — Option 1: Create a service principal via app registration
Microsoft does a thorough job of explaining service principals in their documentation. But in layman’s terms, a service principal is a tool -with an identity- that is created and managed at the highest level in a directory, just like users. When you create a service principal, a one-way relationship of trust is established between the service principal and the Azure identity platform.
Hmm okay… but why does this matter? Partly because certain values created and gathered during this process will be used as environment variables in our code later on (specifically for identities 2 and 3 mentioned in Step 5 below). The values we will be generating/collecting are: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID, and AZURE_SUBSCRIPTION_ID. By all means, there are multiple ways to collect the latter two values, but for arguments sake I am using the method below.
Login to the portal and navigate to Azure Active Directory via the search bar.
On the Azure Active Directory blade select App registrations and then + New registration.
Give your app a name and then select Register. I chose ‘it-eda-dev-platforms-sp’ for mine because I will be using it for my platform applications in my dev environment.
Under Manage on the service principal blade select Certificates & secrets , then select + New Client Secret.
Add a good description for the client secret, set an expiry date, and hit Add.
Microsoft will generate a client secret value and secret id for your service principal. Please copy both of them down (in a Notepad app for now), you will NOT be able to access the secret value after you leave the page. We will use the secret value as an environment variable later.
After you have copied those items down, go to the Overview blade of the service principal. Copy the Application (client) ID, Object ID, and Directory (tenant) ID values onto the same Notepad.
You can find your Subscription ID on the homepage under Subscriptions. Copy the value I have blacked out and store it on that Notepad again.
Step 2 — Option 2: Create a user-assigned managed identity
Managed Identities — What are they? Long story short, they are a special (low-maintenance) type of service principal that does not require users/developers to manage credentials, such as the client secret value created in the last step.
For Azure resources that support Azure Active Directory authentication, like a key vault, managed identities automatically generate an identity in Azure Active Directory. There are two identity types, system-assigned and user-assigned. Microsoft explains the differences here, along with common applications. We won’t be using system-assigned managed identities but they are directly tied to one resource (VMs are the most common). User-assigned managed identities are their own resource and can be used by multiple resources in Azure.
Their documentation also gives pretty straight forward directions for creating a user-assigned managed identity so I don’t have to, yay for me.
I named my managed identity ‘it-eda-dev-platforms-mi’. After you create your managed identity, you will need to go to the overview blade and jot down the Client ID and Object (principal) ID on the notepad.
Step 3: Granting the service principal & managed identity access to the key vault
Navigate to your key vault and on the left blade choose Access Policies then + Create.
For the purposes of the demo choose Get and List for the secret permissions then select Next. Of course you can add additional permissions or choose use a template configuration but please, please please do so with caution.
Type the name of your service principal my service principal is named ‘it-eda-dev-platforms-sp’, select the service principal (ensure it is listed as selected down below) hit Next. On the Application blade hit Next again, and under Review + create, select Create. Unfortunately key vault access policies can only be assigned to one object, principal, or user at a time. Azure Active Directory Groups are a great way to overcome this limitation.
If you decided to create a managed identity, repeat the motions above to assign it the same access policy.
On the access policies blade of the azure key vault you will see that both the service principal and the managed identity have the associated permissions.
The last thing you need to do (or your Azure administrator needs to do) is assign the managed identity a Contributor role under access control .
Step 4: Updating the .env file values and referencing them in your script
Back in VS Code open the .env file. You will need the following environment variables:
# Setting the subscription variables
AZURE_TENANT_ID = '[YOUR TENANT ID]'
AZURE_SUBSCRIPTION_ID = '[YOUR SUBSCRIPTION ID]'
# Setting the service principal variables
AZURE_CLIENT_ID = '[YOUR SERVICE PRINCIPAL APPLICATION (CLIENT) ID]'
AZURE_CLIENT_SECRET = '[YOUR SERVICE PRINCIPAL SECRET VALUE]'
# Setting the managed identity variables
MI_CLIENT_ID = '[YOUR MANAGED IDENTITY (CLIENT) ID]'
MI_OBJECT_ID = '[YOUR MANAGED IDENTITY OBJECT ID]'
# Setting the key vault variables
KEY_VAULT_URI = 'https://it-eda-dev-platforms-kv.vault.azure.net/'
KEY_VAULT_NAME = 'it-eda-dev-platforms-kv'
After you add the details hit ctrl+K S or File> Save All.
Move to your .py file or .ipynb file, this time I chose to use a Jupyter Notebook (azureIntegrations.ipynb) instead of a regular python file because it is easier to unit test. Import the following libraries that were installed in step 0. I’m importing three credential libraries that are covered below in addition to a secret client library that allows us to interact with Azure key vault secrets.
import os
from dotenv import load_dotenv
from azure.identity import InteractiveBrowserCredential, ClientSecretCredential, DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
The following cells in the Jupyter notebook , or lines in a python file, will change depending on the azure identity credential requirements. I will walk through the interactive browser credential, client secret credential, and managed identity credential in detail.
Step 5: Choosing an Azure identity credential and connecting to key vault
Whew, now we get to tie everything together. How exciting! Hopefully you are still with me. We are going to get a key vault object after authenticating to the Azure identity platform using each method below. The list is either/or and will change based on your needs.
1. Interactive Browser — probably the easiest, but least programmatic, way to authenticate to Azure. When the client is called, your default browser (i.e. Chrome) will pop up and ask you to sign-in to your account. You definitely can’t use this in a CICD workflow, but it is good to use locally, and the scope of the identity will be limited to the access your account has on the platform. You will need to make sure you’ve completed Steps 0 and 1 to use this authentication method.
After the libraries are installed, two variables need to be created. The first is KVUri and it pulls in the key vault uri that was stored as an environment variable in Step 4, ‘https://it-eda-dev-platforms-kv.vault.azure.net/’. The second is MEDIUM_TOKEN and it’s referencing the key vault secret that was created in Step 1, ‘medium-token’.
# Grabbing environment variables from the .env file
KVUri = os.getenv('KEY_VAULT_URI')
# Creating a variable to store the key vault value in
MEDIUM_TOKEN = 'medium-token'
Create a variable named credential to call the interactive browser credential class. This variable will also be passed as a parameter in the SecretClient class along with the key vault url, which needs to be stored as a client variable. Note: I needed to add the additionally_allowed_tenants parameter because I work with multiple tenants and it was throwing an error. Take a look at this blog post about working with multiple tenants.
# This section of code authenticates to an Azure Key Vault Using Interactive Browser Credentials
credential = InteractiveBrowserCredential(additionally_allowed_tenants=['*'] )
client = SecretClient(vault_url = KVUri, credential = credential)
Finally, you are going to retrieve your secret from the key vault using the get_secret method with your MEDIUM_TOKEN variable as a parameter and add .value afterwards to get the actual secret value, otherwise you will be getting the secret uri.
When you run the cell, your browser should pop up and you will need to login to the platform using your credentials. I’ve used a print statement to show that the script does what I say it does.
# Lastly the secret is retrieved from the key vault using get_secret
get_secret = client.get_secret(MEDIUM_TOKEN).value
print(get_secret)
Ta-da! We’ve used the interactive browser credential.
2. Service Principal — is my favorite way to connect to the platform but it requires some additional set up. I am not sure why they named this library ClientSecretCredential instead of ServicePrincipalCredential because it could easily cause some confusion with the SecretClient method… But anyways, you need to ensure that Steps 0, 1, 2 Option 1, 3 and 4 are completed above before trying this code out.
Once you’ve installed the libraries above, create the 5 variables below. The first 4 are environment variables being called from your .env file. The last variable is the name of the secret you wish to grab from your key vault.
# This section of code authenticates to an Azure Key Vault Using Client Secret Credential
# It pulls the medium-token stored in the Azure Key Vault
# This keeps senstive information out of the code.
# Grabbing environment variables from the .env file
KVUri = os.getenv('KEY_VAULT_URI')
client_id = os.getenv('AZURE_CLIENT_ID')
client_secret = os.getenv('AZURE_CLIENT_SECRET')
tenant_id = os.getenv('AZURE_TENANT_ID')
# Creating a variable to store the key vault value in
MEDIUM_TOKEN = 'medium-token'
You’ll then pass the client_id variable as the client_id parameter, the client_secret variable as the client_secret parameter, and the tenant_id as the tenant_id parameter in the ClientSecretCredential method and assign everything to a variable named credentials.
# This section of code authenticates to an Azure Key Vault Using the Client Secret Credentials
credentials = ClientSecretCredential(client_id=client_id, client_secret=client_secret,tenant_id=tenant_id)
client = SecretClient(vault_url = KVUri, credential = credentials)
To finish getting your secret value from the key vault, you use the get_secret method and the MEDIUM_TOKEN variable as it’s parameter. You must add .value after the closing bracket or you’ll be retrieving the secret uri which is not useful here.
When you run the cell, the client retrieves your secret from the key vault via api and returns it’s value. The print statement displays the correct password being requested.
# Lastly the secret is retrieved from the key vault using get_secret
get_secret = client.get_secret(MEDIUM_TOKEN).value
print(get_secret)
You now know how to authenticate to the platform using a service principal. YAY.
3. Managed Identity — probably the most popular credential that is used because it doesn’t require password management or much administration. HOWEVER, you cannot use the ManagedIdentityCredential library locally. In order you use a managed identity credential locally you need to use the DefaultAzureCredential library. Microsoft explains the default credential algorithm in more context here. Make sure Steps 0, 1, 2 Option 2, 3, and 4 are completed from above to use this authentication method.
When called, the library essentially goes through a list of all possible authentication types, starting with Service Principals and ending with the interactive browser. If the Service Principal is set up it will choose that method first, if it isn’t, it will go down the list of scenarios so on and so forth until one option authenticates.
In your .env file, ensure that you store the ID of the managed identity as a variable (MI_CLIENT_ID) along with your key vault url (KEY_VAULT_URI) so you can call them from your .env file in to your python script. After, switch to your python file and add the lines below. The script below creates three variables, one for the Key Vault url, another for the Managed Identity ID, and the last one is the name of the password you wish to retrieve from the Key Vault.
# Grabbing environment variables from the .env file
KVUri = os.getenv('KEY_VAULT_URI')
MI_CLIENT_ID = os.getenv('MI_CLIENT_ID')
# Creating a variable to store the key vault value in
MEDIUM_TOKEN = 'medium-token'Note: I was a tad lazy here and stored the KEY_VAULT_URI environment variable under the name KVUri. You can really choose any name to store your environment variables but for more readable code it’s best to consistent.
After you will create a variable named credential to call the Default Azure Credential class. The DefaultAzureCredential class may require 2 parameters, a managed_identity_client_id parameter which will be your MI_CLIENT_ID variable from the .env file above and the additionally_allowed_tenants parameter which you will need if you use multiple tenants or can help sometimes if other errors are being thrown.
In the next line, the credential variable will also be passed as a parameter value in the SecretClient class along with the key vault url variable (KVUri) which is then assigned as a variable called client (same same as other methods mentioned above).
# This section of code authenticates to an Azure Key Vault Using Default Browser Credentials
credential = DefaultAzureCredential(managed_identity_client_id = MI_CLIENT_ID, additionally_allowed_tenants=['*'])
client = SecretClient(vault_url = KVUri, credential = credential)
You retrieve the key vault secret value using the get_secret method with your MEDIUM_TOKEN variable as a parameter and add .value after the closing bracket. I’ve printed the secret to show that the retrieval is successful.
# Lastly the secret is retrieved from the key vault using get_secret
get_secret = client.get_secret(MEDIUM_TOKEN).value
print(get_secret)
Bada bing bada boom. Congratulations! You know how to authenticate and interact with the Azure platform in multiple ways.
You can apply this knowledge to other Azure libraries to programmatically interact with databases, logic apps, etc. Connect with me on LinkedIn or follow me on here for more content!
