How to handle Secrets in Python
Introduction
In today’s era of cloud computing, security is paramount. As developers and administrators work with a plethora of tools and platforms to deploy applications, the need to manage sensitive information — API keys, database passwords, certificates, and more — has grown.
This sensitive information, aptly named “secrets,” often dictates the level of access an application has to vital resources, making their protection crucial. And while cloud providers like AWS, Azure, and GCP have furnished robust secret management tools, there still exists a void on how to use them efficiently, especially when integrating with Python applications.
In this blog post, we’ll delve into the intricacies of secret management in Python, both locally and across three major cloud providers. By the end, you’ll have a clearer understanding of how to securely handle and access secrets, ensuring your applications remain both functional and safe.
Pre-requisites
Before diving in to the details, you should have
- Python and pip installed on your environment
- A Python IDE comes in handy like PyCharm or VSCode
- For the Hyperscalers secret managing services you need an account on either of the provider and a key-vault configured. Details on that in a bit.
Read secrets from a local file
The easiest way to store and retrieve a secret or a variable in Python is via a local hidden file like “.env” or “.secret”. When using these files always ensure to exclude these files from being added to your source control. (.gitignore shall be your friend..).
Assume we have a file locally, called “.env” with the following content:
API_KEY=dummy-key
PROJECT_ID=dummy_project
REGION=europeWithin our python code we will use the dotenv library, so we must provide the library to our python environment via
pip install python-dotenvIn our Python code we then will use the load_dotenv() function to access the secret:
from dotenv import load_dotenv
import os
load_dotenv()
api_key= os.getenv("API_KEY")
print("API_KEY: ", api_key)Eventually you may want to manage multiple secrets, so using a dictionary might be a more comfortable approach:
from dotenv import dotenv_values
my_secrets = dotenv_values(".env")
#print all secrets from .env file
print(my_secrets)
#print a dedicated secret
print(my_secrets["API_KEY"])If you have multiple files with secrets, you can use the dotenv_values function with different file names maybe from different environment us it like this:
dev_secrets = dotenv_values(".dev-env")
tst_secrets = dotenv_values(".tst-env")Read secrets from environment variables
If you’re working in a local environment, you could use OS environment variables, but keep in mind that everybody who has access to the local console easily can read the environment variables.
Assume you’ve set an environment variable like this on a unix based OS (Linux, macOS):
export API_KEY=dummy_keyOn a Windows machine it would look like this:
setx [API_KEY] "[dummy_key]To access the variable from a Python script, we will use the os library.
import os
#read environment variable
api_key = os.environ.get("API_KEY")
print("API_KEY: ", api_key)There is one caveat: this does not work from within a Jupyter Notebook, as the Jupyter kernel cannot read directly from environment, at least not when executed in a IDE like VSCode or PyCharm. For Jupyter notebooks it is easier to use the dotenv approach, otherwise you would have to configure your notebook kernel to be able to read the environment, which is out of scope for this post.
But for normal python scripts (.py) the os.envriron.get() function works pretty fine. Alternative you could use the os.environ class instead, but that might throw an error, when the variable is not known to the environment. the os.environ,get() function just would return “none” if the environment variable is not set:
# no error
print(os.environ["API_KEY"])
# will throw an error
print(os.environ["APU_KEY"])
# return "none" and no error
print(os.environ.get("APU_KEY"))So far, we could read secrets from file without the necessity to store them directly in out python code, which provides more security to our source code.
this is fine for single development and local execution, but it become tricky if the code is being shared in your team or shall be executed in any kind of pipeline or productional environment.
You would have to deal with syncing the “.hidden” files over the environments, with could lead to a security leakage.
On Cloud environment a secure way to handle secrets is to use a cloud based secret manager. All major cloud provider offering such services and we’ll discuss how to use them in out python scripts in a bit.
We will discuss examples for AWS, Azure and GCP separately.
Read a secret from AWS Secrets Manager
I’ll assume that you already have configured your AWS secrets Manager, following the AWS documentation and have a secret named “API_KEY” already stored in the secret manager.
Furthermore your aws environment has to be configured to enable python to authenticate (aws configure..). Have a look into the AWS Python SDK documentation for details.
To retrieve a secret from AWS Secrets Manager, you’ll first need to set up the AWS SDK for Python, boto3.
pip install boto3Python script to retrieve the secret from AWS key vault:
import boto3
from botocore.exceptions import NoCredentialsError, PartialCredentialsError
def get_secret(secret_name, region_name=region_name):
"""
Retrieve secret value from AWS Secrets Manager.
Args:
secret_name (str): The name of the secret to retrieve.
region_name (str): AWS region where the secret is stored.
Returns:
str: Secret value.
"""
# Create a Secrets Manager client
session = boto3.session.Session()
client = boto3.client('secretsmanager', region_name=region_name)
try:
response = client.get_secret_value(SecretId=secret_name)
except NoCredentialsError:
print("Credentials not available")
return None
except PartialCredentialsError:
print("Incomplete credentials provided")
return None
except Exception as e:
print(f"Error retrieving secret: {e}")
return None
# Depending on whether the secret is a string or binary, one of these fields will be populated
if 'SecretString' in response:
return response['SecretString']
else:
return response['SecretBinary']
if __name__ == "__main__":
secret_value = get_secret("API_KEY", "<region>")
if secret_value:
print("Secret value:", secret_value)
else:
print("Failed to retrieve secret.")Save the above script to a file, e.g., “retrieve_secret.py”.
Run the script:
python retrieve_secret.pyThis script will print the value of the API_KEY from AWS Secrets Manager. Make your that to replay “<region>” with the region where you configured the secret manager.
Read a secret from Azure Key Vault
A quite similar approach works for a key vault in azure as well. To proceed, you need to configure a key vault in azure and already stored the secret you want to retrieve via script in the vault.
Note down the vault URI, as well as the key name, for the example I’ll use “API_KEY” again..
To retrieve a secret from Azure Key Vault using Python, you would typically use the Azure SDK for Python. The libraries azure-identity and azure-keyvault-secrets make this process straightforward. We’re using “DefaultAzureCredential()” function to login, which provides a default chained token credential configuration for development. It attempts to authenticate via multiple methods in this order:
- Environment variables: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID
- Managed identity (if running on an Azure service)
- Azure CLI (if installed and logged in)
One of these functionalities has to be enabled for your environment for the script to authenticate properly. Check the azure python SDK documentation for details if you have any issues.
To get the secret from key vault via script, install the required Python libraries first:
pip install azure-identity azure-keyvault-secretsThen set up Azure CLI and log in (this step is optional but provides an easy way to authenticate):
az loginSave the following snippet as “azure-get-secret.py”:
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
def get_secret_from_key_vault(vault_url, secret_name):
"""
Retrieve secret value from Azure Key Vault.
Args:
vault_url (str): The URL of the Azure Key Vault.
secret_name (str): The name of the secret to retrieve.
Returns:
str: Secret value.
"""
credential = DefaultAzureCredential()
# Create a SecretClient
client = SecretClient(vault_url=vault_url, credential=credential)
# Retrieve the secret
secret = client.get_secret(secret_name)
return secret.value
if __name__ == "__main__":
vault_url = "https://<yourvaultname>.vault.azure.net/"
secret_name = "API_KEY"
secret_value = get_secret_from_key_vault(vault_url, secret_name)
print("Secret value:", secret_value)One executed, the script will then print the value of the API_KEY from your Azure Key Vault. Ensure you have appropriate permissions to access the secret from the Key Vault.
Read a secret from GCP Secrets Manager
As a last example, let’s have a look how to use secrets manager in Google GCP.
To retrieve a secret from Google Cloud Platform’s Secret Manager using Python, you’d typically use the google-cloud-secret-manager library.
Install the required Python library:
pip install google-cloud-secret-managerSet up the Google Cloud SDK and authenticate:
gcloud auth application-default loginThis command will open a browser window for you to log in. Once logged in, it will create application default credentials that the Python script can use.
The Python script to retrieve the secret:
from google.cloud import secretmanager
def get_secret_from_gcp(secret_id, project_id, version_id="latest"):
"""
Retrieve secret value from GCP Secret Manager.
Args:
secret_id (str): The name of the secret to retrieve.
project_id (str): GCP project ID where the secret resides.
version_id (str): Version of the secret to retrieve. Default is "latest".
Returns:
str: Secret value.
"""
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret.
name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
# Get the secret version.
response = client.access_secret_version(request={"name": name})
# Convert the payload to a string and return.
return response.payload.data.decode("UTF-8")
if __name__ == "__main__":
secret_id = "API_KEY"
project_id = "<YOUR_GCP_PROJECT_ID_HERE>"
secret_value = get_secret_from_gcp(secret_id, project_id)
print("Secret value:", secret_value)Replace YOUR_GCP_PROJECT_ID_HERE with your Google Cloud project ID and run the script:
python retrieve_gcp_secret.pyThe script will print the value of the API_KEY from GCP Secret Manager. Ensure you have appropriate permissions (roles/secretmanager.secretAccessor) on the secret in the GCP Secret Manager to access it.
Note: Always remember to keep credentials and sensitive data secure and avoid hard-coding them directly into scripts. Use environment variables, managed identities, or other secure methods of storing and retrieving such data.
That’s it. This should give you at least an idea how to store and retrieve secrets depending on your requirements. To utilize key vault in one of the major hypervisor, it must be ensured, that the environment is configured properly upfront running the examples. There are plenty of tutorials available. providing the details, so I skipped these details here.
Conclusion
Secret management is more than just a technical requirement; it’s a foundational element of cloud security. By efficiently and securely handling secrets in Python, developers fortify their applications against potential breaches, data thefts, and unauthorized access.
Each cloud provider, be it AWS, Azure, or GCP, offers a unique approach to secrets management, yet the underlying principle remains: secrets should be protected with the same vigor as one would guard their personal treasures.
As explored in this post, the tools are available and accessible; it’s up to us to implement them wisely.
Happy coding !!
If you have read it to this point, thank you! You are a hero (and a Nerd ❤)! I try to keep my readers up to date with “interesting happenings in the AI world,” so please 🔔 clap | follow
