Securely managing AWS credentials using AWS Vault

Ludwig Dieter trying to open the legendary Valkyrie safe in Netflix’ Army of Thieves

Our mission at Borneo is to build guardrails for the new data economy. We thus take the security of our own cloud infrastructure seriously.

One key aspect of that is to ensure that the AWS credentials our engineering and DevOps team uses to access our organization’s AWS accounts are kept secure, and rotated frequently. For managing AWS access keys, used to access and manage resources via AWS’ command line interface (CLI) and APIs, we use AWS Vault.

What is AWS Vault?

AWS Vault stores IAM credentials in your operating system’s secure keystore and then generates temporary session credentials from those long-lived user credentials to use in your shell and other applications. It’s designed to be complementary to the AWS CLI tools, and is aware of your profiles and configuration in ~/.aws/config.

In this article, I will explain how we use AWS Vault to access different accounts and roles in our AWS organization. For more information about AWS Vault, please refer to the project readme and usage guide.

How we use AWS Vault?

Following security best practices, our AWS organization consists of a number of different accounts for development, testing, staging, and production workloads, as well as separate accounts for security monitoring, shared infrastructure, etc. All IAM user accounts for our team are managed in the organization’s management, or “root” account. (Recommended best practice is to use a separate, dedicated account for that.)

Based on their team and function, each team member has permission to use, or “assume”, IAM roles in one or more of these accounts. This might sound complicated, but the aws-vault command line tool makes it super easy to quickly switch between different accounts and roles, all while keeping your main AWS access key secure!

aws-vault works on macOS, Windows, and Linux. If you are using a mac and the Homebrew package manager — like most of our team — you can simply install aws-vault using brew install --cask aws-vault. For other platforms, please refer to the aws-vault readme.

Next, you need to add the AWS access key and secret key for your IAM user to the vault. You use the aws-vault add for that and specify the name of the profile the key belongs to:

> aws-vault add borneo-root
Enter Access Key ID: AKIA**************** 
Enter Secret Access Key: 
Added credentials to profile "borneo-root" in vault

The access key is now securely stored in your OS’ secure keystore, e.g. the macOS Keychain, or the Windows Credential Manager. aws-vault supports other backends — called “vaults” — as well.

Securing IAM user accounts using multi-factor authentication (MFA) is another important security measure, and we enforce MFA for all users in our org. To ensure the (virtual) MFA token gets used when aws-vault assumes an IAM role, you need to configure the MFA serial in your AWS config (~/.aws/config) — the account ID in the MFA serial should be that of your root account:

[profile <name>]
mfa_serial = arn:aws:iam::<AWS account ID>:mfa/<username>
region = <AWS region>

For example:

[profile borneo-root]
mfa_serial = arn:aws:iam::123456789012:mfa/jan
region = ap-southeast-1

Next, create a separate profile entry in ~/.aws/configfor each IAM role that you want to assume. In that profile, you specify the IAM role, as well as the source profile to use when assuming the role, i.e. your root account profile:

[profile <name>]
role_arn = <ARN of the IAM role to assume>
mfa_serial = arn:aws:iam::<AWS account ID>:mfa/<username>
source_profile = <source profile name>
region = <AWS region>

For example, to assume the admin role in our dev account, I would add a new profile like this:

[profile borneo-dev]
role_arn = arn:aws:iam::789012345678:role/admin
mfa_serial = arn:aws:iam::123456789012:mfa/jan
source_profile = borneo-root
region = ap-southeast-1

Note that the MFA serial should be the same as in your root account profile, since the MFA token is associated with the IAM user in that account.

I like to add a add a few more optional config settings to my Bash profile, to make working with aws-vault and the AWS CLI/SDK a bit more comfortable:

# Pop up a GUI for the MFA prompt on macOS.
export AWS_VAULT_PROMPT=osascript
# Ensure the AWS SDK uses the ~/.aws/config file.
export AWS_SDK_LOAD_CONFIG=true
# Extend the default session duration when chaining profiles.
export AWS_CHAINED_SESSION_TOKEN_TTL=16h

Please refer to the AWS Vault usage guide for a full list of supported environment variables.

Working with Session Credentials

There are several options to use aws-vault to create short-lived session credentials. To execute a single command with temp. session credentials generated using a specific AWS profile, you can use the aws-vault exec command:

aws-vault exec <profile> -- command

For example:

> aws-vault exec borneo-dev -- aws sts get-caller-identity
{
    "UserId": "AROAXXXXXXXXXXXXXARND:1234567890123456789",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/admin/1234567890123456789"
}

If you don’t specify a command, running aws-vault exec <profile> will create a new shell session for you, and populate it with the temp. session credentials. You can use the AWS CLI or other tools in this shell and they will automatically use the right profile.

Starting a Local Metadata Service

If you need valid credentials for a specific profile over a longer period of time, i.e. for development work in your dev account, the most convenient option is to use aws-vault to launch a local service that mimics the AWS EC2 instance metadata service.

When you use the --ec2-server option with the aws-vault exec command, it launches a local instance of the EC2 instance metadata service in the background. This service will generate session credentials using the specified profile on demand. It will also automatically refresh the session credentials, when they are about to expire.

All recent versions of the AWS CLI and the AWS SDKs support retrieving session credentials via the metadata service automatically, i.e. there is no further configuration required. To use the metadata service, the commands which need the session credentials don’t need to run in the same shell session either.

To launch a new instance of the EC2 instance metadata service, run:

aws-vault exec --ec2-server <profile>

There are a couple of things to be aware of when using the --ec2-server option:

• To launch the metadata service, aws-vault needs to bind to a privileged port, which requires root/administrator permissions. So you may be asked for your sudo password.
• The exec --ec2-server command starts a new shell session (unless you specify a different command to execute). If you terminate this shell session, the metadata service will be terminated as well.
• You can only run a single instance of the metadata service at a time.

Last, not least, this approach exposes the local metadata service to all programs running on the same machine. That can be a security issue, if any untrusted processes can get access to your AWS session credentials. Thanks to aws-vault, the long-lived user credentials are never exposed, but the ability to use them to connect to AWS is!

To avoid this problem, you can also use the --ecs-server option. It works similarly to the --ec2-server option, but starts a local ECS container metadata service instead. This service binds to a random, non-privileged, port. The port number as well as an authentication token are exported into the shell session started by the command. This way, only processes that have access to the auth token can use the service to fetch AWS session credentials. The AWS CLI and SDKs will automatically detect and use this container metadata service as well.

Rotate your AWS credentials frequently

One of the benefits of using aws-vault is that it makes it easy to rotate your AWS access key — something that you should do frequently. Amazon recommends that you rotate your key every 90–180 days as a security best practice.

Using the aws-vault tool this takes just a couple of seconds:

❯ aws-vault rotate borneo-root
Rotating credentials stored for profile 'borneo-root' using a session from profile 'borneo-root' (takes 10-20 seconds)
Creating a new access key
Created new access key ****************PSCC
Deleted 1 sessions for borneo-root
Deleting old access key ****************3KMP
Deleted old access key ****************3KMP
Finished rotating access key

If you have made it this far, you should now have a good idea of how you can use the AWS Vault tool to securely store your AWS access key(s), and to generate temporary session credentials to work with one or more IAM roles. Thanks to the team at 99designs, and the whole open-source community behind aws-vault, who helped develop, maintain, and improve this great tool!

To understand more about how we can help you with driving data privacy compliance, you can request a quick demo to get started!

Borneo | Real-time data security and privacy observability