Securing Local AWS Credentials

I’ve assisted with a volume of high profile data breaches at AWS companies over the last couple of years. A root cause often involved a leaked AWS key.

In those specific cases, keys were often used days, weeks, or months after the initial key leak. Oddly enough, I’ve also been intentionally leaking keys into the wild, and they’re consistently used days or weeks later as well:

It’s important to move completely away from IAM Access Keys and towards the use of roles, where credential exposure is altogether very limited.

Let’s talk about how you can do this with your local CLI access to AWS. This will also make it easier to work in a large role based environment where instances and lambda functions are also assuming roles.

Your interaction will then look something like this (see --profile admin and “Enter MFA code”):

Overview

There are a few ways that an IAM user can cause more damage than intended:

• IAM user has a bunch of dangerous policies attached.
• IAM user in groups, groups with a bunch of dangerous policies attached.
• IAM user temporarily assumes roles with dangerous policies attached.

Dangerous policy: PutPolicy allows you to escalate privilege without limit

When used by an engineer, the first two ways of authenticating rely on a key/secret pair that Amazon recommends storing in a credential file associated with the user on disk in ~/.aws/credentials. If stolen, an attacker has that user’s permissions until you’ve noticed a breach.

The third method reduces the impact of an attack by reducing the exposure of privileged credentials from all the time… to only sometimes. These credentials live in .aws/cli/caches temporarily. Throw MFA on role assumption, and it’s a nice security win.

First we’ll walk through an example, then describe how incident response would work when it ultimately fails. Remember, everything fails eventually.

Create User

1. Create a user.
2. Enable MFA and set it up and take note of the ARN for the user’s MFA device (in the user’s details).
3. Then apply the SecurityAudit managed policy. This will allow you to perform read only actions to metadata on the entire account.

Create the Role

1. Create a role and name it admin.
2. For the role type, select “Role for Cross-Account Access” and “Provide access between AWS accounts you own”. Enter the same account ID for the account you’re in (found in account settings) and check the “Require MFA” box.
3. When you attach a policy, attach the AdministratorAccess managed policy. Of course, you can change this policy to whatever role you feel is appropriate, if this is too much.
4. Once it is created, note the ARN of the role.

Allow the User to Assume the Role

Now we have a role that the user can assume, but the user has no permission to assume it.

1. Get the ARN for the role, which can be found by inspecting the role in the console.
2. Go back to the user and inspect their permissions, we’ll be adding a new policy. In this policy, we want to Allow the user to AssumeRole to the ARN of the role we created, and require that MFA is required to assume it.

Here’s the policy language below (change as needed)

{
 “Version”: “2012–10–17”,
 “Statement”: [
 {
 “Effect”: “Allow”,
 “Action”: [
 “sts:AssumeRole”
 ],
 “Resource”: [
 “arn:aws:iam::[YOUR ACCOUNT ID]:role/admin”
 ],
 “Condition”: { “Bool”: { “aws:MultiFactorAuthPresent”: true } }
 }
 ]
}

Make sure this policy is attached to the user.

Configure the CLI

Update your ~/.aws/credentials and ~/.aws/config file with the credentials and configuration for our user.

1. Make traditional credentials for the user account. This example uses your [default] profile, so be sure to back up your current default user credentials somewhere if they exist.
2. You will also add the configuration for the role in [profile admin]. This configures a new profile for your CLI that identifies which user it will assume the role from ([default]) and which MFA device to prompt you with.

[default]
aws_access_key_id = ACCESSKEY
aws_secret_access_key = SECRETKEY
[profile admin]
role_arn = arn:aws:iam::[YOURACCOUNTID]:role/admin
source_profile = default
mfa_serial = arn:aws:iam::[YOURACCOUNTID]]:mfa/[NAME OF USER]

Make sure the mfa_serial is from the user’s account. If you run into problems here, it’s probably because you didn’t create an MFA device with the user when you originally created it.

Now we have local credentials with minimal permissions on the [default] user, with the ability to MFA into a temporarily more powerful role.

Test It

We’ll illustrate how your normal user cannot make changes to infrastructure, but can after it has MFA’d and assumed a role temporarily.

If you run aws ec2 describe-instances, it should succeed. It will run describe-instances with the user you made and SecurityAudit permissions.

If you run aws s3api create-bucket --bucket this-should-fail, it should fail because SecurityAudit can’t create, modify, or delete anything.

Next, we introduce --profile to the CLI command, which is configured to prompt for MFA into the admin role.

We’ll test it by creating a bucket. Run aws —-profile admin s3api create-bucket --bucket this-should-succeed, you will be prompted for your MFA code and you’ll successfully create the bucket. The admin role was responsible for the action and AdministratorAccess was the policy it referenced.

You can dive into ~/.aws/cli/cache and inspect the temporary credentials it used to create the bucket. They’re just like normal AWS credentials, with a session key. That key is useless to an attacker (and you) in only an hour.

Benefits

The primary security benefit is the temporary keys that carry your privileged access. An attacker (malware, physical access, etc) can only reliably access the read only (SecurityAudit) key. The key that accesses the role only lives on disk when you request it with MFA (which an attacker can’t do at will) and is only useful temporarily for the hour you’ve requested it.

Downsides

It’s much trickier to set up than just slamming some root / admin keys into an ~/.aws/credentials file and accessing everything. Temporary access means you’ll have to MFA repeatedly for a multiple-hour project, which might convince you to add more continuous permissions to the user account itself. If you’re in a federated environment, assume’d roles can last up to 12 hours, which is better.

Also, Lyft identified that roles are not yet good to go with multi-tenant containerized environments, if that’s how you’re built. Though they offer a place to contribute to a solution.

Incident Response

Temporary security tokens are a perk, not a silver bullet.

If an attacker wants to execute an attack within the hour that you’ve used the key, they could succeed simply by moving fast. But then again, there may be many reasons why they can’t move fast and breach you immediately.

You can’t make that prediction.

This is one of many parts of increasing attacker cost. Always prepare to investigate when they can foot the bill. It’s extremely important to have CloudTrail logging setup with log file validation.

CloudTrail investigations are a little bit trickier with roles, because the identity information related to any actions taken by an assumed role are not immediately attached to the user that assumed the role. The steps to investigate this are well described here, but are quickly described as:

1. Find the exact log where the abuse began, and note the userIdentity.arn value.
2. Search logs for the last occurrence of AssumeRole* with responseElements.assumeRoleUser.arn equal to the value from #1
3. You’ll be investigating the information that appears in userIdentity for abuse, along with the sourceIPAddress that made the call.
4. If you are doing role assumption across accounts, you’ll need to take the sharedEventID and search for it in the other account and find it assuming the role in your account to discover the user that did it in the remote account.

If you did not intentionally grant role access to another account, and it assumed it anyways… congratulations, you’ve discovered a backdoor in your account. At least you logged it, right!?

Conclusion

There are several benefits in moving away from IAM user access keys and towards roles in a production environment, and getting familiar with it on the endpoint can help get an organization away from them too.

@magoo

I’m a security guy, former Facebook, Coinbase, and currently an advisor and consultant for a handful of startups. Incident Response and security team building is generally my thing, but I’m mostly all over the place.