Securing Local AWS Credentials

Ryan McGeehan
Nov 23, 2016 · 6 min read

I’ve assisted with a volume of high profile data breaches at AWS companies over the last couple of years. A root cause often involved a leaked AWS key.

In those specific cases, keys were often used days, weeks, or months after the initial key leak. Oddly enough, I’ve also been intentionally leaking keys into the wild, and they’re consistently used days or weeks later as well:

It’s important to move completely away from IAM Access Keys and towards the use of roles, where credential exposure is altogether very limited.

Let’s talk about how you can do this with your local CLI access to AWS. This will also make it easier to work in a large role based environment where instances and lambda functions are also assuming roles.

Your interaction will then look something like this (see --profile admin and “Enter MFA code”):

Overview

  • IAM user has a bunch of dangerous policies attached.
  • IAM user in groups, groups with a bunch of dangerous policies attached.
  • IAM user temporarily assumes roles with dangerous policies attached.

Dangerous policy: PutPolicy allows you to escalate privilege without limit

When used by an engineer, the first two ways of authenticating rely on a key/secret pair that Amazon recommends storing in a credential file associated with the user on disk in ~/.aws/credentials. If stolen, an attacker has that user’s permissions until you’ve noticed a breach.

The third method reduces the impact of an attack by reducing the exposure of privileged credentials from all the time… to only sometimes. These credentials live in .aws/cli/caches temporarily. Throw MFA on role assumption, and it’s a nice security win.

First we’ll walk through an example, then describe how incident response would work when it ultimately fails. Remember, everything fails eventually.

Create User

  1. Enable MFA and set it up and take note of the ARN for the user’s MFA device (in the user’s details).
  2. Then apply the SecurityAudit managed policy. This will allow you to perform read only actions to metadata on the entire account.

Create the Role

  1. For the role type, select “Role for Cross-Account Access” and “Provide access between AWS accounts you own”. Enter the same account ID for the account you’re in (found in account settings) and check the “Require MFA” box.
  2. When you attach a policy, attach the AdministratorAccess managed policy. Of course, you can change this policy to whatever role you feel is appropriate, if this is too much.
  3. Once it is created, note the ARN of the role.

Allow the User to Assume the Role

  1. Get the ARN for the role, which can be found by inspecting the role in the console.
  2. Go back to the user and inspect their permissions, we’ll be adding a new policy. In this policy, we want to Allow the user to AssumeRole to the ARN of the role we created, and require that MFA is required to assume it.

Here’s the policy language below (change as needed)

{
“Version”: “2012–10–17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“sts:AssumeRole”
],
“Resource”: [
“arn:aws:iam::[YOUR ACCOUNT ID]:role/admin”
],
“Condition”: { “Bool”: { “aws:MultiFactorAuthPresent”: true } }
}
]
}

Make sure this policy is attached to the user.

Configure the CLI

  1. Make traditional credentials for the user account. This example uses your [default] profile, so be sure to back up your current default user credentials somewhere if they exist.
  2. You will also add the configuration for the role in [profile admin]. This configures a new profile for your CLI that identifies which user it will assume the role from ([default]) and which MFA device to prompt you with.
[default]
aws_access_key_id = ACCESSKEY
aws_secret_access_key = SECRETKEY
[profile admin]
role_arn = arn:aws:iam::[YOURACCOUNTID]:role/admin
source_profile = default
mfa_serial = arn:aws:iam::[YOURACCOUNTID]]:mfa/[NAME OF USER]

Make sure the mfa_serial is from the user’s account. If you run into problems here, it’s probably because you didn’t create an MFA device with the user when you originally created it.

Now we have local credentials with minimal permissions on the [default] user, with the ability to MFA into a temporarily more powerful role.

Test It

If you run aws ec2 describe-instances, it should succeed. It will run describe-instances with the user you made and SecurityAudit permissions.

If you run aws s3api create-bucket --bucket this-should-fail, it should fail because SecurityAudit can’t create, modify, or delete anything.

Next, we introduce --profile to the CLI command, which is configured to prompt for MFA into the admin role.

We’ll test it by creating a bucket. Run aws —-profile admin s3api create-bucket --bucket this-should-succeed, you will be prompted for your MFA code and you’ll successfully create the bucket. The admin role was responsible for the action and AdministratorAccess was the policy it referenced.

You can dive into ~/.aws/cli/cache and inspect the temporary credentials it used to create the bucket. They’re just like normal AWS credentials, with a session key. That key is useless to an attacker (and you) in only an hour.

Benefits

Downsides

Also, Lyft identified that roles are not yet good to go with multi-tenant containerized environments, if that’s how you’re built. Though they offer a place to contribute to a solution.

Incident Response

If an attacker wants to execute an attack within the hour that you’ve used the key, they could succeed simply by moving fast. But then again, there may be many reasons why they can’t move fast and breach you immediately.

You can’t make that prediction.

This is one of many parts of increasing attacker cost. Always prepare to investigate when they can foot the bill. It’s extremely important to have CloudTrail logging setup with log file validation.

CloudTrail investigations are a little bit trickier with roles, because the identity information related to any actions taken by an assumed role are not immediately attached to the user that assumed the role. The steps to investigate this are well described here, but are quickly described as:

  1. Find the exact log where the abuse began, and note the userIdentity.arn value.
  2. Search logs for the last occurrence of AssumeRole* with responseElements.assumeRoleUser.arn equal to the value from #1
  3. You’ll be investigating the information that appears in userIdentity for abuse, along with the sourceIPAddress that made the call.
  4. If you are doing role assumption across accounts, you’ll need to take the sharedEventID and search for it in the other account and find it assuming the role in your account to discover the user that did it in the remote account.

If you did not intentionally grant role access to another account, and it assumed it anyways… congratulations, you’ve discovered a backdoor in your account. At least you logged it, right!?

Conclusion

@magoo

Starting Up Security

Guides for the growing security team

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store