AWS Credential Hygiene on Dev Machines

3 min readNov 21, 2017

Background

One of the many things that keeps me up at night is the risk of losing control of privileged AWS keys. The default storage mechanism for the aws cli tools (.aws/credentials in plaintext) has always struck me as very risky and ripe for issues from various fronts (rogue libraries, malware, accidental push to github).

As our team at RAIS grows, the risk of key exposure from developer machines also increases. To help mitigate this risk, we’re using aws-vault and MFA-protected roles for good credential hygiene. This ensures the following:

Long-lived keys exist only in the system keychain
Long-lived keys are only ever invoked to create short-lived keys for use in the development process (~ 8h ttl)
Escalation to powerful keys is completed via role assumption and the resulting keys are very short lived (1h max ttl; 15m by default)
Privileged role assumption requires multi-factor authentication
Keys only ever exist in memory (as environment variables) and are never persisted to disk in plaintext

AWS Vault — Getting started

The folks at 99designs wrote up a great blog post on the goals of the toolset: https://99designs.com.au/tech-blog/blog/2015/10/26/aws-vault/

The source code is available at: https://github.com/99designs/aws-vault

We use the homebrew cask install + validation:

$ brew cask install aws-vault$ codesign -dvv $(which aws-vault) 2>&1 | grep Authority
Authority=Developer ID Application: 99designs Inc (NRM9HVJ62Z)
Authority=Developer ID Certification Authority
Authority=Apple Root CA

Our Approach

AWS developer account setup + IAM policies

We’ve setup an IAM group (rais-developers) that includes the following privileges:

Access to necessary DEV s3 buckets, dynamo tables, parameters and keys required in the development process
Permission to assume two privileged roles with MFA: rais-dev-read-only and rais-dev-admin. The admin-role provides full account privileges and the read-only role full read access to the account (for reports, config checks, etc)

The IAM policy for assume-role looks like this (granted to each developer via the rais-developers group):

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::0000000000000:role/dev-admin-role"
    }
}

Likewise, the trust relationship for each allowable role is setup to require MFA:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::0000000000000:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuthPresent": "true"
        }
      }
    }
  ]
}

Developer Machine Setup

Each of our developers is issued a development account with membership in the rais-developers group. They then enable MFA and generate an associated keypair. The keypair is saved in aws-vault:

$ aws-vault add rais

We then update the .aws/config file to include the necessary profile information:

[profile rais]
region=us-east-1
mfa_serial = arn:aws:iam::0000000000:mfa/netid-dev[profile rais-admin]
source_profile = rais
role_arn = arn:aws:iam::0000000000:role/rais-dev-admin[profile rais-read-only]
source_profile = rais
role_arn = arn:aws:iam::0000000000:role/rais-dev-read-only

Once this is complete, we create some helpers in our .bash_profile:

alias rais-aws-dev="aws-vault exec rais --session-ttl=8h"
alias rais-aws-ro="aws-vault exec rais-read-only"
alias rais-aws-admin="aws-vault exec rais-admin"

Running one of these helpers in the terminal results in a new shell instance where the necessary AWS credentials have been set in the environment:

$ echo $AWS_ACCESS_KEY_ID$ rais-aws-dev
Enter token for arn:aws:iam::0000000000000:mfa/netid-dev:$ echo $AWS_ACCESS_KEY_ID
XXXXXXXXXXXXXXXXXXXX

Once the credentials are available in the environment, aws cli tools understand to find them there rather than in the .aws/credentials file.

Using with Docker development environments

Since we use docker as the base for all of our development projects, we use the docker-compose environment pass-through option to allow these to propagate to containers that require development keys. To do this, it is only necessary to declare that the environment variable exists (i.e., don’t actually set a value):

$ cat docker-compose.ymlbuild: .
  ports:
    - "8888:8888"
  environment:
    - AWS_ACCESS_KEY_ID
    - AWS_SECRET_ACCESS_KEY
    - AWS_SESSION_TOKEN

At runtime, the system will map those environments into the container so they are available.

Regular key rotation

We automatically run a number of configuration checks in our VPC each morning. One of those checks is the credential age — no keys are allowed to be greater than 90 days old (full check code is here). The job fails and and alerts us when any of these checks fail.

Key rotation with aws-vault (i.e., rotating the single set of long-lived credentials) is delightfully easy:

$ aws-vault rotate rais

Closing thoughts

We’ve just started using this approach, but so far it seems reasonable and doesn’t add a lot of friction to development and devops workflows.

Feel free to reach out if you have questions or additional tips around managing credentials on dev machines!