Mitigating a compromised AWS account

My wife contacted me whilst shopping to ask why the credit card was declining. At first I thought it was a terminal/card problem (most likely the AMEX playing up) and attempted to shrug it off, until she mentioned it was both cards on the account.

Upon investigation, I noticed a debit for AU$4,295 from Amazon and was suddenly filled with dread.


What went wrong

In my spare time, I had been working on bringing my wife’s site online. At some point, I had foolishly added my API credentials into a temporary run file (instead of parsing my AWS credentials file). It happened that this file was not so temporary as it was mistakenly committed into my public dotfiles repo on Github (since deleted).

Needless to say, evil-doers and their bots trawl public repo’s and look for this very thing (and all other manner of credential files — SSH keys, OAuth tokens, username/password, etc).

The damage

The aforementioned evil-doers (and/or their bots) spun up the max limit of c4.xlarge spot instances (lots of compute power) across all the inactive AWS regions. In total, during cleanup, I had to kill 100 instances.

Spot instances have a number of features, one which immediately stands out as an advantage to any wannabe douche — Maintain Amazon EC2 Spot Instance Availability with Spot Fleets. In short:

Spot fleets detect disrupted or manually terminated instances, and automatically replace them.

As to what they were doing, who knows, I’ll leave that for Amazon to figure out. Bitcoin mining???

Actions Amazon took

I just want to point out that Amazon actually detected the (possible) compromise quite early on and raised a case on the account to alert the root user to the fact.

When reading the case, they had narrowed in on the cause and provided steps to correct the compromise.

Three attempts at web contact (send an email to account contacts) were tried at which point the case was closed.

Why it went unawares

Well at least until the credit card was debited.

  1. The root account was held by my wife and she was(is) oblivious to anything tech-related. As this was quickly spun up and focus was on getting her product to a point where it was ready to go into production, there had been no emphasis on setting the account up properly, the main one being to add myself as the Account and Security Point of Contact.
  2. I only worked within the AP-Southeast-2 region, and when I log in I see no indication of what is happening in other regions (nor did I think to check on occasion).
  3. There were no billing alarms.

Mitigating the infallible

Infallibility is part of the human condition. Posting secure credentials into a public space (akin to losing your car keys) is a silly thing to do, but it can happen and for some of us, will happen.

Therefore, what precautions can we put in place to help mitigate the impact when this occurs? The next section lists some actions to take.


Mitigation

Some actions to take to help protect an AWS account.

Using managed policies

Creating and using custom policies that only include specific actions which are locked down to specific resources is a great way to restrict access to your AWS account.

Please note: Not all actions are resource assignable and instead can only be assigned to “*”. The Policy Simulator can help you determine whether an action is resource assignable.

An example policy highlighting use of resource assignable/non-assignable actions:

{
"Version": "2012–10–17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:Describe*",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws:ec2:us-east-1:1234567890:network-interface/*",
"arn:aws:ec2:us-east-1:1234567890:volume/*",
"arn:aws:ec2:us-east-1:1234567890:key-pair/*",
"arn:aws:ec2:us-east-1:1234567890:security-group/*",
"arn:aws:ec2:us-east-1:1234567890:subnet/*"
]
},
{
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws:ec2:us-east-1:1234567890:instance/*"
],
"Condition": {
"StringEquals": {
"ec2:InstanceType": "t2.micro"
}
}
}
}
}

Billing alerts

It is worth noting that billing alerts would be required for all regions, however, when combined with policies restricting actions to specific regions, you should only need to set it up for those aforementioned regions.

Refer to the AWS documentation for further reading.

Third party tools

AWS console does not provide (that I have found) a single view of all active resources within all regions, which would be especially handy at login to see this information.

I have started playing around with a python package called Skew.

One simple solution is:

# create `$HOME/.skew` and add:
$ cat << EOF > $HOME/.skew
---
accounts:
"123456789012":
profile: dev
"234567890123":
profile: prod
EOF
# Run the following python script:
$ python
>>> import skew
>>> arn = skew.scan(‘arn:aws:ec2:*:*:instance/*’)
... for resource in arn:
... print(
... "id: %s; type: %s; az: %s" % (
... resource.data['InstanceId'],
... resource.data['InstanceType'],
... resource.data['Placement']['AvailabilityZone']
... )
... )
...

# OUTPUT
id: i-a561586b; type: m3.large; az: us-east-1a
id: i-2ce611f4; type: m3.medium; az: us-east-1b
id: i-476fb799; type: t2.micro; az: ap-southeast-2a
id: i-45c1178c; type: m3.large; az: ap-southeast-2b

Conclusion

First and foremost, don’t put your credentials into a public space. That being said, it is still good practice to ensure you have provisioned mechanisms to a) alert you to the fact that something untoward has happened and b) minimise the damage unauthorised access can have, preferably without impacting your developers or you.

A single golf clap? Or a long standing ovation?

By clapping more or less, you can signal to us which stories really stand out.