A Very Expensive AWS Mistake

Last Thursday, I was working on my Flatiron Module 3 project and attempting to figure out how to use image uploads in conjunction with AWS’ S3 (Simple Storage Service). After many failed attempts, I decided to take a break for the night, and try again the next day. However, when I awoke the next morning with a mysterious email stating that I had exceeded my free plan limits. This was very confusing, considering I couldn’t get anything working. I immediately logged into my account and to my horror, I found that I had racked up a $3,000+ bill! The charges were all a result of EC2 services, and my account had 20 servers running in each of AWS’ global regions. I immediately contacted AWS’ customer support team, and thankfully, they were very responsive, understanding and helpful. They directed me to secure my account by deactivating my access tokens and to shut down all the running instances of EC2 servers. By the time my account was secured, my bill had climbed to over $7,000.

Not only was I discouraged that I couldn’t figure out how to upload images on my website, I now made a potentially very costly mistake. My first instinct was to shut down my AWS account completely, curse image uploads, and never touch the stuff again out of sheer embarrassment. However, I’m going to use this mistake as a learning opportunity, and hopefully this post can prevent others from making the same mistake.

How did my AWS Account Get Hacked?

I made a rookie mistake, and I exposed my AWS IAM credentials on Github. At the time, I didn’t know the severity of the consequences. I was so fixated on getting my project working that I had absent-mindedly committed my code to Github, with my access token and secret key exposed. I knew I shouldn’t post API tokens on Github, but I never thought someone would actually want to steal my credentials. Who was really going to look at my beginner project repositories?

As it turns out, there are bots that not only scan my repositories, but all public repositories on Github, searching for exposed tokens. AWS tokens are particularly valuable because they are tied to a credit card through a “pay-as-you-go” program if you exceed the free limits. With bots in the picture, free limits can be exceeded very quickly.

EC2 (Elastic Cloud Computing), the resource targeted in the hacker attack, provides resizable servers that can be set up very quickly. Although I don’t know for what exact purposes the unauthorized EC2 servers on my account were being used, these virtual servers are often used for cryptocurrency mining. Virtually any kind of cryptocurrency can be mined on EC2 (with varying degrees of efficiency) and recent articles have discussed mining of Bitcoin, Ethereum, and Litecoin. Here’s a nice how-to article, for those interested in mining cryptocurrency legally: How to mine bitcoins using an AWS EC2 instance…

Steps to Protect Your AWS Account

1. Protect your API keys from the get-go

Going forward, when starting any project, I will always protect my access tokens from the very beginning. For Ruby projects, Figaro is a good option, as it makes it easy to securely configure Rails applications using using ENV and a single YAML file. Setting up Figaro can be done in a few steps:

Add the Figaro gem to your Gemfile and bundle install .

gem 'figaro'

Then, install Figaro via the Terminal. This will create a config/application.yml file and add it to your .gitignore :

$ bundle exec figaro install

Add the necessary access tokens and secret keys to the application.yml file:

# config/application.yml

aws_access_key_id: "xxx"
aws_secret_access_key: "xxx"

These keys are added as environment variables and can be easily accessed in your application.

AWS_ACCESS_KEY = ENV['aws_access_key_id']
AWS_SECRET_KEY = ENV['aws_secret_access_key']

2. Don’t use root access keys. Create individual IAM users

AWS recommends creating IAM users and giving them only the permissions they need (even if you’re the only person using the AMS account). The root account provides unrestricted access to your AWS resources and can leave your account unnecessarily exposed. Search for IAM in the console, and the dashboard will walk you through the steps of securing your account.

AWS’ IAM Dashboard will guide your through each security measure

Once you create individual IAM users, it is important to grant least privilege to those users — that is, granting only the permissions required to perform the task. You decide user privileges by attaching managed policies through IAM user groups. For my purposes, I gave myself full S3 access:

3. Monitor activity in your AWS Account through CloudWatch and Cloud Trail

CloudWatch monitors your AWS resources in real time. CloudWatch can send notifications or make changes to the resources you are monitoring based on rules that you define. For security purposes, you can choose to stop or terminate EC2 instances that exceed your predefined limit. This is likely most helpful for AWS users that are already running EC2 instances and wish to monitor activity.

CloudTrail logs AWS API calls and related events made by your AWS account. It captures activity as a “CloudTrail event” and this information can be viewed in your CloudTrail event history or sent to an S3 bucket for storage.

For a full list of AWS best practices, see here. I would highly recommend reading in full before starting any AWS project. Some of this may seem like overkill for a small project, but this experience has taught me that you can truly never be too careful on the internet.