Security in the Cloud: What I learned when AWS (almost) billed me for $29,594 in one day.
Alright it wasn’t $29k in one day it was $8,570.60 in less than 20 hours but twenty-nine was the projected. And I know I’m not the first to write about this, but here you go.
Now for some companies this would be a cheap day (Netflix AWS bill has over 700 million lines) but when you are running only Serverless projects (I usually don’t even break the free tiers) this bill would be extremely high.
The bill was for running 20 very large (m4.16xlarge) EC2 instances per region for over 20 hours. Why would I do this? Well I did this because I’m an naive idiot! I accidentally, without thinking, pushed code with hard-coded Admin AWS security credentials to a public GitHub repo.
I know I’m going to loose you now because your thinking, well why would anyone do that, what did you expect, what a rookie move, no-wait, a beyond a rookie move!
Let me justify my actions to make me feel better. I was working on a small test project, the kind of project you just code as fast as possible, hard-code something (security credentials – without at least setting up an external .env). Then blindly push to a public repo thinking your pushing to a private one. Easy mistake, right?
Then while I was sleeping, I had a security breach with my IAM keys. AWS were very quick and noticed the unusual activity almost immediately and emailed me of the breach. However I was asleep while my IAM keys were used to launch 320 very large EC2 instances and let running.Since I mentioned I run Serverless environments, any EC2 instance is unusual activity. When I logged back in, I immediately realised what I’d done wrong. So my first move was to make all my IAM user keys ‘inactive’. Remove that stupid public repo. Terminate all instances and email AWS and beg for mercy as I wouldn’t be able to pay even the start-up hour of 320 instances.
So, what did I learn? (beside the obvious)
- Practicing Least Privilege. I started creating separate IAM user and credentials for each service. Each having just enough privileges (get rid of those ‘*’). Its frustrating to setup and as you go you find out your service needs more permissions than you first thought. It’s definitely worth the time to set this up.
- Restrict your account to only the regions you need. I didn’t know this was possible. Login to your AWS console, under IAM, Account Settings, you can restrict any usage to regions you don’t plan on using in the near future. This is helpful if you only ever focus on one or two regions like myself.
- Check and re-check any ‘*’ in any IAM policy. When you’re developing its the easiest thing to do is add * in IAM policies. Especially out of frustration, when you just want to get it working. However theres a very slim chance this is the correct move. This is why I have even restricted my ‘admin’ IAM group policy. In my case, I added ‘Deny’ statements to restrict launching EC2 instances and many other AWS services (to protect against myself).
- Rotating Access Keys. Another obvious one. However, easily overlooked and regarded as over-kill. But helps to limit access keys going stale.
- Use Parameter Stores. Especially in a serverless environment. Instead of playing around with .env files. Using Parameter Store allows you to keep one source of truth (across your whole team) as well as encrypted variables. And, when using the Serverless framework or SAM, your Env variables never touch a local machine.
Security in a constant learning experience.
I will continue to add to this post as I continue to learn.