I was recently victim of an insecure malpractice in Rails involving Carrierwave, Fog, Amazon S3 and a Hacker. I wanted to share my story with other developers so that it doesn’t happen to someone else.

The Story

It’s Thursday, 3AM in Toronto. I go on my startup website before deploying and I realize that all the pictures are down. Alert: Something is wrong with my image hosting server. I then try to login in my AWS console and Amazon is telling me that my account has been shut down. ??? Still not sure what’s going on, I click everywhere and realize that the Bill & Payment page is still working ( of course..)… There, something terrifying is waiting for me…..

It’s been only 17 days in April, how the heck did my bill mount to that amount?

What Happened

I was recently overwhelmed with all my contract jobs so I hired a remote developer to help me with my startup. After asking him to sign a Non Disclosure Agreement (NDA), I added him to my private Github repository. He then forked my repository and publicly exposed a copy of it on his own repository. (What was he thinking?) It still absurd to me that he didn’t think of keeping it private even after signing an NDA… This negligence leaded to an exploit. Someone out there found the key and used it to create 68 EC2 instances in my AWS it was at least 12 per server region. My bill suddenly went from $15.00/mo to +$10,000 in a matter of a few days. It took me sometimes to realized that because my AWS billing alerts weren’t setup. It would have probably taken a few more days before I found out about if I didn’t decide to wake up that night and check one of my site.

As soon as I noticed it, I immediately removed my compromised key then called Amazon and terminated all the instances.

In order to allow me to shutdown the instance Amazon had to give me back access to my account since it was shut down. During that process my bill moved to:

What’s Then?

Well, “Amazon feel very sorry for the concern the charges are causing me due to unauthorized access on my account”. They are currently processing a concession request for all the unauthorized charges. There are not guaranteeing that they will wave the fees on my amount but I am really hopeful for it.

ABC. Always Be Cautious

The developer wasn’t the only person at fault, I should have removed the hard coded value before handing him access to my repository. Actually, they shouldn’t have been there at the first place.

The only reason I put them there was because I was migrating my application from one server to another (Heroku to DigitalOcean) so I temporarily hardcoded the value so that I could refer to them if needed in development (especially when I quickly switch from one laptop to another). I would recommend to always play it safe with this even when you are the only developer. I learned a lot of thing from this negligeance. Unfortunately I have the curse to learn my lessons the HARD WAY. I have made many mistake , ran into many bugs and obstacle but I’ve always learned from each experience.

A man must be big enough to admit his mistakes, smart enough to profit from them, and strong enough to correct them. (John C. Maxwell )

Always avoid hard coded security keys

I had temporally hardcoded the security keys because I was the only one working on the project and when I gave access to the developer; I forgot to remove them and use AWSIdentity and Access Management instead. We tend to be negligent with security keys even with services we pay for. That’s a habit I noticed on a lot of developer.

Don’t Encourage Bad Habit

Many tutorials tends to encourage these bad habits as they tell us where to put the keys but not how to make them available securely. If you google a stack overflow on how to setup Carrierwave/Fog on Rails it will tell you to create a fog.rb as such:

Bad Habit:

CarrierWave.configure do |config| config.storage = :fog config.fog_credentials = { provider: ‘AWS’, aws_access_key_id: ‘YOUR_KEY_ID_HERE’, aws_secret_access_key: ‘YOUR_KEY_HERE’ }

config.fog_directory = ‘MY_S3_BUCKET_HERE’ config.fog_public = false

“YOUR_KEY_ID_HERE” and “YOUR KEY_HERE” are exactly the type of recommendation that encourage bad behaviour. If you are recommending someone to do something like that always use ENV[“..”] so that they know that they have to create an environment variable to make it more secure and not directly available to the app.

Good Habit:

aws_access_key_id: ENV[‘YOUR_KEY_ID_HERE’],

aws_secret_access_key: ENV[‘YOUR_KEY_HERE’]

If you would like to use environment variable in development I would recommend to use the dotenv gem (https://github.com/bkeepers/dotenv) or the figaro gem (https://github.com/laserlemon/figaro).

Never give root access ( even to people you trust )

When people are not the one paying, they usually care less about the consequences of negligence. As long as my repository was private I wasn’t too worried about it but I should have anticipated the worse. There a french proverb that says:

“Mieux vaux prévenir que guérrir”

which literally mean “It’s better to prevent than to heal” but the english equivalent is actually

“ A stitch in time saves nine”

Because my key was hard coded, I had given that developer the main credential for my AWS account so he could do whatever he wants. If I had been cautious, that would have never happened even with a public repository. Always Be Cautious.

Best Practices


As a best practice, it is recommended not having root access keys at all on Amazon Web Service. Instead, you should use AWSIdentity and Access Management (IAM), which enables you to securely control access to your AWS services and resources for your users.

On Github

Github has a great article on how to remove sensitive data from your repository ( https://help.github.com/articles/remove-sensitive-data ).

Remove Sensitive Data (Github)

Another simple trick to avoid committing sensitive data to Github is to add all your app configuration key into a yml file and adding that file to your .gitignore so that you end up never committing them.

On Unix

Setting Unix Environment Variables is pretty straight forward. All you need to do is adding the following to your ~/.bashrc file. You can do this from the terminal:



Related Stories

Online Thief Steals Amazon Account to Mine Litecoins in the Cloud http://www.coindesk.com/online-thief-steals-amazon-account-mine-litecoins-cloud/

Did you read my article How I Trained To Lear Rails? Since the Creator of Ruby recommended it more than 90,000 people have read it https://medium.com/how-i-learned-ruby-rails/e08c94e2a51e

Subscribe to my blog


Ruby or Rails

Anything about ruby or rails. And learning tips.

Richardson Dackam

Written by

Self-taught Designer, Developer, Videographer & Maker. Host of Shortcut to Mastery Podcast. Youtuber. Founder of Timedlinks.com, Checkups.io, MyReater.com

Ruby or Rails

Anything about ruby or rails. And learning tips.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade