Connecting AWS’s S3 to your Heroku hosted Rails App, or…

a guide to 1) not wasting 7 days getting your AWS repeatedly hacked, 2) not reformatting your production.rb file 100 times because Heroku keeps spitting back 500 server errors + general tidbits that you wish your coder friends would have told you before you started trying to get your app live on the web.

Assumptions: assuming you are using Rails as your backend, you already have the Ruby gem Paperclip successfully storing your files in your local repo on upload, you have correctly created a Heroku app to host the production version of your backend and have added an S3 bucket to your AWS account to manage file storage for your app, read on. Otherwise, this guide is not for you.

March 4th, 2:32 pm. The day this bloody battle begun. March 10, 10:21 pm. The day the battle was won. Location: Ruby on Rails land. Combatants: me vs. the Axis of Junior Developer Evil, a.k.a. Heroku, Amazon Web Service’s file storage tool S3, a Ruby gem known as Figaro, and Environment Variables. Victor: me. Since this format worked pretty well for my last blog let me point those looking for answers now to the section “GIVE ME THE ANSWERS NOW!” For those that enjoy a good story, grab a cup of tea, a nice blankey, curl up by the virtual fire and read on.

In the process of sprucing up my resume I felt it necessary to get my recent projects online not only so people could poke around if they so desired but also so I could learn the intricacies of deploying an app to a bunch of hosting services. After my recent success hosting an SPA (Single Page App) on GitHub Pages with a corresponding backend on Heroku, I figured it was time to take it to a whole new level: integrating Amazon Web Service’s S3 into the mix. *Gasp*. I am happy to say that after 7 days of zero progress I can finally admit that things are working as they should. I have split this guide into 3 sections that correspond to the material covered in each: Figaro, Heroku, S3 + Rails.

After hours of pouring over StackOverflow and Heroku documentation plus a spattering of compromised AWS accounts, I’ve learned a few things about connecting Heroku to AWS’s S3 the hardest of ways and I hope this guide will save you the days of time it took me to finally get my project working.

I Figaro, you Figaro, we Figaro

Tidbit #1: Simply requiring the ‘Figaro’ gem in your Gem file and bundle install-ing is not enough to stop Git from pushing your application.yml file to GitHub.

March 6th started just like any other day. The birds chirping, the sun stretching its curious rays through the cracks in my blinds, caressing my face with its warmth. Then I checked my email: “Your AWS Account is Compromised.” 😯.

After a few lengthy conversations with AWS support staff, a couple password resets and two or three more “Your AWS Account is Compromised” emails, I finally uncovered a clue: my GitHub repo still had an application.yml file! Which meant that every time I pushed to GitHub with the recently reset AWS keys I was exposing my credentials to the public… again! None of this made any sense to me at the time though because my .gitignore file did include a reference to my /config/application.yml file, which should tell Git to not push that particular file to the remote GitHub repo. However, because I originally pushed my project without the reference to the /config/application.yml file the application.yml file remained and updated every time I pushed. So I manually removed the applcation.yml file from the remote Github repo and my local application.yml file and reran the following command:

$ bundle exec figaro install

which correctly generated an application.yml file.

Tidbit #2: In order to avoid this unnecessary headache in your Junior Development life listen carefully: Figaro is a Ruby gem that generates an application.yml file in your Ruby project and places it in the config folder and then adds a reference to that file in the .gitignore file. That way when you push your code to your GitHub repository your environment variables (like an AWS access key) are not exposed to the public. To reiterate, after adding the ‘figaro’ gem to your Gemfile make sure you run this command:

$ bundle exec figaro install

This will

1. create an application.yml file and wire it to your .gitignore file so your secrets will be safe and more importantly
2. stop your AWS account from getting compromised.

Now the way I think this AWS hack works, and this is purely speculation by the way, is there’s someone out there that wrote a script that scrapes every commit on GitHub and checks for the words ‘AWS_SECRET_ACCESS_KEY’ and ‘AWS_ACCESS_KEY_ID’ or something to that effect and if it finds them it uses those credentials to hack into that AWS account and then sets up a bunch of EC2 instances to make massive, sketchy data transfers across the globe. When I finally got word from Amazon that my account was compromised this hacker who shall from this moment on be referred to as ‘Dick’ had launched over 100 EC2 instances from Tokyo to Seattle and wracked up a bill of $818 under my account. Luckily I had reported it in time and AWS waved the charge but imagine if I forgot about my AWS account for weeks?

Heroku

Wonder if I told you that your app can take on many different forms? We are all used to launching a Rails project in its “development form”, or what is reffered to in the coding world as ‘locally hosted’ with the following command:

$ rails s

This will run your project in the development environment, plopping your app on a locally hosted server, most likely localhost:3000. Since the server is hosted locally it still has access to that config/application.yml file, but when you deploy your app to a server not on your computer, let’s say Heroku for instance, because we instructed that application.yml file to be .gitignore-d and we used git to push that local repo to the remote Heroku repository, your app will lose access to it when you run

$ git push heroku master

Tidbit #3: because your app no longer has an application.yml file in its directory but still requires those ENV[‘variables’] in parts of the code base, you need a way to define those same environment variables so Heroku can access them.

On Heroku, navigate to the correct project’s Settings tab and click the ‘Reveal Config Vars’ button.

Say hello to your old pals Ernie and Val, a.k.a. your environment variables. Better yet, think of this panel as your application.yml file. You can now safely set your environment variables for this production version of your app and it will now know what those references to ENV[‘whatever’] are in your code. Mine looked like this:

With Heroku now understanding what those variables values are it’s time to set up the Rails app to talk to S3, but before we do let me show you an awesome Heroku feature. Tidbit #4: if you navigate to the More button at the top right of your Heroku project dashboard you can enter into a console session like you would in your Rails project (rails c) by clicking Run Console. To view logs coming into your backend in realtime click View Logs. These can (sometimes) provide you helpful error messages so that you can more effectively debug.

S3 + Rails

You must now tell paperclip how to behave in the context of a production environment. You will do this in your Rails app in the config/production.rb file by adding the following bolded lines of code:

#app/config/production.rb
Rails.application.configure do
  # Settings specified here will take precedence over those in config/application.rb.
  # AWS
  config.paperclip_defaults = {
    storage: :s3,
    s3_host_name: ENV["S3_HOST_NAME"],
    path: ENV["S3_PATH"],
    s3_credentials: {
      bucket: ENV["S3_BUCKET_NAME"],
      access_key_id: ENV["AWS_ACCESS_KEY_ID"],
      secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
      s3_region: ENV["S3_REGION"]
    }
  }

With that done it’s time for another tidbit you wish your coder friends would have told you before you started trying to get your app live on the web (Tidbit #5). In order to get your Rails app to work with AWS, you must first add the gem aws-sdk to your Gemfile. Now it’s time to bundle install the aws-sdk gem, which builds a Ruby library that simplifies the process of making various API calls to AWS’s S3 service. Please note: the various variable names S3 requires to actually work with a Rails app seem to change based on what type of a) aws-sdk gem you are using, which is a fact that is not documented at all and one of the major causes of me blasting imaginary holes through my computer for hours, but also b) what version of Ruby you have installed. I used gem ‘aws-sdk’, ‘~> 2.3’ and Ruby version ruby 2.3.1p112 and after the 100th time of changing the way I named my environment variables this was my winning combination:

S3_HOST_NAME
S3_BUCKET_NAME
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION
S3_PATH

Since I wanted to associate a profile image with a player in my app, I had to configure my Player model to use the aws-sdk S3 baked-in API calls :

has_attached_file :image,
 styles: { small: “64x64”, med: “100x100”, large: “200x200” },
 :s3_protocol => ‘https’,
 :s3_host_name => ENV[‘S3_HOST_NAME’],
 :path => ENV[‘S3_PATH’],
 :storage => ‘s3’,
 :s3_region => ENV[‘AWS_REGION’]

One more git add, git commit -m "look ma, i learned stuff" and git push heroku master and I was officially up and running on Heroku with my image storage now correctly connected to my AWS S3 bucket. In other words, pure bliss.

GIVE ME THE ANSWERS NOW!

Assuming you are running Ruby version ruby 2.3.1p112, have successfully created your Rails Heroku app as well as an S3 bucket on AWS (docs on deploying to Heroku and setting up S3), and your paperclip gem is working correctly and able to save files to your local repo:

First let’s make sure your project is working locally or in “Development mode” with S3:

1. Add the gem ‘figaro' to your Gemfile

2. In the command line run bundle exec figaro install

3. Add the gem 'aws-sdk' ~> '2.3' to your Gemfile

4. In the command line run bundle install

5. Add the following environment variables and their corresponding values that you get from your AWS credentials to your figaro-generated application.yml:

S3_HOST_NAME: ‘s3.whateverregionyouareusing.amazonaws.com’
S3_BUCKET_NAME: ‘yourbucketname’
AWS_ACCESS_KEY_ID: YOURAWSACCESSKEYALLCAPSANDNOTINQUOTES
AWS_SECRET_ACCESS_KEY: youraccesskey
AWS_REGION: ‘yourregion, us-east-45 perhaps’
S3_PATH: ‘/profileimages/:filename'
# the :filename denotes whatever the image uploaded is called will be dynamically assigned a path based on the name of that file

6. Navigate to whatever Ruby model has the paperclip image associated with it(mine was the Player model) and add the following lines:

has_attached_file :image,
 styles: { small: “64x64”, med: “100x100”, large: “200x200” },
 :s3_protocol => ‘https’,
 :s3_host_name => ENV[‘S3_HOST_NAME’],
 :path => ENV[‘S3_PATH’],
 :storage => ‘s3’,
 :s3_region => ENV[‘AWS_REGION’]

 validates_attachment_content_type :image, :content_type => [“image/jpg”, “image/jpeg”, “image/png”, “image/gif”]
 validates :name, :password, presence: true

And that’s it. Now to get things working in production only requires two more steps.

7. Add the following lines to your config/environments/production.rb file:

config.paperclip_defaults = {
    storage: :s3,
    s3_host_name: ENV["S3_HOST_NAME"],
    path: ENV["S3_PATH"],
    s3_credentials: {
      bucket: ENV["S3_BUCKET_NAME"],
      access_key_id: ENV["AWS_ACCESS_KEY_ID"],
      secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
      s3_region: ENV["S3_REGION"]
    }
  }

8. Navigate to your Heroku app's Settings Tab and click the Reveal Config Vars button:

Enter the environment variables you previously placed in your application.yml file here. These are what mine looked like:

One more git add, git commit -m "look ma, i'm o.k." and git push heroku master and you should be officially up and running on Heroku with image storage handled by AWS’s S3.

You’ve made it this far? Much appreciated for reading! I am always looking for feedback on my blogs and projects. Don’t be afraid to leave a comment or send me a message. Also on the journey of transitioning to a career in Software Development? I’d love to connect on Linkedin.

Enjoy my writing? Check out my other articles👇 down below.

So you want to host your Single Page React App on GitHub Pages?

ActiveRecord, Dirty Attributes and Michael Jackson

Recreating Twitter’s `Like` effect with with mo.js