AWS Ephemera — Debugging Doubled S3 Hostnames
It’s Engineering Project week at Makers and I have had a mightily frustrating day…
I’ve been trying to figure out how to save images uploaded using carrierwave into a remote database hosted by Amazon Web Services.
That comes directly after another very seriously frustrating day where I managed to crash our production site, hosted on Heroku, because (who knew?) it didn’t store image files.
Apparently it is ephemeral.
I was apoplectic.
The team weren’t best pleased either.
I signed up for AWS yesterday, handed over my credit card details and then scoured the internet for helpful tutorials, while waiting for my S3 access to be authorised.
The helpful tutorials sent me down one rabbit hole after another. AWS informed me I needed gem x, carrierwave proposed gem y and z, the helpful bloggers of the interweb went for a combination approach of x,y and z but with a troubling inclusion of deprecated gems like figaro.
I’m getting used to wading my way through deep, muddy pools of uncertainty and copy-pasting my way towards a working app but I was actually flummoxed.
It didn’t help that we haven’t fathomed out how to test our production environment anywhere other than in our production environment — which is the public facing app on Heroku. As I’d bolloxed that up the day before, and we don’t seem to have found our public yet, it didn’t seem so bad to run re-build after re-build as I tinkered with the code. But it was a slow, sloooow process and error testing on a production platform must be terribly bad form!
We did eventually get it to work and I learned some some pretty useful debugging tips along the way so I thought I better jot down some steps just in case I ever have to become an AWS code-sleuther in the future.
Initial steps — note these didn’t work for me. They may for you but more importantly they worked enough to give me some very valuable stack traces.
The most detailed post that includes great visual guides to setting up S3 on AWS is Switching CarrierWave to use S3 with Heroku and localhost — Firehose Project. This uses the deprecated Figaro though and also flips all environments to AWS which broke my local development environment as well — see lightbulb moment below though.
I ended up using this guide by JiaHung Lin as it allowed me to split the handling of files between file storage for the local development environment and AWS for production (but again, note my retrospective lightbulb moment documented below).
If you’re planning to use this JiaHung’s guide I would suggest getting carrierwave setup to work locally before muddying the waters with S3. There will be more detailed guides for that stage. If you’ve done that you can just skip to the section called *Upload to s3*
Heroku crashes the moment you try to upload an image and the Heroku logs reveal two issues or CLUES.
- Fog is telling me that I’m not using the right region and so it’s going to redirect. It suggests I use US-West-2
- The redirect fails but looking at the hostname it clearly looks odd as my S3 bucket name is duplicated.
Google tells me that I can amend the region to remove the Fog error message.
Interweb resources for AWS repeated bucket name issues:
- Repeated bucket name in the URL on AWS and issue with :path_style · Issue #228
- S3 Bucket name gets duplicated in case of redirect from AWS
So I change the carrierwave.rb file to include reference to the region rather than relying on the default:
This still didn’t fix my problem. Clue 1 has disappeared but my hostname still has the double bucket issue.
Google stops being helpful so I nip over AWS which has the world’s worst user interface but actually a very handy developer guide. It revealed something about virtual paths which had a format very similar to my dodgy hostname. It led me to think that perhaps the duplication was happening as a result of this redirection so I needed to find a way to stop that — a get it right first time approach.
The logs helped again. The redirected, double bucket path was actually heading off to a totally different region us-east-2
So we finally got our production site working when carrierwave.rb looks like this:
Lightbulb Moment
Having struggled so much with AWS over the last couple of days, I felt compelled to document the route to success but rather annoyingly I had forgotten to screenshot all the error logs.
I’ve now spent a few more hours trying to recreate the same problems in another project and in doing so I have had a bit of a lightbulb moment.
*Now that we’ve fixed the problem, I’m convinced that it probably is a good idea to have development and production use the same storage solution. So what I thought was a flaw in the original guide may well actually answer my question of how the heck do you test the production without breaking production?*
So in my new project I have coded the solution like this:
This has the benefit of using AWS for file upload when run from localhost (development) and also Heroku (production).
Originally published at whatapalaver.co.uk on September 5, 2018.
