The Seven(ty-ish) Endpoints of Highly Effective S3 Hosting Buckets

Published in

victoryteam

13 min readMay 1, 2019

By Boyd Hemphill

The serverless movement was launched in November of 2014 with the advent of AWS Lambda. Oh … wait, that is not true. It actually happened in 2008 with the ability to host static websites in an AWS S3 bucket.

Yep, all you serverless hipsters out there thinking you’re the new, cool, hotness? Serverless was born in your grandpa’s cloud.

Rise of the SPA

Today many companies use the S3 Website hosting capability to serve Javascript web applications known as Single Page Apps (SPA). Why would you do this?

Enterprise Capabilities: Amazon offers

99.99% — “4 nines” — availability (uptime)
Built in DDoS mitigation — availability
99.999999999% — “11 nines” — durability.

People: Smaller IT Operations footprint

Cost: Hosting an SPA in AWS S3 is less than $0.10 per month.

For startups and small companies, the enterprise capabilities alone should drive consideration of adopting Javascript as the web delivery technology of choice.

The ability to compile an application into a single artifact that is easily deployed in single location that is then distributed world wide also means that teams have far fewer tools and complexities to cope with in their DevOps tool chain.

Hosting your web application in S3 is a great way to focus on features and not on technology. But, it comes with some sharp edges. This article describes some of the problems setting up a secure bucket for hosting your SPA.

Technical Summary

A combination of factors makes nearly every bit of documentation about setting up an S3 Bucket for website or SPA hosting wrong. This includes both AWS docs and the myriad blog posts on how to do it. Usually the errors are by way of example. This article’s purpose is to point out the nature of the errors.

The problem is rooted in these factors:

The canonical hosting endpoint for an S3 Bucket is different in its form from region to region.
The AWS Console (specifically CloudFront) suggests the wrong endpoint when setting up a distribution’s origin as a S3 Bucket.
Articles and AWS docs show examples of a single region and the canonical format for the endpoint of that region.
If the reader picks out that they are using the wrong hosting endpoint, then they are still likely to infer the wrong endpoint format.
Problems only comes up in the case that there is a website redirect created. Redirects will appear not to work.

These inconsistencies can cost a Cloud Engineer hours of time. There is little in the way of error logging to track down the source of the problem. Rather than accepting the default origin choice from CloudFront when setting up for secure hosting, be absolutely certain to use the canonical hosting endpoint for your region found here!

The rest of this article details:

How to set up a simple S3 hosting bucket
Obtain a TLS certificate from Amazon Certificate Manager
Set up a CloudFront Distribution with the certificate and bucket for secure hosting
Test the set up.
Set up a redirect
See it not work
Adjust the Endpoint
Show that the redirect works.

Simple Hosting

Let’s create an S3 bucket for hosting our static site or SPA. For our purpose here we will host a couple of HTML pages.

Here you can see we name the bucket endpoints.victorycto.com.

Pro Tip: The bucket must be named exactly the same as the (sub)domain it will be serving.

While it would be great to simply say, “Now pick the region of your choice,” and leave it at that, it’s not so simple. See “Except the Endpoint Madness” below for the details. We will get our bucket set up and working here.

Gotcha: If you intend to follow along, you will need to name your bucket “endpoints.YOUR_DOMAIN.YOUR_TLD”. This is because there is a step involving DNS.

In a real world scenario, collecting metrics about the bucket hosting your application is important. It is most important when something goes wrong. Take the time to do the following:

Server Access Logging

Log requests for access to your bucket. In the example above we have a second bucket “endpoints-logging” where requests will be sent. Because we want to use the bucket to collect other information we will prefix this data with the word “requests”. The prefix can be thought of as a folder or directory within the bucket for the data.

Object-level Logging

Data here is about access to the bucket itself. For example when you do a deployment to the bucket via the API this activity will be recorded. If you change permissions on the bucket that activity will be recorded.

CloudWatch Request Metrics

Gotcha: You must scroll down in the light box to see this setting.

This box will give you behavior data around requests to the bucket. Things like the number of 200, 300, 400 and 500 responses. Tracking trends over time in these can help identify problems and opportunities. Details on the exact metrics are described in the AWS docs.

Permissions

S3 Buckets are defaulted to be private and tightly controlled. However to host a website the public clearly needs to be able to read the files in the bucket.

Create the S3 Bucket

After reviewing the information, click “Create the Bucket”

Configuring the S3 Bucket for Hosting

Doubleclick your new bucket and choose the “Properties” tab.

Click the “Static website hosting” tile and set it based on the example below:

Pro Tip: Be sure to copy the endpoint some place for later use. Note that your endpoint may be slightly different than the above. See Endpoint Madness below for details.

While we are using a static HTML page, you could use the index.js page from a React project just as easily.

Click “Save” and then back to the overview tab.

Upload our Website

Here is the code for our pages:

index.html

<html><body><h1>The Many Endpoints of S3 Hosting</h1><a href=”https://medium.com/victoryteam/">The Victory CTO Blog</a>You will reach this page first through the endpoint:http://endpoints.victorycto.com.s3-website-us-west-1.amazonaws.comUnless you are in a region where the format of the endpoint is<a href=”https://amzn.to/2EqnvLH">slightlydifferent</a>.</body></html>

404.html

<html><body><h1>Our bucket overfloweth</h1>Just not with what you are looking for.Yep, this is a 404.</body></html>

Let’s upload these pages to the bucket.

Set the “Manage public permissions” to “Grant public read access to this object(s)” so that a visitor to the site can read the file. Click “Next”

For a website, especially a busy one, we will keep the “Standard” setting. Note this keeps copies of the data in 3 or more availability zones. In our example this means our site is likely to be up unless the entire Oregon (us-west-2) region goes offline.

Pro Tip: While not in the scope of this article, you could even go so far as to have the bucket replicate across regions to achieve more durability and availability. Remember the time and money should correspond to a real business need beyond the 99.99% availability achieved here.

The rest of the “Set Permissions” settings (cannot be seen in the screen shot above) can be left at their defaults.

Pro Tip: If your company has not created a tagging strategy for its AWS infrastructure, now is a good time to stop and consider doing so. Tagging allows for outstanding visibility in billing to control costs.

Click “Next”, then click “Upload.”

The S3 Bucket is Alive

Now visit the endpoint. For us, that means: http://endpoints.victorycto.com.s3-website-us-west-1.amazonaws.com

And to show a 404 will happen as expected visit:

http://endpoints.victorycto.com.s3-website-us-west-1.amazonaws.com/bogus.html

Except the Endpoint Madness

Here is the problem. If you were in the newer US East Ohio (us-east-2) region the endpoint would look subtly different: http://your.url.here.s3-website.us-east-2.amazonaws.com

Gotcha: Note the dot ( “.” ) after “s3-website” rather than the dash ( “-” ). The differences between regions is truly confusing unless you happen to know the chronological order in which AWS regions came on line.

Worse, if you are clicking around the AWS Console you might never notice. As you go to translate this hosting effort from R&D to Cloud Formation or Terraform you can lose hours wondering why you cannot get your bucket to work.

The canonical list of hosting endpoints can be found in buried in the AWS documentation.

DNS for the S3 Bucket

You might be asking yourself, “But shouldn’t we set up a CNAME record in DNS?” We could, but it’s a waste of time. Without HTTPS, browsers will complain about the site being insecure. We need to set up a certificate for our hosting bucket. Except ….

Secure Hosting in an S3 Bucket

Look at the browser screen shot above, and notice that Safari is griping about the site not being secure. In 2019, it is expected that traffic will be on the https protocol. Since hosting in S3 has been happening since 2008, surely this can be done easily.

Nope.

To secure our site via TLS we will need:

A Certificate
A CloudFront distribution

That’s right, you need a CDN to be secure!

TLS Certs — The Easy Button

The Amazon Certificate Manager (ACM) service is a truly wonderful thing. These directions will assume you have DNS in Route53, but are easily generalized if you do not.

The following instructions are quick and dirty without the myriad screen shots for brevity.

It is worth repeating: In order to secure a bucket hosted site you must create a Cloudfront distribution. The certificate we create here is only the first step.

Gotcha: The certificate must be created in the us-east-1 N. Virginia region for it to be found by CloudFront. This is a requirement and it is true in all regions worldwide.

Gotcha: We are issuing a wildcard certificate for each environment with the same domain *.victorycto.com. Be sure to note the ARN or Identifier along the way so you are clearly dealing with the same certificate during the setup process.

Navigate to the ACM page, and be sure to go to the top nav bar and change your region to us-east-1 (N Virginia).

Request a Certificate

Choose “Request a public certificate”
Click “Request a certificate”

2. Add domain names

Domain name: “*.victorycto.com”
Click “Next”

3. Select validation method

Choose “DNS validation”
Click “Review”

4. Review

Ensure the Domain name is correct and that the validation method is DNS
Click “Confirm and Request”

5. Next to the *.victorycto.com click the triangle to expand the request.

Click the “Create record in Route 53”
Review the proposed DNS entry and click “Create”
Click “Continue” — You will be returned to the cert page

6. Next to the *.victorycto.com click the triangle to expand the request.

Check the Amazon Resource Name (ARN) to be sure you are looking at the right certificate

It can take up to 30 minutes for the cert to validate.

Gotcha: You must manually refresh the page to see the Success status.

Except the Need for a Content Delivery Network

Cloudfront is a very old service and the UI is like nothing else in the AWS ecosystem. Again, for the sake of brevity, written instructions are provided.

Navigate to the CloudFront page.

Click the “Create Distribution” button
Click the “Get Started” button under “Web” (Do not use RTMP)
Create Distribution

Origin Settings

Origin Domain Name: endpoints.victorycto.com.s3.amazonaws.com
Gotcha: Note that this endpoint is different from the canonical one. This will come back to haunt us later! Pay careful attention.

Default Cache Behavior Settings

Viewer Protocol Policy: Redirect HTTP to HTTPS
Object Caching: Customize
Maximum TTL: 300
Default TTL: 300
Compress Objects Automatically: Yes

Distribution Settings

Price Class: Use All Edge Locations
Alternate Domain Names (CNAMEs): endpoints.victorycto.com
SSL Certificate

Custom SSL Certificate: Set

Choose the cert that matches the . You will have to look this up by its ARN.
Gotcha: If you don’t see the certificate you just created likely you forgot to do so in us-east-1 (Virginia)

Security Policy

TLSv1.1_2016 (recommended)

Default Root Object: index.html

Logging: On

Bucket for Logs: endpoints-logging.s3.amazonaws.com

Log Prefix: distribution

Comment: endpoints.victorycto.com

Pro Tip: This shows up in the summary page as a human friendly name for the distribution.

Click “Create Distribution”

4. Click on the distribution and choose the “Error Pages” tab

Click “Create Custom Error Response”

Custom Error Response Settings

HTTP Error Code: 404: Not Found
Error Caching Minimum TTL: 0
Customize Error Response: yes
Response Page Path: /404.html
Gotcha: note the leading /
HTTP Response Code: 404: OK
Click “Create”

5. Make a note of the value in the “Domain Name” column

It will look something like d3vet0wwamg7dh.cloudfront.net

It will take up to 30 minutes for the distribution Status (look for the column on the Distributions summary page), to go from “In Progress” to “Deployed”

While we are waiting, let’s set up DNS.

Setting up DNS for an S3 Bucket

Go to Route 53 in the AWS Console and choose the zone for your domain name.

Click the blue “Create Record Set” button

Name: endpoints
Type: CNAME
Alias: No
Value: d3vet0wwamg7dh.cloudfront.net (The domain name of the Cloudfront distribution)

2. Click “Create”

Be sure that DNS has propagated and check that you can get a response from the distribution via HTTPS and HTTP:

HTTP is redirected to HTTPS

victorycto:~ 14:17:06 > curl -I http://endpoints.victorycto.comHTTP/1.1 301 Moved PermanentlyServer: CloudFront

HTTPS responds correctly

victorycto:~ 14:15:45 > curl -I https://endpoints.victorycto.comHTTP/2 200content-type: text/html

But most of all, Safari (and other browsers) are no longer complaining about the lack of security:

Redirects when Hosting in an S3 Bucket

Websites mean redirects to manage. This can be a set of mod_rewrite rules, or it can be a curated list left over from the last site migration. So clearly this must be possible when hosting a site in S3 as well right?

Setting up redirects for S3 Bucket hosting is easy. The real difficulty comes in using the right endpoint for the bucket. First let’s set up the situation and then we can get to the huge gotcha.

We have an original file called this-is-an-ugly-url.html that is empty.

We also have the file beautiful-url.html who code is simply:

<body><html>I am beautiful (but my URL could be less so).</html></body>

Upload these files to your bucket as you did above with index.html and 404.html.

Gotcha: Be sure to allow them to be read publicly!

You can check that the file beautiful-url.html can be served:

victorycto:~ 14:30:00 > curl -Ihttps://endpoints.victorycto.com/beautiful-url.htmlHTTP/2 200content-type: text/html

Double click on the bucket name (endpoints.victorycto.com)

Click on the new blank object this-is-an-ugly-url.html
Click the “Properties” tab
Click the “Metadata” tile

Set the “Content-Type” head to “text/plain”

This necessary so that the browser will not immediately attempt to download the file it thinks is binary.
Set the “Web-Redirect-Location” to “https://endpoints.victorycto.com/beatiful-url.html"

4. Click “Save”

5. Test your redirect

victorycto:~ 14:40:00 >curl -Ihttps://endpoints.victorycto.com/this-is-an-ugly-url.htmlHTTP/2 200content-type: text/plain

But that is wrong! You could lose hours to tracking this down, but it boils down to the AWS Console giving you a red herring …

Except the CDN gives you the Wrong Endpoint

Documentation at AWS and around the web will state that accessing the bucket URL from endpoints.victorycto.com.s3.amazonaws.com will function properly.

Gotcha: This is the case and in fact, it is the endpoint CloudFront will automatically suggest!

In fact, to get all web hosting features, you MUST include the region in which the bucket is located: endpoints.victorycto.com.s3-website-us-west-1.amazonaws.com

Gotcha: Note the “-us-west-1” and also be wary that yours could be a “.us-east-2” )

Gotcha: Note that if you are in a different region (e.g. us-east-2) you could would use a dot in place of the leading dash! For example: endpoints.victorycto.com.s3-website.us-east-2.amazonaws.com

Few posts out there on the web will show your the difference and it is easy to assume the format and drop your own region in there based on whatever format they are showing you.

Setting the Cloudfront Endpoint for S3 Bucket Hosting Redirects

On the CloudFront Distributions summary page, click on the distribution you set up. Click the “Origins and Origins Groups” tab.

Set the tickbox by the incorrect endpoint and click the “Edit” button.

Replace the “Origin Domain Name” with the canonical endpoint: endpoints.victorycto.com.s3-website-us-west-1.amazonaws.com

Gotcha: The AWS Console will helpfully try to lead you astray here. Ignore the suggest and paste the canonical endpoint over their suggestion.

Click the “Yes, Edit” button. You will not have to wait for the distribution to converge. This can take up to 30 minutes. Once converged you can see the redirect happens as expected:

victorycto:~ 14:57:32 > curl -Ihttps://endpoints.victorycto.com/this-is-an-ugly-url.htmlHTTP/2 301content-length: 0location: https://endpoints.victorycto.com/beautiful-url.html

In Conclusion

There are a number of moving parts to the common problem of hosting a website or single page app (SPA) in a S3 Bucket.

The interaction between S3 and CloudFront coupled with the many, various, hosting endpoints that will work in some cases, leads to confusion.

When a cloud engineer wants to set up redirects, the difference in endpoints can lead to a nearly impossible problem to track down. Rather than accepting the default origin choice from CloudFront when setting up for secure hosting, be absolutely certain to use the canonical hosting endpoint for your region found here!

Boyd E. Hemphill is the Chief Technology Officer at Victory

Victory is a hypernetic solutions consultancy. What does that mean? We use cross-functional expertise to solve the big, complex problems businesses and organizations face. We like to bring in smart people like Boyd to help us do just that.