Blazing CDN with Amazon S3 + Cloudflare
In dealing with high-volume websites you will need to eventually separate out your assets and have them served over a CDN. There are a whole list of specifications you will need to take into account for this CDN:
- How many edge locations does it have for lowering the latency to your users?
- Can I set this CDN up over a cookie-less domain?
- How do I store assets (on my own source server or on their server)?
- Where will bottlenecks occur when moving / delivering assets?
- Of course, costs vs budget.
There are some great CDNs out there. MaxCDN, CloudFront, Akamai, and others. When I worked with some larger clients we would go with Akamai for their support and delivery promises, but now I am really leaning towards CloudFlare for several reasons…
- They have tons of edge locations in the rest of the world (they’re not just focusing on a US / Euro market. In the Middle East / Central Asia they have some 10 edge locations! You can read more about their CDN on their site.
- Bandwidth is free. I setup a site the other day that was paying some $500-$1000 for their content delivery (that didn’t even have edgepoints in their region). Their latency dramatically decreased, and their costs did too — now they pay for a simple pro version of CloudFlare and no bandwidth costs. We also quickly saved ~5TB of load on the app server for delivering those assets over a CDN.
Setting Up Your CDN
Step 1: Get your assets on S3! In order to use the static content delivery you will need to have your S3 bucket named to the domain you want to deliver it from (if you have a website called mycdn.net you will need to setup a bucket named mycdn.net).
Step 2: You will need to then enable static content delivery or add a bucket policy like the one below (be sure to change the bucket name):
Step 3: Setup a CloudFlare account and add your domain name to the system (point your nameservers to CloudFlare). I would recommend at least the Pro account because their traffic favors their paid accounts (as they should).
Step 4: Optimize the account for a CDN. You can certainly setup your CDN as a subdomain, but the issue is that your main domain will send its cookies with each piece of content you deliver increasing your load time slightly. When delivering many assets, it adds up. In order to minimize this, you will need to purchase another domain name and reserve it for your CDN. This will keep the cookie madness away from your CDN’d content. Here are some optimizations / settings I recommend:
- Turn on Flexible SSL for delivery. You can use other forms of it, but you will need to setup CloudFlare on Amazon in order to use Full SSL. Flexible SSL will make sure your CDN can deliver over HTTPS or HTTP. It will take a moment (~10 minutes) to activate the SSL certificate, but until then it will still serve over HTTPS.
- Security Level: Essentially Off
- Speed Tab: Enable Polish (maybe Basic+JPEG) which will compress and remove meta data from images and also provide progressive downloads to your end users.
- Caching Tab: Turn Caching Level to “Ignore Query String”. Set Browser Cache to 1 Year. This will keep your assets cached for as long as possible. Make sure Always Online is on.
- Network Tab: Make sure HTTP/2 + SPDY is checked for maximum delivery speeds and batch HTTP requests.
NOTE: This is “technically” not a cookie-less domain because cloudflare stores a uid so it can track usage and implement its security measures. But it is not a bad exchange because it will only ever be a few bytes.
Step 5: I removed a ton of DNS entries that come with your domain scan. Remove everything and then add a CNAME entry for your S3 bucket. Example: “mycdn.net.s3-website.eu-central-1.amazonaws.com”. Make sure the cloud is orange.
You’re now good to go! Your CDN SSL should be active after a moment and you’re set to serve your assets over a cookie-less CDN’d domain. You should see a dramatic increase in your latency and delivery speeds.
UPDATE: June 30, 2017
I wanted to add that since posting this site I had encountered a major hit in traffic. It was no longer caching any objects over a certain size and anything of a certain type. I later found out that CloudFlare has this CDN delivery option for non-assets (had to do some digging) even though when I first turned it on it was caching both assets and non-assets. So at some point they decided it was too expensive to cache our assets (I am not surprised at all as they were saving us a few thousand bucks a month. I was more surprised that they were allowing this for so long — they’re really nice). So please know that this is completely expected when dealing with high traffic — bandwidth costs ;) But the CDN should work for a short time until you become millionaires. But don’t be surprised if you start seeing millions of requests to your origin servers.