Enable faster uploads and downloads with your S3 Bucket
Transfer acceleration is a way to speed up both uploads and downloads with your S3 bucket. It works by utilizing Amazon’s global network of edge locations, the same infrastructure used by other AWS services such as Cloudfront and Route53. An edge location is a data center located close to your customers, which provides low latency access to cached content.
The user performing the upload will be uploading to his nearest edge location instead of directly to the S3 bucket. AWS has only 24 regions but has 216 edge locations.
Imagine we have an S3 bucket located in Virginia, but our customers are on the other side of the world in Australia. When uploading a file, the file will need to travel from Australia all the way to Virginia. With transfer acceleration enabled, the data will only need to travel to a local edge location in Australia. From there, the file will still need to make it’s way back to Virginia, but AWS will use an optimized network path to speed up the transfer between the edge location and the S3 bucket.
Performance
As a starting point, Amazon provides a tool you can use to determine whether you will see an increase in speed with transfer acceleration enabled.
The tool works by uploading a test object directly to a bucket in 15 different regions with and without transfer acceleration enabled.
By default, the tool will first test the US-Virginia region. To have the tool focus on a specific region, you can copy/paste the following URL replacing the region with the specific one you are interested in testing. https://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html?region=region
The limitation of this tool is that it runs based on your location while you may want to mimic your customers, which may be on the other side of the world.
The general rule is the further your customers are from your S3 bucket region, the more beneficial transfer acceleration will be to them. Another good rule is that bigger files will generally show more acceleration.
You will also see an increased benefit if your internet connection is not stable, and there is a higher amount of packet loss.
If your files are less than 1GB in size or your total data transfer is generally less than 1 GB, you should use Cloudfront instead. Cloudfront is usually known as a CDN, but it can also handle file uploads as well.
Here is how AWS describes who will see the most benefit:
“The amount of acceleration primarily depends on your available bandwidth, the distance between the source and destination, and packet loss rates on the network path. Generally, you will see more acceleration when the source is farther from the destination, when there is more available bandwidth, and/or when the object size is bigger.”
Remember that the uploaded object will still need to travel from the edge location to the bucket, but this will happen over an optimized network path. The API calls are synchronous, which means that you will not receive a success response until the object has reached its final destination, the bucket.
Security
Using transfer acceleration has the same security guarantees as S3. Your data is never stored at the edge location.
Enabling Transfer Acceleration
You can enable transfer acceleration through the console, API, SDK, or CLI.
Only the bucket owner or a user with the permission PutAccelerateConfiguration can change the state of transfer acceleration for a given bucket.
After enabling transfer acceleration, it might take up to 20 minutes until you notice any speed improvements.
Using Transfer Acceleration
Transfer Acceleration works for both object uploads and downloads (PUT/GET requests). You will need to modify your bucket endpoint, as shown below:
old: bucketname.s3.region.amazonaws.com
new: bucketname.s3-accelerate.amazonaws.com
The transfer endpoint must have the bucket name as the host. You cannot do something like s3-accelerate.amazonaws.com/bucketname/filename.
You can continue to use your regular S3 bucket URL for cases where you do not want to incur the cost of using transfer acceleration.
You can point all your PUT and GET requests to the new URL. On each request, AWS will determine whether to use transfer acceleration or not based on the performance benefit you may realize. Using the endpoint doesn’t guarantee the file will be routed through an edge location.
The AWS console does not make use of transfer acceleration on uploads.
Pricing
Any object uploaded through transfer acceleration will be charged based on which edge location is used as the intermediary between the user and the s3 bucket. Edge locations in the US, Europe, and Japan are 50% cheaper than any other location.
AWS will intelligently decide whether to use Transfer Acceleration for a given transfer, thereby saving you money. The response of the Upload API call will tell you which edge location was used to accelerate the transfer.
For a list of Edge locations and their codes, refer to here https://www.feitsui.com/en/article/3.
When using the CLI or the SDK, it’s easy to accidentally configure it always to use the accelerated endpoint when you may only want to use it on specific transfers. One way to handle this is with AWS profiles; you can have one for the acceleration endpoint and another one for the non-accelerated version.
Disabling Transfer Acceleration
When disabling transfer acceleration, remember that your specific accelerated endpoint will no longer work.
Limitations
If your bucket name has a period in it, you will not be able to enable transfer acceleration. Unfortunately, AWS allows you to create buckets with a period and doesn’t warn against it.
File storage gateway does not support S3 transfer acceleration.
The Transfer acceleration S3 endpoint does not support the following operations:
- list buckets
- create bucket
- delete bucket
- copy object to another region
Summary
Transfer acceleration is mostly useful when moving big files or large amounts of data into or out of S3 and where geographic locations of your end-users vary, or you are not situated close to your S3 bucket. If you want to speed up S3 uploads or downloads on smaller files, try Cloudfront instead.
Make sure only to use the S3 accelerate endpoint selectively to avoid extra costs on data transfers you may not care to speed up.
