S3 Exercise 2.4: Adding Objects to a S3 Bucket with Cross Region Replication
(This post is part of the AWS for the Bootcamp Grad series. The series consists of exercises to build familiarity with various aspects of the AWS ecosystem. Again, all of these posts are “exercises” for introductory exposure to AWS — they are NOT represented as best practices.)
Background
In this exercise, you will create source and destination S3 buckets in different regions, and use S3’s cross region replication capabilities to automatically replicate the objects in the source bucket to the destination bucket. You will create a static website hosted on S3 as an interface to upload files to the source bucket and see them replicated to the destination bucket. To do that, you will write a browser-side script that uses the AWS SDK, and authentication through Amazon Cognito.
To see more information about Cross Region Replication, and common use cases, see this documentation.
Specifically, you will do the following:
- Create an S3 bucket to store files.
- Create another S3 bucket in a different region to store replicated versions of the objects in the first bucket.
- Configure the S3 bucket for cross region replication.
- Create an Amazon Cognito identity pool (the browser script will need credentials to access your S3 resources).
- Configure the Amazon Cognito’s IAM role for unauthenticated users to obtain access to your S3 bucket.
- Create another S3 bucket and configure it to host a static website.
- Create a browser side script to list all objects in both buckets.
- Create a browser side script to upload a new file to the first bucket and see the file replicated into the second bucket.
Create the Source S3 bucket
Follow the Create an S3 Bucket section of Exercise 2.1. For clarity, when naming the bucket, you can use the word source in its name.
Create the Destination S3 bucket
Follow the Create an S3 Bucket section of Exercise 2.1 — however, select a region that is different from the region for the first bucket. I used US East (N. Virginia) for my source bucket and EU (Ireland) for my destination bucket. For clarity, when naming the bucket, you can use the word destination in its name.
Configure Replication on the Source Bucket
From the S3 dashboard, click your source bucket. Click the Management tab and then click Replication.
Click Add Rule. You will see an error indicating that you must enable versioning on this bucket in order to configure replication. Click Enable Versioning. Keep the default selections on the Source page and click Next.
On the Destination page, select the destination bucket you just created. Again, you will be prompted to Enable Versioning on the destination bucket. Click to enable and click Next.
On the Permissions page, select Create a new role and click Next (a role will be created automatically). Click to finish and your replication configuration will be complete.
Configure Permissions on the Both Buckets
Because your website will interface with the Source and Destination buckets to get and upload objects, you will need to configure CORS on both buckets.
Add a CORS configuration to the Source bucket from the Permissions tab. Select CORS configuration and add the following.
<CORSConfiguration>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>
</CORSConfiguration>This rule will allow GET and PUT requests from any origin, with any header.
Once you configure CORS, you will also need to ensure that this bucket has public read and write permissions. From the Permissions tab, under Public Access, click Everyone, then ensure that List Objects and Write Objects are selected. Click Save.
Do the same steps for the destination bucket, however, you will not need PUT permissions in the CORS configuration, and you will not need Write Object permission for Public Access.
Create an Amazon Cognito Identity Pool and Configure IAM Role
Follow the Create an Amazon Cognito Identity Pool and Configure IAM Role section of Exercise 2.3.
However, when you create the IAM role for unauthenticated identities, you will need to write a policy that allows access to both of your buckets. You can use the following for your policy (inserting the names of your buckets below):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::SOURCE_BUCKET_NAME/*"
]
}, {
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
]
}
]
}
Create Another S3 Bucket and Configure it for Website Hosting
Follow the Create an S3 Bucket and Configure the Bucket to Host a Static Website sections of Exercise 2.1.
Create a Browser Side Script to List Objects from Both Buckets
In this exercise, you will be uploading two objects to the website bucket you just created — index.html and app.js. The app.js file will be the browser-side script that performs actions on the S3 bucket containing your files.
Create an index.html file with the following:
<html>
<head>
<!-- Javascript AWS SDK -->
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.247.1.min.js"></script>
<!-- Your Browser-Side script -->
<script src="./app.js"></script> <!-- Latest compiled and minified Boostrap CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
</head>
<body>
<div class="container"> <!-- input button to upload a file -->
<div class="row">
<form>
<div class="form-group">
<label for="upload">Add File</label>
<input id="upload" type="file" class="form-control-file" >
</div>
<button type='button' class="btn btn-info btn-xs" onclick="addFile()">Upload</button>
<form>
</div> <div class="row">
<h4>My Origin Bucket Files</h4>
<div id="originContent"></div>
</div>
<div class="row">
<h4>My Replication Bucket Files</h4>
<div id="replicationContent"></div>
</div> </div> <script>
//calling a method on your browser-side script - app.js
listFiles();
</script> </body>
</html>
Create an app.js file with a listFiles function (called in the index.html file). This function will make use of the JavaScript AWS SDK to get the list of files from both of your buckets (see the S3 class documentation here). You will also need an addFile function that we will implement in the next step:
//insert name of your source bucket
const originBucket = 'SOURCE_BUCKET_NAME';//insert name of your destination bucket
const replicationBucket = 'DESTINATION_BUCKET_NAME';//insert identity pool id for your cognito pool
//you can get this by clicking Edit Identity Pool from your Cognito Pool
const IdentityPoolId = 'YOUR-POOL-ID';const credentials = new AWS.CognitoIdentityCredentials({ IdentityPoolId })//insert region for your source bucket
const originRegion = 'us-east-1';//insert region for your destination bucket
const replicationRegion = 'eu-west-1';const originBucketUrl = `https://s3.amazonaws.com/${originBucket}`const replicationBucketUrl = `https://s3.amazonaws.com/${replicationBucket}`//uses the AWS SDK
//updates its configuration with your credentials and bucket region
AWS.config.update({
region: originRegion,
credentials
});//uses the AWS SDK
//creates an S3 service object that can interact with your source bucket
const originS3 = new AWS.S3({
region: originRegion,
credentials,
apiVersion: '2006-03-01',
params: { Bucket: originBucket }
});//uses the AWS SDK
//creates an S3 service object that can interact with your destination bucket
const replicationS3 = new AWS.S3({
region: replicationRegion,
credentials,
apiVersion: '2006-03-01',
params: { Bucket: replicationBucket }
});function contentContainer(type) {
return document.getElementById(`${type}Content`)
}//function called from index.html
//gets name of all objects in both of the s3 buckets
//updates HTML with the file name
function listFiles() { originS3.listObjects({Delimiter: '/'}, function(err, data) {
if (err) {
return alert('Error: ' + err.message);
} else {
const files = data.Contents;
const fileNames = files.map(function (file) { return file.Key; });
const html = fileNames.map(function(file) {
return `<div>${file}</div>`
})
contentContainer('origin').innerHTML = html.join('')
}
}); replicationS3.listObjects({Delimiter: '/'}, function(err, data) {
if (err) {
console.log(err)
return alert('Error: ' + err.message);
} else {
const files = data.Contents;
const fileNames = files.map(function (file) { return file.Key; });
const html = fileNames.map(function(file) {
return `<div>${file}</div>`
})
contentContainer('replication').innerHTML = html.join('')
}
});
}function addFile() {
console.log('not implemented yet')
}
Upload both files to your website bucket (granting public read permission to both).
Then go to your source bucket and upload any file to the bucket, making sure to grant public read permission on it.
Finally, visit your website URL and you should see the file name displayed in both the origin and replication sections. If your destination bucket is in a region that is located far from you, you may notice that the source bucket file list will display more quickly than the destination bucket file list.
Create a Browser Side Script to Upload A File
In this step, you will update your app.js script to allow you to upload a file to the source bucket. Update the addFile function to the following:
function addFile() {
const files = document.getElementById('upload').files;
const file = files[0];
const fileName = file.name;
const params = {
Key: fileName,
Body: file,
ACL: 'public-read'
}originS3.putObject(params, function(err, data) {
if (err) {
return alert('There was an error uploading your file: ', err.message);
} else {
listFiles()
setTimeout(listFiles, 5000)
}
});
}
You will notice that you are only adding the file to the source (origin) bucket. The callback function after that succeeds makes two calls to the listFiles method. The first one is executed immediately, and the second one is executed after a timeout of 5 seconds.
Upload the new app.js file to your website bucket (grant public read permission). Navigate to your website URL, refresh the page, and upload a file.
You should see the new file listed nearly immediately in the origin/source list. However, you will not see the file listed in the replication/destination list until the second call executes 5 seconds later (as it was not available on the destination bucket immediately after being put to the source bucket).
Clean Up
Navigate to your S3 dashboard. Delete all three buckets, following the instructions from the prior exercises.
Delete your Cognito identity pool following the instructions shown here.
Next: Exercise 2.5: Add Objects to a Bucket With Versioning Enabled
