AWS cross-account S3 automated bucket replication with Jenkins.

Published in

Globant

7 min readOct 22, 2021

Introduction

In daily operations performed against S3 services, one of the most common tasks is actually enabling S3 bucket replication between local or external AWS accounts. This kind of task can become a regular operation and therefore an automated strategy can be set to supply the demand of this service.

In this post, I will share with you how to turn hours setting these bucket configurations into seconds when it comes to S3 replication configuration.

Key features

Automated replication between buckets.
Automated granular IAM permission set for buckets replication.
Automated rule configuration for bucket replication.
Automated S3 rule policy injection for bucket replication.
Exiting S3 bucket policy validation to avoid legacy policy permission removal.
Existing bucket validation.
Detailed log operations.
Automated replication removal (clean up).
Pipeline strategy with SCM.
Resources and policies are dynamically labeled to be recognized as an automated integration.
Error control for an empty parameter in Pipeline Build.
Error control for nonexisting buckets.

Diagram

Workflow

Jenkins gets a Jenkins file from the source code (SCM) and sets the pipeline as well as the bash scripts for the automation.
Jenkins triggers the job with the bucket names as parameters.
Jenkins uses its AWS role to assume a role in account A to set replication in the source bucket.
Jenkins uses its AWS role to assume a role in account B to grant replication permission in the destination bucket.
Jenkins logs to screen if either the operation is succeeded or failed.

Recipe

Code from an SCM provider (Github, bitbucket, etc).
A Jenkins instance.
A Jenkins Pipeline
Jenkins Git plugin.
AWS Roles for cross-account strategy.
AWS S3 existing buckets.

Note: Although this Jenkins pipeline was planned to actually create buckets on-demand, the feature was retired since some organizations must comply with extra security measures when comes to bucket creation. Since these measures are not in the scope of this article the feature was dismissed.

Steps:

AWS DevopsMasterRole

The AWS DevOpsMaster role will act as the role with the capability to assume the so-called DevOpsSlaveRoles in either local or external accounts. Therefore you must:

Deploy the DevOpsMaster role in the so-called AWS Account X either using the AWS web console or deploying the cloudformation template provided here:

Note the commented out lines, these should be changed to match your AWS external accounts Ids.
Use the resource arn:aws:iam::*:role/DevOpsSlaveRole for testing purposes only.

2. Attach the DevOpsMaster role in your Jenkins instance (EC2, EKS, ECS, etc).

Note: As seen in the cloudformation template the DevOpsMasterRole will be used from an ec2 instance since the service allowed to assume the role is ec2.amazonaws.com. Make sure you grant this permission to the right principal (AWS service, I am, etc.) according to the AWS guides referenced below.

Refer to the AWS guides to attach the role in your Jenkins instance depending on the environment used:

AWS DevopsSlaveRole

The AWS DevOpsSlaveRole role will act as the role assumed from the Jenkins instance set in account X and it will have the capability to perform tasks in the AWS external accounts so-called AWS Account A, AWS Account B, etc. Having said so, you must:

Deploy the DevOpsSlaveRole role in every external account either using the AWS web console or the cloudformation template provided here:

Note the commented out lines, these should be changed to match your AWS granular permisions.
Use the policy arn:aws:iam::aws:policy/AdministratorAccess for testing purposes only.

Jenkins Plugin

Install the GitHub plugin. Refer to the plugin installation guide here for further details.

Jenkins Pipeline Creation

Download the files needed for the automation from here (s3replication folder).
Browse the envawsaccounts.sh script in the path/s3replication/scripts/envawsaccounts.sh and look for the following line:
export awsaccountids=”accounta accountb”
Replace the ”accounta accountb” with your actual AWS account Ids corresponding to local or external accounts. It should look as follows:
export awsaccountids=”11111111111 22222222222”
Note: These AWS account ids must match with the ids you have included in the DevOpsMasterRole in the previous step.
Upload the modified code to your SCM system.
Create a Jenkins pipeline:
- Browse your Jenkins installation.
- New Item.
- Provide an Item Name
- Select Pipeline and click Ok.

5. Inside the Job’s Screen:

Scroll down to Pipeline:
- In definition, select the Pipeline script from SCM option.
- In Repository URL, provide your repository URL.
- In Credentials, provide or configure your repository credentials.
- In Branch Specifier, provide the branch in which your code was uploaded.
- In script path, provide the path s3replication/jenkinsfile/s3rpl.Jenkinsfile
- Hit the SAVE button.

6. Build the Jenkins Job:

In SourceBucket parameter, provide the name of your existing S3 bucket.
In DestBucket parameter, provide the name of your existing S3 bucket.
In Terminate Parameter, leave this value by default unless you want to disable bucket replication.
Hit the Build Button.
In Build History, roll your mouse pointer over the build number, a small arrow will deploy a drop-down list. Select Console Output from the list.
See the log output either in the Jenkins console or just scroll down this page to see the attached logs from Jenkins.

Detailed Jenkins Job Logs:

#########################
Validating buckets existence.
##########################

SOURCE BUCKET sourcebucket FOUND
DESTINATION BUCKET destinationbucket FOUND

############################################################################################################################################
ALL BUCKETS FOUND - SUMMARY
SOURCE BUCKET INFO: NAME: sourcebucket ACCOUNT: AWSACCOUNTA - 1111111111
DESTINATION BUCKET INFO: NAME: destinationbucket ACCOUNT: AWSACCOUNTB - 2222222222

############################################################################################################################################

#################################################################
Preparing destinationbucket bucket policy for replication.
###################################################################


An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist

###############################################################################################################################
The destination destinationbucket bucket does not contain an S3 Replication policy. Creating S3 Replication Policy...
#################################################################################################################################


#######################################################
The destinationbucket bucket S3 Replication policy is now set.
#########################################################


########################################################
Checking versioning status at Destination bucket destinationbucket.
##########################################################


###############################################################
Versioning successfully enabled at Destination Bucket destinationbucket.
#################################################################


########################################################
Checking versioning status at source bucket sourcebucket .
##########################################################


###############################################################
Versioning successfully enabled at Source Bucket sourcebucket.
#################################################################


########################################################
Checking IAM Role for replication from sourcebucket.
##########################################################


############################################################
Creating IAM Role sourcebucket-S3ReplicationRoleAUT.
#############################################################

{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17", 
            "Statement": [
                {
                    "Action": "sts:AssumeRole", 
                    "Effect": "Allow", 
                    "Principal": {
                        "Service": "s3.amazonaws.com"
                    }
                }
            ]
        }, 
        "RoleId": "YOURAROAROLEID", 
        "CreateDate": "2021-10-21T21:39:58Z", 
        "RoleName": "sourcebucket-S3ReplicationRoleAUT", 
        "Path": "/", 
        "Arn": "arn:aws:iam::11111111111:role/sourcebucket-S3ReplicationRoleAUT"
    }
}

An error occurred (ReplicationConfigurationNotFoundError) when calling the GetBucketReplication operation: The replication configuration was not found

########################################################
Checking S3 replication rule for bucket sourcebucket.
##########################################################


#################################################################
Creating replication rule ReplicateTo-destinationbucket-AWSACCOUNTB for bucket sourcebucket.
###################################################################


###############################################################
The replication S3 rule ReplicateTo-destinationbucket-AWSACCOUNTB was created with success
#################################################################

[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Final results

Let's check how the resources are named and look like in AWS:

AWS Account A / Source bucket.

Objects Tab:

Management Tab:

Replication Rule:

IAM Role / Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetReplicationConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::sourcebucket",
                "arn:aws:s3:::sourcebucket/*",
                "arn:aws:s3:::destinationbucket",
                "arn:aws:s3:::destinationbucket/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::sourcebucket/*",
                "arn:aws:s3:::destinationbucket/*"
            ]
        }
    ]
}

AWS Account B/ Destination bucket.

Objects Tab:

Permissions Tab / BucketPolicy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3RPLFrom-AWSACCOUNTA-sourcebucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::11111111111:root"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetReplicationConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:GetBucketVersioning",
                "s3:PutBucketVersioning",
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": [
                "arn:aws:s3:::destinationbucket",
                "arn:aws:s3:::destinationbucket/*"
            ]
        }
    ]
}

Conclusions

As seen with an automated solution there are key benefits as described below:

A quite fast configuration when running the job. 13 seconds approximately on each execution.

Automated S3 replication configuration with no human interaction in AWS console.
Granted validations for the overall operation.
Human-friendly names/descriptions on each resource that is modified by the automation.
A convenient roll-back/disable replication button when needed.