Continuous integration of Github repo with S3 bucket using TravisCI
Overview
Continuous integration might seem to be very complicated at first, especially when it comes to coders, but don’t intimidate we will try to understand the concept of CI as simple as we can and then will try to implement continuous deployment of a Git repo on AWS S3 bucket using TravisCI.
To understand CI, imagine a scenario where your team is working on a project, committing changes to version control, and every time they commit a change, you want the code to be built, tested and deployed on S3. One way can be, you build the code locally, test it and then upload all static files in S3 manually or using aws cli to accomplish the deployment task. But then doing all these manually is a bit overhead right? You have to do all these steps, every time a new commit occurs.
And this is where continuous integration comes into the picture.
Its a process of building, testing and deploying the code automatically every time team member checks into version control
In this article, we will see how we can integrate TravisCI with our project in Github to AWS S3. We will use following components:
- Version Control : Git
- Project Type : Node JS
- CI tool : TravisCI
- Cloud Object Storage : AWS S3
There are many CI tools available, but we will be using TravisCI because its free for all open source projects hosted on the GitHub and for the first 100 builds otherwise. You can find the list of available CI tools here
Steps to follow
- Setting up Git Repo
- Setting up Travis account
- AWS Setup
- Setting up S3
- Creating IAM user for TravisCI
- Test the IAM user has access to S3 bucket or not. (Optional)
- Installing and using AWS CLI
- Uploading files to S3 using aws-cli through IAM user
4. Configuring .travis.yml file in repo
Setting up Git Repo
Create a git repository and push your node js project code to it.
Setting up Travis account
Go to this link https://travis-ci.com/ and signup with GitHub account. Provide Github username and password -> Authorize to Travis -> Press on Activate button -> Choose either all repos or selected a repo -> Press Approve and Install.
Once you are in the home page, on your desired repository, press on setting button -> Enable Build only if .travis.yml file present.
AWS Setup
Setting up S3
- Login to AWS console and create an S3 bucket.
- Once the bucket is created, go to Properties tab and enable
Static website hosting
Creating IAM user for TravisCI
- Go to AWS console and search for IAM -> add a new IAM user
- Give
Programmatic Access
- Copy
Secret Access Key
andAccess Key ID
, we are going to need it later. - Once the user is created, click on them and on the
Permission
tab, hitAdd Inline permission
link. - Open the JSON tab and past the following JSON. Make sure you replace the resources by your bucket name.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::your_bucket_name"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:DeleteObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::your_bucket_name/*"
]
}
]
}
- Click on create policy button, give a proper name and save.
Test the IAM user has access to S3 bucket or not
This step is optional, you may skip this step.
Installing and using AWS CLI
Let’s use AWS cli to login with our new user and uploading files to S3… This is for Ubuntu.
- Now that you have installed python, use the command to install awscli
sudo apt install awscli
- Verify aws is installed,
aws --version
- Configure aws with a newly created user.
- Type command
aws configure
- Provide Access Key ID, Secret Access Key that you have copied earlier.
- Provide
Default region name
as us-east-1 (depending upon your configuration) andDefault output
as JSON.
Note: If some error occurs while installing awscli, please install python first.
Uploading files to S3 using aws-cli through IAM user
- Create index.html file in a folder and cd to into it.
- Use command
aws s3 cp index.html s3://[bucket_name]/index.html
to copy the file to s3 bucket. Don’t forget to replace the bucket name in the command.
Configuring .travis.yml file in repo
In TravisCI, builds are configured using .travis.yml file which contains the build tasks that will be executed on running the build.
Clone the existing repo and create .travis.yml file in root folder. Add following code into it.
language: node_js
node_js:
- node
cache:
directories:
- node_modules
deploy:
provider: s3
access_key_id: "YOUR_ACCESS_KEY"
secret_access_key: "YOUR_SECRET_KEY"
bucket: "YOUR_S3_BUCKET"
acl: public_read
skip_cleanup: true
local_dir: build_webpack
branches:
only:
- master
addons:
apt:
update: true
- This config file is telling Travis to build the project as node project. Hence the language is specified as node_js
- Replace “YOUR_ACCESS_KEY” and “YOUR_SECRET_KEY” with keys that you have copied earlier.
- Change the bucket field as well.
- local_dir field tells Travis to deploy only the specified directory (here build_webpack) to the bucket so that you won’t have to deploy your whole code.
- apt -> update true : property tells to run apt-get update.
- acl property can set Travis correct file permission. In this case it’s public. To read more here.
Now you can do changes in any file, say index.html. Build your project using command, say npm run build
then commit the changes. Travis will detect the new commit and trigger the build for you.
You might have noted that we are providing Access Key and Secret Key in .travis.yml and then checking this file along with other changes. Doing this will reveal your key so it’s better to encrypt these keys first and then add them to .travis.yml file. You can accomplish this task in either of these two ways.
Download travis client -> Open terminal -> type command travis encrypt --add deploy.secret_access_key secret_access_key="YOUR_SECRET_KEY"
OR
(Easiest way) Open TravisCI web app -> login -> click on setting
button near your repo -> look for environment variables section -> Add YOUR_SECRET_KEY, YOUR_ACCESS_KEY and YOUR_S3_BUCKET as variables and specify their proper values as well. Now, you can refer to environment variables from .travis.yml like this :
....
access_key_id: "$YOUR_ACCESS_KEY"
secret_access_key: "$YOUR_SECRET_KEY"
bucket: "$YOUR_S3_BUCKET"
....
Hope this article will help you in CI next time. For more, please refer to TravisCI doc