Continuous Integration with CircleCI 2.0, GitHub and Elastic Beanstalk

CircleCI, GitHub and Elastic Beanstalk

At Gradient, we’ve spent a fair bit of time looking at how we do continuous integration and delivery within our development pipeline. We recently evaluated some of the hosted CI platforms on offer including TravisCI, CircleCI and Codeship as well as more traditional build platforms such as a self-hosted Jenkins server. We decided on CircleCI, specifically their v2.0 API, as our framework of choice for our Continuous Integration.

CircleCI 2.0 is a big overhaul of the architecture of their CI platform, moving to a container driven approach. This makes it very easy to build out custom environments or take advantage of environments that already exist. More details can be found here https://circleci.com/blog/say-hello-to-circleci-2-0/

The big attraction for us was the balance between speed, cost and ease of use when getting started. We found CircleCI was incredibly fast at initiating and running builds and their customisation through Docker was fantastic. The two big features of interest were the ability to SSH directly onto your builds in order to debug and the ability to run local builds.

Having settled on CircleCI we wanted to integrate our existing pipeline which deploys to Elastic Beanstalk. The rest of this article will show you how we went about setting this up.

Create a deployment key for GitHub

We are fans of GitHub and keep a lot of our code in private repositories. In order to get deployments to work with CircleCI, we had a couple of approaches we could take.

1. Create new ssh keypairs for each repository and add them as deploy keys.
2. Create a machine user and invite them as read only collaborators on each project.

We opted for a machine user as it makes management and configuration far easier. We only have a single user and key pair that we need to manage rather than one per repository. Additionally, it makes internal processes for deployment far simpler for developers.

There are also risks with this approach, the obvious one being that having only one key pair with access to a lot of your private repositories makes the impact much greater should that key pair be compromised.

To create a machine user you just need to create a brand new GitHub user account, we opted to call ours gradientbot.

Once you have created the account, you need to set up a keypair for the user. This can be done by running the following command at a terminal.

ssh-keygen -t rsa -f <username>-github-deploy

where <username> is the name of the new user you created.

Once you have created the key pair you need to grab the public key and add it to the profile of your machine user. To get the public key either open the public key file or run in the terminal.

cat <username>-github-deploy.pub

Copy the output and go to Settings in the account menu of your machine user and select SSH and GPG keys. Click New SSH Key in the top right and give your key a title. Paste the contents of your public key into the input box and click Add SSH Key.

Adding an SSH Key to your machine user

Add deployment key to S3

In order to get CircleCI to deploy your repository into Elastic Beanstalk, it is going to need to have access to the private key of your machine user. If you followed the steps above you should have already generated a key pair.

After a bit of reading around, we settled on the suggestion of this StackOverflow post. Essentially we are going to create a bucket in S3 and store our private key in it. This will then be readable by an IAM deployment user that we will create in AWS.

First we need to create the bucket in S3 and store our private key in it. This can be done by visiting the S3 web console in AWS and clicking Create Bucket. Give your bucket a meaningful name, we named ours gradient-deployment-key.

We need to add the following policy to the bucket under the set bucket policy tab to allow it to be read from by users with the Elastic Beanstalk role. Note <ACCOUNT_ID> needs to be changed to your AWS account id.

{
  "Id": "Policy1497534632804",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1497534629565",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::gradient-deployment-key/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT_ID>:role/aws-elasticbeanstalk-ec2-role"
        ]
      }
    },
    {
      "Sid": "Stmt1497534629566",
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::gradient-deployment-key",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT_ID>:role/aws-elasticbeanstalk-ec2-role"
        ]
      }
    }
  ]
}

Once the bucket is created you’ll need to upload your private key, if you are following along with the post, this will be the key created earlier called <username>-github-deploy.

Create IAM deployment user

Now we have our bucket with our private key we need to set up a user for deployment. In the AWS IAM console create a user called circle-ci and set them as a console user. Save the access key and secret.

Once the user is created we are going to attach the following policy to it that enables it to interact with Elastic Beanstalk when deploying. This is a slightly modified policy based upon the work of jakubholynet

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "elasticbeanstalk:CreateApplicationVersion",
                "elasticbeanstalk:DescribeEnvironments",
                "elasticbeanstalk:DeleteApplicationVersion",
                "elasticbeanstalk:UpdateEnvironment",
                "elasticbeanstalk:CreateStorageLocation"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "sns:CreateTopic",
                "sns:GetTopicAttributes",
                "sns:ListSubscriptionsByTopic",
                "sns:Subscribe"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sns:<REGION>:<ACCOUNT_ID>:*"
        },
        {
            "Action": [
                "autoscaling:SuspendProcesses",
                "autoscaling:DescribeScalingActivities",
                "autoscaling:ResumeProcesses",
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:UpdateAutoScalingGroup",
                "autoscaling:DescribeLaunchConfigurations"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "cloudformation:GetTemplate",
                "cloudformation:DescribeStacks",
                "cloudformation:CreateStack",
                "cloudformation:CancelUpdateStack",
                "cloudformation:ListStackResources",
                "cloudformation:DescribeStackResource",
                "cloudformation:DescribeStackResources",
                "cloudformation:DescribeStackEvents",
                "cloudformation:DeleteStack",
                "cloudformation:UpdateStack"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:cloudformation:us-east-1:525824562834:*"
        },
        {
            "Action": [
                "ec2:DescribeImages",
                "ec2:DescribeKeyPairs",
                "ec2:describeVpcs",
                "ec2:*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetBucketPolicy",
                "s3:Get*",
                "s3:List*"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::elasticbeanstalk-<REGION>-<ACCOUNT_ID>",
                "arn:aws:s3:::elasticbeanstalk-<REGION>-<ACCOUNT_ID>/*"
            ]
        },
        {
            "Action": [
                "s3:CreateBucket",
                "s3:GetObject",
                "s3:ListAllMyBuckets"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "elasticloadbalancing:DescribeInstanceHealth",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "autoscaling:DescribeScheduledActions",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
}

Add deployment user to CircleCI

Find your project in CircleCI and click the settings gear to the right of it. Scroll down to the Continuous Integration section and select AWS CodeDeploy.

Circle CI AWS settings menu

Input the Access Key ID and the Secret Access Key that you were given to you when you created the AWS IAM user circle-ci

Click Save AWS Keys. Circle CI says that it will create a config for your builds in ~/.aws/config with these details, however, in CircleCI 2.0 I could only find them presented as environment variables.

Configure Elastic Beanstalk to install the machine users private key

With the help of markplindsay’s StackOverflow post we can specify a configuration file in the .ebextensions/ directory of our project. This is picked up be Elastic Beanstalk when it starts up the environment and causes it to retrieve the deployment key. This key can then be used for pulling in private GitHub dependencies (in the case of npm or pip) as well as pulling your project source code.

Just make sure you remember to invite the machine user as a collaborator on the project.

As you can see the config is very straight forward, all that needs updating is the output of ssh-keyscan -H github.com into the content property and update the commands section to point to the bucket and private key names you created. This file should be saved with a sensible name, e.g GitHub-deploy.config. Config files are executed in alphabetical order in Elastic Beanstalk so prepending a number e.g. 04-GitHub-deploy.config is a practise often used by developers.

files:
    "/root/.ssh/config":
        owner: root
        group: root
        mode: "000600"
        content: |
            Host github.com
                User git
                Hostname github.com
                IdentityFile /root/.ssh/<username>-github-deploy
    "/root/.ssh/known_hosts":
        owner: root
        group: root
        mode: "000644"
        content: |
            #
            # paste output of `ssh-keyscan -H github.com` here
            #
commands:
    01-command:
        command: sudo aws s3 cp s3://gradient-deployment-key/<username>-github-deploy /root/.ssh
    02-command:
        command: sudo chmod 600 /root/.ssh/<username>-github-deploy

Configure CircleCI

We now need to modify our .circleci/config.yml file in our project to actually deploy into Elastic Beanstalk.

The start of the file defines the version CircleCI API that we will use. This is then followed by a jobs declaration.

Our build job defines a working directory and a docker image. CircleCI provides lots of Docker images in the Docker Hub. We have created our own image based upon the circleci/node suite of images that installs python, pip and the awsebcli. This is publicly available for anyone to use.

version: 2
jobs:
  build:
    working_directory: ~/build
    docker:
      - image: gradientco/node-elastic-beanstalk:7.10.0

We now need to specify the job steps

steps:
      - checkout

      - run:
          name: Run tests
          command: yarn test
      - run:
          name: Setup AWS credentials
          command: |
            mkdir ~/.aws && printf "[profile eb-cli]\naws_access_key_id = ${AWS_ACCESS_KEY_ID}\naws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}" > ~/.aws/config
      - deploy:
          name: Deploy to Elastic Beanstalk
          command: |
            eb deploy environment-name

The steps are broken down as follows

1. Checkout code.
2. Install dependencies using Yarn.
3. Run our Test Suite.
4. Create the ~/.aws/config file that the eb cli relies upon for authentication. As stated earlier, despite claiming that it is, this file does not appear to be created by CircleCI automatically so we just need to pull in the environment variables from a file.
5. Deploy the code where environment-name is the name of your Elastic Beanstalk environment.

That’s it, if you have got this far you have made it and should now be able to commit your code, push it to CircleCI to have it build and deploy directly into Elastic Beanstalk!

I hope this is of some use to you. I’ve pulled together this guide based on notes I made over the course of an afternoon implementing the above solution at Gradient. We’re big fans of CircleCI and look forward to working with it more. If there are any corrections please let us know and we will fix them as soon as possible. Thank you to all the blog posts and StackOverflow questions that helped us get this done.