CI\CD workflow with BitBucket Cloud, Bamboo, AWS CodeDeploy

TL;DR

A bunch of different tools exist in this universe for building CI\CD pipelines. In this article I just want to shed some light on another set of instruments, that you can use to reach the goal. This article can be considered as a continuation of our previous article dedicated to CI\CD practices: Taming CI pipeline with Jenkins, Kallithea, Nexus and more by Yevgen Volchenko

This time we are going to build an example of a classic workflow, based on Atlassian and Amazon products, specifically: BitBucket, Bamboo and AWS CodeDeploy.

You should also take a look at BitBucket — AWS CodeDeploy integration. Atlassian is now trying to move away from Bamboo and add similar (CI\CD) tools to BitBucket. Nowadays it’s possible to skip Bamboo and integrate BitBucket with AWS CodeDeploy directly, however it’s not a thing, we want to have at this point.

This Bitbucket integration is simply intended to streamline pushing code artifacts to instances quicker, but isn’t intended to serve as your primary CI/CD workflow. If you’re looking to build a full continuous integration and continuous deployment workflow, take a look at Atlassian Bamboo. Bamboo is designed for CI, builds, and testing, and you can use the Bamboo CodeDeploy plugin to push code to instances.

Installation and getting your hands dirty

In this setup I will make use of my AWS account and I’m going to store everything on their cloud. We will have to cover following steps:

• Configure AWS side for deployment of your application. You can either use some simple php application or a real world application like WordPress. I’ve tried to play with both.
• Create your account on BitBucket. I’m not going to describe this step in details since it’s pretty trivial and everything is in “Next=>Next=>Finish” style.
• Install Bamboo on AWS. I will add few words about this step later in this article.
• Put it all together and start deploying our application

Configuring AWS side

• Create an IAM instance profile for Amazon EC2 instance that will be used as part of Deployment group for your PHP application. How to is well described in Amazon docs
• Create a service role for AWS CodeDeploy to give it access to AWS resources. Ref. Amazon docs.
• Create a new S3 bucket for CodeDeploy to store its revisions.
• Prepare and launch EC2 instance, where you will deploy your application. I’ve used Amazon Linux t2.micro instance. Make sure you are launching EC2 instance with IAM role, created earlier. Also you should tag this instance (for example, Keyname:Name, Value:CodeDeploy_target), tags are used in CodeDeploy to add EC2 instance to a deployment group. Open required ports and make sure you can connect to your instance, I usually allow 443,80 and 22 from my company IP ranges, should be enough for our deployment testing and 22 for connecting to instance via SSH.
• Install CodeDeploy agent on newly created EC2 instance, by following this guide. You can save this instance as an AMI later just so you don’t have to play with agent installation every time you want to add instance to a deployment group.
• Create IAM user for Bamboo to access CodeDeply and S3 bucket. Note down Access key ID and Secret access key. Here’s a policy attached to my IAM user for CodeDeploy access from Bamboo.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "codedeploy:*",
                "s3:*"
            ],
            "Resource": "*"
        }
    ]
}

• Create Application and Deployment group in CodeDeploy console. Add EC2 instance, created earlier, to your Deployment group. Here you will understand why you have to tag your instance. You will also discover that you can deploy to either EC2, Auto Scaling group or on-premises instances. In this example we are deploying to a single EC2 instance. You can also add some triggers, for example, notification to SNS topic about your deployment status. Integration with CloudWatch is also available. I’d recommend reading this section of AWS docs to get familiar with a full list of CodeDeploy capabilities.

Bamboo installation on AWS

There is a guide on Atlassian’s confluence and a little Bamboo EC2 Wizard binary jar file, that will do the job for you. You can just run this script on your local machine, provide AWS credentials, follow on-screen instructions and get it all done automatically. As a result you will have:

• t2.medium (as minimal available) EC2 instance running Ubuntu 16.04. If you use free tier note that usage of t2.medium instance type is not eligible to free tier accounts and you will be charged by Amazon.
• New data volume attached to this EC2 Instance.
• A bunch of components, required for Bamboo to work properly, and bamboo itself installed on this instance.

All done for now, only one little thing: you have to add inbound rule for TCP port 8085 from your company’s IP ranges to this EC2 instance. I will explain why do you need this in next chapter.

Putting it all together

So, at this stage we have a BitBucket account, configured AWS CodeDeploy and Bamboo build/testing server installed on EC2 instance. Time to glue it all together. Let’s start with Bamboo configuration, and, I bet, first step you would like to cover in Bamboo is linking it with a BitBucket repo, doesn’t sound like a big deal, but I bumped into an obstacle here.

A hitch or “Nice to meet you, CSRF protection”

Bamboo is installed on AWS EC2, using java script from Atlassian. And here’s a little output with script execution results:

Roger that, Atlassian guys! I’m now going to open Bamboo web interface using a link you just provided https://ec2-name.compute-1.amazonaws.com. Everything looks cool so far, lets go to “Shared credentials” and store our credentials that will be used to connect to private BitBucket repository, just so we can reuse it later.

Now lets try to link our Bamboo with BitBucket Cloud repository. Click on a little cogwheel button in a top right corner, choose Linked repositories then Add repository and choose BitBucket Cloud. Give it some name, select private in Repository access then select a shared credential we just created, I’m sure credential pair has admin access to a repository, I‘m trying to add. Looks cool so far, let’s load our repositories by clicking Load repositories, oops Failed to load data from Bitbucket. [403 Forbidden]

I tried to create a public repository on BitBucket and link it with Bamboo by providing just a BitBucket Cloud username and it has failed again.

OK, google, now it’s time for you to step in, let me know what’s happening. What!? You cannot give me an answer? Then, it might be something very very simple and obvious, preventing me from loading my repositories.

As a workaround I was able to connect to my BitBucket repository using GIT provider instead of native BitBucket Cloud.

Let’s browse around and explore Bamboo a bit more, there are some plugins that can be updated: cogwheel button in a top right corner => Add-ons => Update on one of installed plugins:

OK, there I saw a warning message, saying we should pause Bamboo server (button on the same page), let’s do that and as a result: It doesn’t do anything, the same button appears and it looks like we’ve never clicked that one.

At this point I recognized that using workaround and connecting repositories with a GIT provider was not enough, since that was not the only problem we have in our Bamboo installation.

So, let’s dig a bit deeper, there is an atlassian-bamboo.log file under /media/atlassian-data/data/bamboo/current/logs, let’s examine it and see what’s going on in there.

It’s pretty informative, there are my POST request to Bamboo API following by some weird “XSRF Check failed” message, let’s monitor the same log file in real time. We see that “Check failed” message appears every time we click either Load repositories or Pause server, or Update on add-ons page.

Ok, here we can read what XSRF (or CSRF) prevention is and how it works:

• CSRF Wikipedia
• Another article with useful information about CSRF Prevention with a guidelines for setting it up. We are mostly interested in this part of the document.
• CSRF changes in API of Atlassian products
• Even a guide how to Enable\Disable XSRF prevention on Bamboo

Atlassian enables XSRF prevention on their products by default (as for Bamboo, it’s enabled starting from version 5.3). Looks fine, but disabling Cross Site Request Forgery in bamboo doesn’t give any effect and honestly, after reading about CSRF prevention, I don’t think it’s a good idea to disable it at all.

So basically, my problem was caused by XSRF defense blocking API calls due to target-source origins mismatch.

Resolution

Just as I said before, I knew that was something very simple and obvious and I was right, but I didn’t expect that to be that obvious!

Remember Java script you used to roll out Bamboo to your AWS infrastructure? It has given us HTTPS:\\ URLs to access our Bamboo installation. It does not work by default! Use http://ec2-name.compute-1.amazonaws.com:8085 instead. And now XSRF checks passed successfully. Believe it or not, I’ve spent roughly 1,5–2 days trying to fix all my problems in Bamboo due to a lack of proper description in documentation.

!UPDATE:

After submitting a support case to Atlassian I got a reply pretty quick and this article appeared in Bamboo knowledge base. It is related to a bug BAM-18063 reported earlier.

The Bamboo EC2 wizard sets up an apache reverse proxy in front of Bamboo that handles SSL and forwards internally to the tomcat listener on 8085. However, it doesn’t setup the tomcat proxy attributes in /media/atlassian-data/installs/bamboo/atlassian-bamboo-5.xx.x/conf/server.xml

Once you configured your server.xml by following this instructions you should also update base URL in General Configuration of Bamboo to match URL you use in a browser.

So we’ve found out what was a problem and how to fix it. Let’s continue to configure our workflow.

• Push your PHP application to BitBucket repository.
• Create a new project and build plan in Bamboo by following steps in this guide
• Feel free to create different stages if you feel like you need to. In my sample project I’m using a default job in a default stage.
• Within a job create a Task. First task should be already there “Source Code Checkout”. It’s ok, we need to have that one and now we need to add another task that will fire up AWS CodeDeploy and make it do its job (Create a deployment and deploy a revision to a target group). Follow the steps on this page. At this steps you will find terms, you already familiar with if you have gone through configuring AWS side part of this article, such as AWS credentials, Amazon S3 bucket, Application name, Deployment group.
• If we want (and we definitely do) our application to be deployed on each BitBucket repository commit we need to add a trigger to our Bamboo Server and a hook (Bamboo service) to our BitBucket. Detailed description can be found here.
• Add test tasks to your Bamboo plan. I don’t use any tests in this sample scenario, anyway here is an example how to add test task for your PHP application in Bamboo. Once you added test task in Bamboo feel free to invite your QA team for a cup of tea and ask them how to write automatic tests for your application.
• To be able to deploy with CodeDeploy you have to add file appspec.yml to a root directory along with WordPress files. Example of appspec.yml and scripts required to prepare target server and deploy application are uploaded to this GitHub repository.
• In this scenario I’m deploying WordPress, so I have to take care of database configuration and overall initial configuration of WordPress. MySql database and user are created manually on a target server, however you can make another script, feel free to experiment here and automate the process. Installation of all dependencies (php, Apache, Mysql server and so on) is covered by scripts\install_dependencies.sh.

Does it work? And some results.

Let’s figure it out by committing some changes to our BitBucket repository. I just modified a single line in a file wp-admin/install.php, commit has triggered a Bamboo plan and here’s what we have on our web server running on target EC2 instance:

OK! Here’s how Bamboo plan summary looks:

Bamboo Plan Summary

Bamboo also reports plan execution statuses back to a BitBucket Cloud:

Build Statuses at BitBucket

Everything looks nice and simple at first glance. You might even wonder what is the reason for having this article if everything is that crystal clear and transparent, if you think so, I just want to remind you of an “Iceberg of success” by Sylvia Duckworth

Hope it has been informative for you. Good luck, have fun!