Continuous Integration and Continuous Delivery with BitBucket Pipelines — Part 1

“A waterfall pouring down from a rocky plateau with steep mountains at the back” by Alexandre Godreau on Unsplash

We use Atlassian’s BitBucket hosted git service to manage our codebase and repositories. It works well for us and integrates nicely with the other systems we use (JIRA, Slack etc.).

We have historically used Jenkins as our continuous integration (CI) solution -a service that runs all the tests for our codebase, after each commit. It’s a powerful piece of software but it can be tricky to configure sensibly: you’ll usually end up relying on many different plugins to get the functionality you need working as required.

There are many less flexible and more opinionated CI services available these days. They can be simpler to get started with, if you’re starting a project from scratch, and in particular if you use a docker-based workflow.

Sometime ago, Atlassian added their own CI offering, Pipelines, as part of their BitBucket service.

Credit: https://bitbucket.org/product/features/pipelines

We’ve had the opportunity to evaluate BitBucket Pipelines with a new data-science API of ours. It wasn’t set up in our traditional Jenkins based CI service, so it was a good candidate for trialing Pipelines.

Like many newer CI services, Pipelines is based on Docker. You create a configuration file that imports a docker container image and add any further configuration steps. Your tests are then run, isolated in the container.

We don’t use docker in production, we use chef to configure our machines. So we had to configure the docker container that would run our Python application’s tests.

This sounds like a tricky extra step but our API doesn’t have many dependencies, provisioning docker containers is actually quite straightforward and we already had a vagrant configuration (for building a VM to run the API) that documented what was required.

The first attempt was a very simple procedural script that went through step by step and installed the software and services we needed:

# base from a standard ubuntu docker image
image: ubuntu:16.04
pipelines:
    default:
        - step:
            caches:
                - pip
            script:
                ## provision git, python, mysql etc. ##
                - apt-get update
                - export LANG=en_US.UTF-8 && export LANGUAGE=en_US:en && export LC_ALL=en_US.UTF-8 && apt-get install -y language-pack-en
                - apt-get install -y software-properties-common
                - add-apt-repository -y ppa:deadsnakes/ppa
                - apt-get update
                - apt-get install -y git
                - apt-get install -y python3.6-dev
                - apt-get install -y python3-pip
                - apt-get install -y libmysqlclient-dev
                - apt-get install -y mysql-client
                ## set up our test database ##
                - mysql -u root -plet_me_in -h 127.0.0.1 -e "CREATE USER 'adzuna'@'%' IDENTIFIED BY 'adzuna'"
                - mysql -u root -plet_me_in -h 127.0.0.1 -e "GRANT ALL PRIVILEGES ON *.* TO 'adzuna'@'%'"
                - mysql -u root -plet_me_in -h 127.0.0.1 -e "CREATE DATABASE ontozuna CHARACTER SET UTF8 COLLATE utf8_bin"
                # setup the python testing environment
                - pip3 install virtualenv
                - make setup_python_dev_environment
                - source ~/.virtualenvs/ontozuna/bin/activate
                - cp env.example .env
                ## run our tests ##
                - tox -e py35
                ## deploy the application! ##
                - pip install awsebcli --upgrade
                - eb init --platform python --region us-east-1 ontozuna_terraform
                - eb deploy ontozuna-prod
            services:
              - mysql
definitions:
    services:
        mysql:
            image: mysql:5.7
            environment:
                MYSQL_DATABASE: pipelines
                MYSQL_ROOT_PASSWORD: let_me_in

This worked well: after every commit to every branch, our container was configured with all the software and services we need, our test suite was run and the results were fed back directly into BitBucket. We could then see the feedback when looking at individual commits and pull-requests.

Towards Continuous Delivery

Pipelines can do more than simply run all your tests. It can also deploy your application for you, i.e. it can help facilitate continuous delivery of your systems.

Continuous Delivery is the ability to get changes of all types — including new features, configuration changes, bug fixes and experiments — into production, or into the hands of users, safely and quickly in a sustainable way.

Source: https://continuousdelivery.com/

In our initial configuration this was achieved by the following snippet:

- pip install awsebcli --upgrade
- eb init --platform python --region us-east-1 
- eb deploy ontozuna-prod

This is using AWS Elastic Beanstalk to deploy our application.

Fantastic! But, every branch is now deployed into production (when the tests pass). This isn’t what we want. If someone is working on an feature branch, we don’t want that deployed into production before it’s merged to master.

So how do we test every branch but only deploy the master branch into production?

You can configure more than one step, some which run on all branches and others that run on specific branches. However, you can’t have a series of steps that depend on one another (as far as I can tell). So for example, you can’t have one step that runs your tests on all branches, then a following step that deploys the application, only if it’s the master branch.

The first solution seemed to be duplicate the steps for the master branch and the default branch but only put the deployment steps into the master branch’s steps.

This would work but it does mean there’s lots of duplication between the steps.

One way to reduce the duplication would be to build a custom docker image and use that for all the branches. This would reduce all the steps to configure the machine.

We haven’t done this yet. Instead we’ve put the steps into groups of shell scripts we can call as needed:

image: ubuntu:16.04
pipelines:
    default:
        - step:
            caches:
                - pip
            script:
                - export LANG=en_US.UTF-8 && export LANGUAGE=en_US:en && export LC_ALL=en_US.UTF-8
                - bash scripts/pipeline/000-configure-container.sh
                - bash scripts/pipeline/001-configure-environment.sh
                - bash scripts/pipeline/002-run-tests.sh
            services:
              - mysql
    branches:
        master:
            - step:
                caches:
                    - pip
                script:
                    - export LANG=en_US.UTF-8 && export LANGUAGE=en_US:en && export LC_ALL=en_US.UTF-8
                    - bash scripts/pipeline/000-configure-container.sh
                    - bash scripts/pipeline/001-configure-environment.sh
                    - bash scripts/pipeline/002-run-tests.sh
                    - bash scripts/pipeline/003-deploy-to-staging.sh
                services:
                  - mysql
definitions:
    services:
        mysql:
            image: mysql:5.7
            environment:
                MYSQL_DATABASE: pipelines
                MYSQL_ROOT_PASSWORD: let_me_in

Now we have a vaguely sensible way to share the setup for different branches and we’ve added an extra step for the master branch so that if the tests pas, we deploy the changes to our staging environment.

Improvements

So we have a working yet basic testing and deployment pipeline but there are definite improvements that can be made.

• Artifacts: we’re not building artifacts and using them throughout the process.
• Deploys: we’re not using the explicit deploy feature of the pipelines that provides better visualisation and tracking of the deployments.
• Production: we’re only deploying to staging here, production deployments have to be done manually by the team.
• Docker image: we don’t have a custom docker image to speed up and simplify the builds.

That’s all for part 1 folks.

In part 2 we look at further improvements that can be made to our pipeline.