Continuous deployment using Jenkins

By: Alexander Kojevnikov

When I joined at Thumbtack, back in late 2015, we had continuous delivery infrastructure for monolith builds. As more engineers joined, we noticed that a significant amount of time went to deploying the latest build. Moreover, there was a trend of having bigger deploys (so call train deploys) and rollbacks tend to be harder. It was a clear indicator we needed to invest into the deployment pipeline.

Continuous delivery

At Thumbtack, we use Gerrit for code review. As soon as a Code Review is submitted, Jenkins starts the build process. The source code is fetched from Gerrit’s Git repository and series of unit tests are executed. In the case when there’s a failure, the build process is aborted and the faulty build is discarded. In the case when unit tests pass, the next phase of the build is triggered-integration tests. Again, in the case of failure, the build is discarded. Otherwise, the build will be marked as safe to deploy and becomes available for deploy.

Image for post
Image for post

Continuous deployment scoping

We kept few things in mind when building continuous deployment:

Gerrit plugin to prevent Code Review submission

As mentioned above, Gerrit is used for code reviews and hosting Git repository. Historically, engineers would learn about restricted period during on-boarding process. In order to deploy code, one would have to look at specific Slack channel and check is there “lock” for monolith repository. Lock would be put manually when we enter restricted period (office off-hours) or when there’s rollback & revert in progress. However, we witnessed multiple times engineers would submit something while the monolith repository was “locked”.

We decided to build a Gerrit plugin that allows engineers to properly lock any repository. The Gerrit UI contains information about current status, and properly rejects commits from non-owner of the lock. An engineer can lock and unlock the repository by running a script which then runs Gerrit SSH command.

Image for post
Image for post

Moreover, we added a Jenkins job that locks the monolith repository automatically during off-hours.

Image for post
Image for post

Continuous deployment Jenkins job

After that, we introduced a new phase of build pipeline. It picks up the latest successful build, sets canary traffic to 0% and deploys build to canary cluster. Once deployed, we increase production traffic gradually over ~10 minutes. At that point, 100% of traffic goes to canary cluster and production cluster gets new build.

Engineers that submitted code are notified about changes going to out via Slack in deploy channel. In case there’s flood or spike of errors, an alert is triggered and posted to deploy Slack channel. If an engineer notices something is off with latest code (e.g. via automated alert, error log, metric dashboards), he/she can easily take a lock which breaks deployment. At that point, it’s up to an engineer to decide what to do — usually move all traffic off canary cluster, revert faulty commit and unlock once safe build is available.

Image for post
Image for post
Continuous deployment flow
Image for post
Image for post
Rollout flow

Future work

This system has been in production for over a three months and we’re pretty happy with it. However, there’s always room for improvement:

If any of those problems sound interesting to you, make sure to check our open positions.

Originally published at https://engineering.thumbtack.com on April 10, 2017.

Thumbtack Engineering

From the Engineering team at Thumbtack

Thumbtack Engineering

Written by

We're the builders behind Thumbtack - an online marketplace that matches customers with local professionals to accomplish their projects.

Thumbtack Engineering

Stories from the Engineering team at Thumbtack

Thumbtack Engineering

Written by

We're the builders behind Thumbtack - an online marketplace that matches customers with local professionals to accomplish their projects.

Thumbtack Engineering

Stories from the Engineering team at Thumbtack

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store