GitHub + Jenkins + Hubot + Slack
The Joys of Continuous Integration and Chat Ops at Meedan
Those trochees make me want to construct a children’s rhyme…
github jenkins hubot slack
test the code, deploy your hack
show your peeps and have a snack
no need to fret, it’s all on track
when you github jenkins hubot slack
Enough of the kiddie stuff. Here’s what’s so fantastic about deployments at Meedan:
By tying these technologies together, deployments at Meedan can be kicked off from any of…
- a push to GitHub
- the Jenkins web interface
- from Slack (via interaction with @hubot)
The build/deployment output is captured by Jenkins and SUCCESS or FAILURE gets reported into our Slack channels by hubot. This provides our distributed team a consistent and highly visible deployment strategy in approachable interfaces.
How we got there
Recently we’ve been migrating our deployment infrastructure from Webistrano over to Jenkins. Webistrano has been really great, and it’s been quite the workhorse for Meedan for several years, offering one click repeatable deployments and good visibility.
Unfortunately Webistrano is unmaintained, and it doesn’t mesh all that well with our Docker infrastructure. And Webistrano is not without its quirks, the most apparent of which is its tendency to overrun the output buffer and leave you blind to the progress of a deployment (will the spinner ever stop? what’s going on in there?)
Webistrano (and Capistrano on which it’s based) relies on ssh’ing into a system to deploy new code, which is very different from the Docker method of building and bundling the entire environment with the code and shipping it as a container. We’ve used a hybrid system for a while by deploying Docker containers that included ssh servers which would then be used for deploying code via Websitrano, but this practice is frowned upon. However, the legacy deployment strategy was working fine, and our development team was well oriented to Websitrano.
GitHub + Jenkins
A couple months ago, I started to assess Jenkins, and after confirming that there was a fantastic Docker Plugin and that it seemed to be getting used more and more for Docker container deployments I downloaded the official Jenkins container from Docker hub and setup a deployment for one of our NLP developers. She had been ssh’ing into a system and kicking off a build script (#!/bash build container, restart container and setup a hipache reverse proxy connection to the container). That worked, but it was a bit clunky and didn’t feel very approachable. About once every other week the developer would reach out to me and ask for my support.
The Jenkins job was an immediate improvement. I set up a deployment key in the GitHub repository for the code base, configured the Jenkins GitHub plugin and set the GitHub Jenkins webhook. At first, I reused the script the developer was already using for the build and deploy, though I quickly moved to using the aforementioned Docker Plugin. She pushed code, then the container got rebuilt and re-deployed.
If something does go wrong, the output is all recorded there in the web interface. My intervention hasn’t been required in several weeks at this point. It just feels great to set something up and then get out of the way of someone trying to get their work done.
Since then we’ve turned off the GitHub hook for automatic rebuild/redeploy. Sometimes we make multiple pushes to this codebase all at once, and then want to deploy at a time of our choosing. All I had to do was click one checkbox in that Jenkins job, and it went back to being a manual deployment.
Hubot + Slack
We started using Slack in early 2014, and within a few hours, our work lives had radically changed for the better. Meedan’s product, design and development teams are spread over four continents and seven cities and a few other locales. Most of us spend a ton of our work days in Slack communicating and (hopefully) getting things done. By tying in the integrations for GitHub, tickets, and deployments, we have very nice visibility into how work is proceeding.
Since the beginning I’d had my eye on the Slack Hubot integration. I’ve long been an IRC user and have witnessed how bots can provide great utility to a conversation. And of course, GitHub’s usage of Hubot for Chatops has become famous. I just knew integrating Hubot into our Slack environment would be a huge productivity boost…. and maybe a bit of fun around the edges :)
@hubot image me bell biv devo poison
@hubot image me cc deville poison
@hubot translate me from english into portuguese poison
Okay, so there are a fantastic number of fun things which can be done with Hubot, and a growing community of plugins and utilities. But it isn’t all just pointless doohikies. For example the @hubot translate command has been helpful for working with translations from Meedan’s product Bridge.
GitHub + Jenkins + Hubot + Slack
When I realized that we could easily use the Jenkins Hubot plugin I knew that this was going to be a fantastic stack. I built a nodejs container for Hubot and included the plugins hubot-jenkins-slack and hubot-jenkins
hubot-jenkins-slack plugin
It’s easy to configure a Jenkins job to send job status to Hubot’s REST API. Hubot will then boadcast Jenkins Job Status messages to the Slack channel specified in the URL
hubot-jenkins plugin
Jenkins jobs can be triggered directly from Slack. This is the true chatops…
bfoote: @hubot help jenkins
hubot: help jenkins…hubot jenkins list <filter> — lists Jenkins jobs
hubot jenkins b <jobNumber> — builds the job specified by jobNumber. List jobs to get number.
hubot jenkins build <job> — builds the specified Jenkins job
hubot jenkins build <job>, <params> — builds the specified Jenkins job with parameters as key=value&key2=value2
hubot jenkins describe <job> — Describes the specified Jenkins job
hubot jenkins last <job> — Details about the last build for the specified Jenkins job
It’s really helpful to kick off Jenkins jobs from Slack. It precludes the entire “Did you do it?”/”Yes I did it” query and declaration since it’s all happening right there in real time.
bfoote: jenkins build cc-varnish, DOMAIN=meedan.com
hubot: bfoote: (201) Build started for cc-varnish
Important things I’ve learned along the way
- backup $JENKINS_HOME!!
- not all jenkins plugins are your friend
- upgrading jenkins plugins requires the care and handling of any other software upgrade
- don’t use deployment keys in GitHub for you private repositories. According to the fine folks at the OSCON GitHub booth, the way to do this is to create a jenkins_deployment user with its own ssh key and assign it to a read only deployment team. This saves a ton of time on both the Jenkins and GitHub sides and is a bit clearer overall in GitHub
- clean out all but the last 5 builds
- have jobs trigger other jobs on SUCCESS
- for most jobs, just post SUCCESS to Slack
- for important jobs post all steps to Slack
@hubot animate me Victory…
And we’re just getting started. As we convert jobs from Webistrano to Jenkins it’s allowing us to take advantage of both our Docker infrastructure and our testing infrastructure in ways which we hadn’t anticipated. Victory indeed!
no need to fret, it’s all on track
when you github jenkins hubot slack
