Collaborating on a Project

with GitHub, Slack, Codeship, HoundCI and Waffle.io

[This article was republished on the Waffle.io blog — thanks guys 👍]

I know quite a few junior developers (including some of my students) who have reached a point with their side projects where they’d like to collaborate with others. While they could just stick their apps on GitHub and get cracking, it’s possible to set up a slightly more complex system of tools to make collaboration a little bit easier.

The configuration of Continuous Integration servers, build pipelines, online task boards and API integrations sometimes scares (or bores) developers, but with the exception of git, none of the tools I discuss here are overly complicated and we can configure a simple pipeline quite easily. This stuff isn’t just the preserve of DevOps people — I promise not to mention Docker at all 😉.

Who should read this?

  • Developers who’ve never set up a CI server and integrated it with tools like Slack and Waffle.
  • Developers who feel their workflow is too clunky, or prone to error.
  • BAs, Testers and Project managers who want to know what all this is about will find Part II useful. (You’ll possibly get lost, but a skim read might help enlighten you.)

If you want to follow along, I assume the following:

  • You’ve got a Github account and it’s all set up locally.
  • You know how to use git to add, commit and push changes to GitHub.
  • You have a Slack organisation and have the rights to add integrations.
  • You’ve got an account on Codeship (they’re free).

We’ll be looking to configure each of these tools so they work together as a coherent system. I’ll also explain one of the workflows we could use to develop an new feature with this setup.


Part I — Setting it all Up

The Project

I’m going to use a really simple ruby project, but the principles (and many of the steps) are much the same for other languages and frameworks.

I’ve set up a simple Sinatra project with rspec and cucumber tests. If you can read ruby, you should take a look at the code on GitHub.

The project we’ll be using

Some general points to note:

  • I’m using bundler to manage my dependancies.
  • There’s a .rubocop.yml file containing settings for Rubocop, which checks that my ruby code meets the Ruby Syleguide. This file switches off some of the rules I don’t agree with.
  • The Gemfile and Procfile are set up to allow us to deploy to Heroku later.
  • I use rbenv to manage my ruby versions, so the correct ruby version is stored in .ruby-version.

The tests are fairly trivial, since they’re just examples. We can run three sorts of tests inside this project:

rspec # Runs the RSpec unit and integration tests
cucumber # Runs capybara automated UI tests
rubocop # Checks the syntax and style of the ruby code
Rubocop, RSpec and Cucumber tests — all are pretty trivial

Let’s assume that we’ve just coded this project locally, and we now want to collaborate on it with some other developers. The first thing is to get it up on github.

(If you want to follow along, you could clone the project from github and the remove the .git directory with “rm -rdf .git”. This will remove everything related to git from the folder.)

Step 1 — GitHub

First things first, we need to turn our projct into a git repository. I have the hub gem installed, so I can create repos at the command line with git create. If you don’t you’ll have to create your repo via the web interface and push to it.

git init
git add .
git commit -m "Initial commit"
git create applejuice
git push -u origin master
git browse

This puts our project on GitHub for us:

Next, we want to invite our teammates to collaborate on the project. We can do this in Settings > Collaborators. If you’ve created your project under an organisation rather than a personal account, you’ll be able to add teams as collaborators here.

That’s it for github configuration — for now at least.

Step 2 — Slack

Although some of the conversation around a project will happen in GitHub’s issues and pull requests, it’s a good idea to have somewhere less formal for the team to chat. For that, we can set up a Slack channel. We’ll obviously want to invite our collaborators to this, too.

Next, we want to connect our new channel to GitHub so it can receive notifications. Head to the Slack Apps & Custom Integrations section and add the GitHub app. You’ll have to authenticate with GitHub if you don’t have it set up already.

Once we’ve authenticated, we can add a new GitHub configuration connected to our channel. I’m going to configure slack to track two branches from github: master and production (we’ll create a production branch later). I’m also going to alter some of the default configuration settings.

We’ll test this out once we’ve set up Codeship for continuous integration.

Step 3 — Codeship CI

Next step: set up Codeship to work with GitHub and Slack. Log into Codeship and choose Create New Project > Connect with GitHub Repository. We should see our repo listed:

We now want to add our setup and test commands. Setup commands will spin up our app, and test commands will execute the tests. These will differ depending on your project, but for our AppleJuice app they look like this:

While we’re logged into Codeship, let’s configure the integrations with Slack and GitHub. Head to Project Settings > Notification. Check that the GitHub integration is enabled, then follow the instructions to set up the Slack notifications. Don’t forget to scroll down and click “Save” in both Codeship and Slack.

While we’re in our project settings, we may want to add our teammates as collaborators in Codeship — you can do that under the Team Members section.

The last thing we’re going to do is test out the integrations we’ve set up so far. To do that, we’re going to add a Codeship badge to our README.md file. You can get the markdown code from the bottom of the Notification section in the project settings.

Let’s add that to our README file, commit and push to github.

If we go to our project in Codeship, we should see that a build has started automatically. We can see what’s going on by clicking on the build:

As we can see, codeship has cloned our repo from GitHub, checked out the latest commit and is now running the first of our setup commands. Presuming the setup commands execute ok, it will run our test commands.

Since Codeship does’t have ruby 2.2.4 installed (at the time of writing), the build fails:

We can also see notifications for the commit and the failing build in Slack. It looks like out integrations are working properly.

To fix the build, we need to switch our .ruby-version and Gemfile to use ruby 2.2.3 and make another commit. We can see that this time, the build is green:

If we dig into it a bit, we can also see the results from our rspec and cucumber tests are displayed:

Finally, we can see that our Slack channel has been updated with the latest build and commits:

Step 4 — A Staging Server on Heroku

Now that we have our automated tests set up, we want to have our app deploy to a staging server whenever those tests pass on CI. We’re going to use Heroku for that. I’m assuming you have the heroku toolbelt installed and have logged in with heroku login. In that case, the next thing is to create a heroku app and push our master branch to it.

heroku create applejuice-staging
git push heroku master

It’s a good idea to get the heroku deploy working from your terminal before we connect it to Codeship since there are often some teething problems.

Once the deploy is working, we should be able to see out app live at http://applejuice-staging.herokuapp.com. We can also view it in the Heroku dashboard. At this point, we might want to add some collaborators to the Heroku account.

Our app live on the staging server (left) and in heroic (right)

Now that we’re all set up on Heroku, we can configure Codeship to push our project to it every time we get a green build. We’re going to set the master branch from GitHub to push to the staging server. Later, we’ll configure a production branch that pushes to our production server. On Codeship, go to Project Settings > Deployment, enter the master branch, select Heroku and enter your Heroku app name and API key:

(If you’re app is using a database, you’ll want to click More Options and set up automatic backups and migrations.)

Now that our staging server is set up, let’s test the build pipeline. We’ll make a change to the app and push it to github. This should trigger the following automatic actions:

  • The slack channel is notified of our commits.
  • A build starts on Codeship.
  • The slack channel is notified when the build passes or fails.
  • If the build passes, the app is deployed to the staging server.

Step 5 — A production server on Heroku

Now we’ve got a staging server sorted, it’s time to set up our production server. Your project setup will dictate how best to do this, but for the AppleJuice project we want the master branch to be our ‘development’ branch — the one we actually do work on — and a seperate branch called production to hold our live app’s code.

With this setup, new work should only happen on master, and when we want to release to production, all we have to do is merge master into production. We’ll come back to workflow later on — for now, let’s create the production branch and push it to Github.

git branch -b production
git push origin production

We should now be able to see the branch on Github:

Next we create our production app. We’ve currently got two git remotes: origin which points to Github and heroku which points to our staging server. We’re going to create a new app on Heroku, but instead of using the default name for heroku git remotes (“heroku”), we’re going to call this remote production.

heroku create applejuice-production --remote production

We can now see our three remotes and where they point:

Next we’re going to push our production branch to the production remote. (Remember the syntax is git push <remote> <branch>). This causes us a bit of a problem though, since Heroku will only build the master branch of things we push to it. We need to tell heroku to treat our production branch as if it’s the master branch.

git push production production:master

We should now be able to see our production app on https://applejuice-production.herokuapp.com/. Since our customers are actually going to use this app, it’d be a good idea to add a custom domain. Note that we need to specify the app, since there are now two heroku apps associated with our repository.

heroku domains:add applejuice.danny.is --app applejuice-production

We’ll also have to add some DNS records on our domain. I use DNSMadeEasy so for me I need to do something like this:

We can now access our production app on http://applejuice.danny.is/.

Since we want any changes to the production branch on Github to automatically deploy to our production app, we need to configure Codeship. The steps are identical to those used when configuring the master branch to deploy to our staging app. Start by choosing Add new deployment pipeline, but this time set production as the branch and our production app as the target.

If you’ve set up a custom domain for your production app, you might want to tell Codeship about it under more options.

We won’t actually test this pipeline now, since we’re going to test the whole end-to-end process once we’ve set up a few more tools.

Recap

We’re now in a position where any commits to master will trigger a build on Codeship and, if successful, deploy to our staging server. Whenever we want to make a release to production we can merge master into the production branch. This will trigger a build and, if it’s all green, deploy straight to our production server.

Step 6 — Checking your Style with HoundCI

We’re already set up with a pretty good development pipeline, but later on we’re going to use a feature-branch workflow to work with it. This means we’ll be using pull requests.

HoundCI is a fairly simple tool, but is great for keeping code quality high. It checks over any commits made against a pull request and highlights places where it deviates from the language styleguide.

Setting up HoundCI couldn’t be easier: log in or sign up with GitHub and select your repo from the list, then click Activate.

Hound uses rubocop to check ruby, but it’s got a fairly strict rubocop configuration by default. Since we already have rubocop configured in our project, We’re going to tell it to use our config by adding.hound.yml file to the repo. We’ll also switch off scss and javascript checks:

We’ll see exactly what hound does later on.

Step 7 — A Task Board with waffle.io and GitHub Issues

Since the whole object of this system is to facilitate collaboration between teammates, it’s probably a good idea to have some way of tracking our tasks. Ideally, we’d use an Agile Task Board. We might choose to use an agile project management app like Sprintly, JIRA or Pivotal Tracker — all of which can be integrated with Github and Slack — but for a small project like this it makes sense to use a feature that’s baked into Github: Github issues. While it’s not as full-featured as the other tools out there, it does have some advantages:

  • It’s part of GitHub, so is already integrated with everything.
  • It works perfectly with Pull Requests.
  • If our project is public, it allows external contributors to raise and discuss issues.
  • It’s how almost everyone manages issues and tasks for open source projects hosted on Github.

Since it’s not an agile project management system it does’t come with a task board, but we can use Waffle.io to bolt one on.

Head over to waffle.io and sign up or login with GitHub. We can add a new board by clicking the plus button and choosing our repo.

This provides us with a nice Kanban Board with some default columns. We can change the columns and ad WIP limits in the settings, and we should invite our teammates to collaborate by sharing the url with them.

Let’s check that everything’s wired up ok by creating a new issue. We’ll assign the issue to ourself and label it as an enhancement. Once we’re done, the issue will apear in our Backlog.

There are also options to size (or estimate) and comment on the issue.

Becaue waffle is just an interface for Github, we can see our new issue on there too, and we get a slack notification to tell us it’s been created:

We’ll see how powerful this setup can be later on, but before we look at the workflow, let’s hook up one last tool to our system.

Step 8 — Improving Slack with Bugbot

Bugbot is a fairly simple slack integration that allows anyone on slack to view and create github issues directly from Slack. You can install it by visiting the bugbot website and clicking the big Add to Slack button. Once you’ve chosen the channel to post to, you can authenticate with GitHub in slack and set your preferred repo:

We’re now able to look at the issues assigned to us by typing /bugbot issues into slack.

Finally, if we need to create an issue, we can do so in slack. If I type “/bugbot add Demonstrate the Feature branch Workflow”, I’ll get a notification back from Github and the issue will show up in waffle.


Part II— How do we use all this stuff?

Now we’re all set up, let’s have a look at a typical workflow.

Likely as not, we’ll start the day with a chat in the slack channel and a few silly gifs thanks to the Giphy integration. We might even have a daily stand-up via videoconference:

Perhaps after some discussion, we head over to the task board to choose a task to work on. We’ll this one, with the ID 4.

In our terminal, we pull down the latest version of the project and create our own branch to work on the feature.

git pull
git checkout -b some-new-feature-#4
git push -u origin some-new-feature-#4

Using the issue ID in the branch name tells waffle to update the task board. The issue will automatically be assigned to me, and moved into In Progress.

Once we’ve done a bit of work (in this case added some code to our layout file), we’ll commit it and use the hub gem to create a pull request right away. Opinion varies on wether this is a good idea, but I’ve found it provides a good place to see what’s been done on the branch in one place.

git add views/layout.erb
git commit -m "Added footer to layout"
git push # push to my feature branch on github
git pull-request -b master -h some-new-feature-#4 -m "Some New Feature. Connects to #4"

We specify that the pull request is from some-new-feature-#4 to master and provide a message explaining what the feature is about. By using “Connects to #4” in our title, we’re telling Waffle link the PR to the existing issue, instead of creating a new one. We can see the pull request on GitHub, and appended to our issue on Waffle.

Slack tells us that our push triggered a build against our feature branch on Codeship, but since it’s not set to deploy that branch nothing further happens — we just know that our tests all pass on the CI environment

We can now use the github pull request to discuss the changes we’re making with teammates and continue to add code to the feature branch. Any new commits we add to the branch will show up in the PR. Let’s change some ruby, commit it and push it to our feature branch.

Although slack tells us that our tests are still all passing, we’re notified of some comments on the pull request. Let’s take a look:

HoundCI commenting on a pull request.

We can see that HoundCI is providing feedback on our code style, in much the same way that one of our teammates might review our code and leave comments. Let’s fix those issues and make a few more changes.

Although Hound has given us the all-clear, it looks like we’ve got some failing tests on our branch in codeship. Because of our setup, we’re notified of this both in Slack and on the pull request. The details link will show us the failure on codeship. In this case we’ve got a failing RSpec test.

Once we’ve got all the tests passing on CI again, Codeship should let us know and the pull request will go green.

When we think we’ve finished work on the feature, it’d be normal to ask a teammate to code review it:

At this point, we’re going to merge our changes into the master branch. There are a bunch of ways we can do this. One is to use the big button on github, but a more reliable way is to use an interactive rebase to clean up the commits on our branch. You should read up on this, but the commands are:

git checkout master && git pull
git checkout some-new-feature-#4
git rebase -i master # Lets us clean up our commit history
git checkout master
git merge some-new-feature-#4
git push origin master

This last push fires off a build against master and if successful, deploys our feature to the staging server. Here’s our new footer:

Providing any manual testing we do on the staging server doesn’t throw up anything that needs fixing, we can safely delete the local branch and close the pull request, optionally deleting the remote branch.

git branch -D some-new-feature-#4 # Delete local branch

When the pull request is closed, waffle will automatically move it into the done column on our task board.

Once our team has added enough features to the master branch for a release, we only need to merge master into production to see those changes go live.

Closing Notes

Our original aim with all this was to build a system to help us collaborate on a project with teammates and simplify our deployment process. Once our team’s become comfortable with this setup, we might consider refining it in any number of ways. We could:

  • Reduce the number of notifications sent to slack by only having master and production branches send notifications.
  • Introduce other checks into our build pipeline. Coveralls, Codeclimate and CodeCov can provide feedback on our test coverage, while Gemnasium can keep our dependancies up to date.
  • Adopt the Gitflow branching model in place of feature-branches.
  • Integrate monitoring tools like Sentry, NewRelic and Airbrake into our system.

If you’re not already using a CI server, hopefully this has given you some inspiration to set one up.

If you enjoyed this article, I’d really appreciate it if you’d take a second to recommend it on Medium (by clicking the ❤ button), and let me know you liked it on twitter 👍.

Show your support

Clapping shows how much you appreciated Danny Smith’s story.