How to build a fully automated kanban power-up for your GitHub Project board using a custom webhook

Published in

The Andela Way

14 min readAug 22, 2018

TLDR; You can tune-up your GitHub Project board with a custom web-hook integration to meet your software development project management needs. You can checkout the code github, remix it on glitch, or deploy a working instance for repository with zeit/now.

Screenshot of a project board used by the creative team at Andela using a custom webhook integration I built for the team.

Managing the software development process can be a bit of a PITA. This is often not due to the actual coding aspect of it but more often than not, it is due to the tools fatigue you get from incorporating different tools to manage the whole process. And of course, there are a million and one tools out there to “help your team focus on the work and not the process”, but in truth, no one tool out there usually fits the bill, at least not entirely–and this is understandable, after all no two teams are alike or work in the exact same way.

In this post you’ll build a fairly simple and extensible kanban power-up for GitHub Projects that automates the triaging of new issues to project boards and to track issue statuses automatically.

The problem you’re solving for here is pretty straightforward. GitHub already has support for managing your projects on a kanban-like board. And GitHub Projects allows for some automation, for example you can set up the native project column automation to add a new card when an issue is opened or closed and move issues along the project column when a pull request is submitted.

Screenshot of an overview of Github Projects features.

As useful as these automations are, there’s still room for improvement. For example, with the current setup, you’re unable to automate the movement of issues along your project columns (or pipelines) to track the status of a feature/issue on a project board. Also, you’re unable to triage issues between multiple projects in a repository.

Built-in automations for Github Projects

With this power-up, you’ll solve the following:

Moving a project card along the board automatically.
Triaging issues to different project boards automatically.

You’ll make use of the GitHub labels feature to solve these, with the webhook events ‘labeled’ and ‘unlabeled’ as the triggering mechanism.

The final solution will look this:

GIF showing how to automate the movement of issues along a project board using status labels.

GitHub has a few things that make it super simple to customize:

A powerful API. You’ll be using the new, all-powerful GraphQL API v4 and a touch of the old GitHub REST API v3.
A web-hook that allows you to subscribe to events in your github repository, in our case you’ll subscribe to the ‘labeled’ and ‘unlabeled’ events on an issue.
A labeling feature for issues with a name, description, and color.

Configuring your GitHub repository

Let’s start with the most challenging bit in my opinion, creating the right taxonomy for our issue labels. Remember we’re looking to solve for two things, to move an issue along the project board based on it’s status and to triage issues to a project board. You’ll employ the not-so-often used description property of a label to help us identify an issue’s status as well as the project it belongs to.

Before I built this solution which is part of a project management solution our creative team at Andela uses, I reached out to GitHub support to figure out the best way to accomplish this.

Email correspondences with Github Support

I ended up employing the two recommendations from Ivan, the GitHub support representative. With the right labels, you can subscribe to the ‘labeled’ and ‘unlabeled’ events and use the payload delivered to your web-hook to add an issue to a project board and to move that issue (now a project card) along our project columns.

GitHub labels have three properties that you can configure, a name, a description, and a color. You’ll use the name and description properties to set up our new labeling system. Feel free to go berserk with your label coloring scheme!

Tracking issue progress using status labels

Tracking issue status with Github labels.

Create a status label by giving it a name that represents a stage in issue management process and description of ‘status'.

The status labels you create will mirror the columns in your project board. As in the screenshot above, you’ll have the following statuses/project columns:

incoming — all new tickets.
scheduled — all tickets that have been reviewed.
in progress — all tickets that are being worked on.
in review — all tickets that have been worked on and are awaiting approval.
completed — all tickets that have been worked on and have been reviewed and accepted.
canceled — all tickets that were canceled for any possible reasons.
blocked — all tickets that are being blocked for one reason or another.

While these might seem like a lot, I think it more closely mirrors what actually happens in most teams. There can be only one issue label with the description of status. When you implement this, there will be only one label with the description ‘status’ on an issue.

Triaging issues to project boards using labels

A list of Github Projects for kanbanize.

To triage issues to different projects, you’ll use a similar technique like you did to track issue statuses. You guessed it, labels to the rescue again! However, naming things can be hard so let’s employ semvar–naming convention for our projects. For example, imagine you’re managing a huge open source library, you may want to employ the major version releases as project names like so v1.0.0, v2.0.0, and so on. Again this is just my preference. You can name your projects anything you want, the only requirement is to add ‘project’ as the description. When you implement this, there will be only one label with the description ‘project’ on an issue.

Example of a Github **project** label. This example uses semvar versioning for project names.

Of course your project names can be completely different just ensure that

Your project name match labels with the label description ‘project’.
Your project columns match labels with the label description ‘status’.

Configuring a GitHub web-hook

To add a new web-hook to your GitHub repository, find the settings tab in the upper right hand corner in your github home.

From the screenshot:

Enter a url that the payload will be delivered to your web-hook. For local development, you can use ngrok or localtunnel.
Select ‘application/json’ as the content type for the payload sent to your web-hook through the url you entered in 1 above.
Enter a simple secret that you’ll use to sign the request you get from github to validate it.
Pick individual events to tailor what events you subscribe to.
Subscribe to event issues.
Hit save.

That’s it!

Simple as that, you now have a fully configured GitHub web-hook. The last thing you need to do before diving into the code is to create a GitHub token. This will allow you to authenticate your requests to GitHub’s API. To get a personal token, go to your GitHub settings, create a new token giving it a memorable name, and copy the token to a secure place.

You’re now done with Github, time for some code!

1-minute introduction to GraphQL

To work with your Github issues and projects, you’ll use the new GitHub GraphQL API v4. Before diving into the code though, I thought it may be useful to give you a brief intro to GraphQL. I won’t do justice to GraphQL here so you can click this link to learn more about GraphQL.

GraphQL is a query language for your API built by the amazing folks at Facebook. The specifications for this query language are such that it allows you to essentially tell the server exactly want you want from it.

Here’s a somewhat contrived example of how you construct a request to a GraphQL service:

{ 
  profile {
   id
   name
   email
   likes
  }
}

The response from the server will looks like this:

{
  "data": {
    "profile": {
       "id": "someuniquealphanumerickey",
       "name": "Jane Doe",
       "email": "jane@doe.com",
       "likes": 200
     }
  }
}

Notice how the JSON returned from the server matches the query you constructed!

If you come from the world of REST APIs, you know that in order to consume an API, you need to send GET, POST, PUT, PATCH, and DELETE requests to different endpoints on the server. Now imagine that instead of consuming the different endpoints that may exist for any of these actions, you can access all that data through one endpoint, picking and selecting the fields you need, and traversing through the data using a simple, intuitive, and self-documenting interface.

Verbatim from the documentation, “every GraphQL service defines a set of types that completely describe the set of possible data you can query on the service”. This means that there’s always a schema that defines how communication is carried out between the server and the client. And in this schema are types that you define for every GraphQL service, two of these are special types– query and mutation–out of several that may exist in the schema. These two special types define the entry point of every request to the GraphQL service. And of these two types, the query is most often used to make requests to the service and it is free of side-effects, whereas the mutation modifies objects on the service.

So, in our case, instead of using the Github REST API v3 (although, you’ll use this to edit the issue labels), that has a root endpoint–https://api.github.com–and a gazillion category endpoints (seriously, I counted over 489 endpoints and you can count them yourself here), we can leverage one endpoint to meet all our needs–https://api.github.com/graphql.

You’ll see examples of GraphQL queries and its advantages below as we figure out the right requests for our power-up.

Consuming the GitHub GraphQL API v4

Now, let’s dig into GitHub’s GraphQL API v4 documentation to figure out what it permits us to do. We need to be able to query the issue and projects fields from the repository object. Also, we need to be able to mutate (or update) a project card and project board. Head over to the GitHub GraphQL API Explorer to see all the fields on the repository object.

query {
  __type(name: "Repository") {
    name
    kind
    description
    fields {
      name
    }
  }
}

Once on the API Explorer, enter query above and you should see a response that looks like the one below (I’ve truncated the output to save space).

{
  "data": {
    "__type": {
      "name": "Repository",
      "kind": "OBJECT",
      "description": "A repository contains the content for a project.",
      "fields": [
        ...        {
          "name": "issue"
        },        ...        {
          "name": "projects"
        },        ...      ]
    }
  }
}

From this, you see that we’re able to query on the issue and projects fields of any repository. When working with a GraphQL service, you’re mostly querying or mutating fields of an object, in this case a GitHub repository object. All in all we need to construct the queries on the issue and projects fields to do the following:

Find an issue by number.
Find a project by name.
Find all project columns.
Add an issue to a project column.
Move a project card to a different column.
Delete a card from a project.

With these, we’ve got everything we need to write the necessary queries and mutations for the web-hook.

GraphQL operations

Here we’ll go through each of the query and mutation operations listed in the previous section so that you understand what’s going on.

To test these out as we go along, you can use the GitHub GraphQL API Explorer with the following object containing the required query variables.

{
  "owner": "ENTER_GITHUB_USERNAME",
  "name": "ENTER_GITHUB_REPO_NAME",
  "number": "ENTER_GITHUB_ISSUE_NUMBER",
  "projectName": "ENTER_GITHUB_REPO_PROJECT_NAME"
  "issue": {
    "contentId": "ENTER_CONTENT_ID",
    "projectColumnId": "ENTER_PROJECT_COLUMN_ID"
  },
  "card": {
    "cardId": "ENTER_PROJECT_CARD_ID",
    "columnId": "ENTER_PROJECT_COLUMN_ID"
  }
}

Note: The explorer makes use of real production data so you may want to try running these queries using a test repo.

FindIssue

Find an issue by number.

GraphQL query to find an issue. In graphql/query/findIssue.js

With this query, you’re looking up an issue by its number, and picking the following fields id, title, projectCards, labels etc. You’re also telling the server to return the first hundred project cards and the first one-hundred labels associated with the issue.

FindProject

Find a project by name.

In graphql/query/findProject.js

With this query, you’re looking up project columns by projectName and picking the first project matching it on the repository object, and picking the first column of that project board and the first twenty cards of that column.

FindProjectColumns

Find all project columns.

GraphQL query to find project columns. In graphql/query/findProjectColumns.js

This query looks up a project by projectName (aka label) and selects the first two projects on the repository object along with first twenty columns of that project board and the first one-hundred cards of each column. It is an example of a complex and expensive query. And as you would expect with any api service, Github provides a guideline on how to moderate api requests. This single call replaces probably 10 or more REST requests. And that’s another advantage of GraphQL, a single complex call replaces possible thousands of REST requests!

The server cost for this query is:

1 (one repository) + 1*2 (first two project cards) + 1*2*20 (first twenty columns for each project card) + 1*2*20*100 (first one hundred cards of a project column) = 4, 043 total nodes.

To get the score, you divide this by 100, meaning that the score is 40.43!

GitHub’s GraphQL service has a limit of 5000 points per hour and you’ve used up only 0.8% for this query. What power at your disposal!

AddProjectCard

Add an issue to a project column. This produces a side-effect since we updating the project object to include a new card.

GraphQL mutation to find add a project card. In graphql/mutation/addProjectCard.js

MoveProjectCard

Move a project card to a different column. This produces a side-effect since we’re updating the project column.

GraphQL mutation to move a project card. In graphql/mutation/moveProjectCard.js

Fields like cardEdge are provided on a mutation request to select what the server will return to you once the mutation operation is complete.

DeleteProjectCard

Delete a card from a project.

GraphQL mutation to delete a card from a project. In graphql/mutation/deleteProjectCard.js

Here’s you’re asking the server to return the deletedCardId once the deleteProjectCard mutation is complete.

Now that you have the queries and mutations and are sure they work, we need to set up the project locally and bring it all together.

Set up our project locally

Let’s wire this thing up!

You can checkout the code on github or remix it on glitch.

kanbanize
├── README.md
├── actions
│   ├── index.js
│   ├── labeled.js
│   └── unlabeled.js
├── bin
│   ├── start-dev.sh
│   └── start-ngrok.js
├── config
│   ├── components
│   └── index.js
├── deploy.sh
├── graphql
│   ├── index.js
│   ├── mutation
│   └── query
├── index.js
├── lib
│   ├── crypto.js
│   └── github.js
├── now.json
├── package.json

Here’s our project structure. You can try to recreate this or clone it from GitHub. We’ll go through each relevant module.

If you’re recreating it run the following code:

mkdir kanbanize && cd kanbanize && yarn init -y # create a ne directorymkdir actions app bin config graphql lib # create subdirectoriesyarn add micro crypto graphql-request github-api joi async # install dependenciesyarn add micro-dev dotenv-safe nodemon ngrok concurrently --dev # install devDependencies

If you’re cloning, run the following in your terminal:

git clone git@github.com:pauldariye/kanbanize.gitcd kanbanize && yarn # install dependenciescp .env.example .env # Enter the env variablesyarn dev             # Run dev server; Visit https://localhost:3000

Again, to save time, we’ll go through only important bits of the code.

You can checkout the code on github or remix it on glitch.

Our “app”

This is your main server file that uses micro, a small and performant library for building asynchronous HTTP services built by the amazing team at Zeit. As you can see, the entire app is just one asynchronous function. No big deal!

Our micro service. In app/index.js.

Here are notes on what the app function (request handler) does:

It the content-type delivered to our web-hook to ensure it is application/json like we configured in our web-hook configuration. [LC14]
It validates the request for the presence of an x-hub-signature. [LC28]
Highlight this if you’ve read this far (trying to see who my real friends are).
It validates the request for the presence of an x-github-event header and checks that the event type is issues. [LC35]
It validates the request for the presence of an x-github-delivery header. [LC49]
It checks that the x-hub-signature sent in the request body is the right one by comparing the hash value you get to validate the request body using the signRequestBody method with the secret key you created when configuring the web-hook . [LC56]
It checks that you have a defined handler for the payload action. [LC63]
Finally, it invokes your handler for the payload action. [LC71]

A tiny module to verify that we’re getting the request from Github. In lib/crypto.js

The signRequestBody function generates a cryptographically signed hash-based message authentication code (HMAC) with a given key (web-hook secret) and string (our request body).

Initialize the GraphQLClient with our github token. In lib/github.js

Here you’re initializing an authenticated graphqlClient using graphql-request library and an issues client using the github-api lib which is a wrapper for the GitHub’s REST API.

Now you may be asking yourself why you’re using the REST API after all the spiel I gave earlier on the merits of GraphQL. Great question and thought! As with any new technology or paradigm, it takes some time to retroactively replace what was formerly in place. You can read more about why GitHub decided to move to GraphQL and track the GraphQL schema changelog to learn more. A more straightforward answer is that we need the REST API to be able to update our github issue, i.e to ensure we have only one status or project label at a time.

Other notes about code in thelib/github.js module:

removeDups : This function checks to see if an issue has other labels besides the recently labeled that have a project or status description and removes them. [LC19]
addProjectCard : This function adds a project name that matches the label name with description project. [LC45]
deleteProjectCard: This function removes a card from a project card. [LC69]
moveProjectCard : This function moves a card to a project column that a label name with description status. [LC83]

Action for the labeled Github event that we subscribed to. In actions/labeled.js

Here you’re handling adding a card to a project and moving the card along project columns using the ‘labeled’ event (action) function which accepts the event payload from GitHub.

The handler does the following:

It finds the issue with the number matching the one received from the event payload and a project by name (or label) using the queries we defined earlier. [LC18]
It then checks the issue’s description to either addProjectCard for a project label or moveProjectCard for a project status label. [LC24]

Action for the unlabeled Github event that we subscribed to. In actions/unlabeled.js

Here you’re handling what happens when a label with a project description is removed from a card. In this case we deleteProjectCard from the project board.

Summary

That’s it! Of course this is only the tip of the iceberg when it comes to what you can do with Github and other tools like it (Gitlab, bitbucket, etc) to bring your software development process ever closer to you codebase. Give this library a try and let me know what you think. Feel free to contribute more custom actions.

You can checkout the code on github, remix it on glitch, or deploy a working instance for repository with zeit/now.

Some noteworthy projects that integrate nicely with Github to give you much more powerful project management features include:

Zenhub — Agile Project Management for Github
Waffle.io–Developer-first Project Management for Teams on Github

Be sure to check these out as well.

Sincere thanks to Tams Sokari, Solomon Osadolo and Akinjide Bankole for reviewing and proofreading this post.

Do you need to hire top developers? Talk to Andela to help you scale.
Are you looking to accelerate your career as a developer? Andela is currently hiring senior developers. Apply now.

Thanks for reading. Give this post a 👏🏿 if you enjoyed it to help others discover it.

Say hi on twitter.