GitHub Board Slack Notifications

Part 3/3: “A fork in the road”

Before continuing see Part 1 the TL;DR there and Part 2

Using slack /commands to subscribe to boards from a channel

Recap

Last time we talked we had finished up a working model of the system. It got messages from GitHub into Slack as Lambda function. It was secured, it went back to GH to get info about the ticket that was moved and the board, knew about notes and posted to Slack via the super simple Incoming Webhooks Feature. We thought it was almost over. 💪

Incoming Webhook configuration in Slack

However, it was one of those cases when 80% done means you’re only 20% done.

Yummy leftovers 🥢

One of the big things that was missing in this initial implementation was having a way to specify which boards should post to which channels. My very first implementation had a hardcoded channel hook where it would post any and all card movements. Not great.

Second implementation actually had a assocs.json file which would define which board belongs to which channel. This file had to be added to .gitignore which means I needed a sample for its structure, an assocs.json.sample. Configuration step that I also need to explain in the README and… ugh. 🥱

const list = require('./data/project-url-to-slack-webhook-url');

More importantly it meant that if anyone wanted to add a subscription to a board to a channel — they had to ask me directly.

No bueno, as I have other stuff to do during the day than updating .json files (that only I can access) and deploying lambdas. 💃

There must be a better way.

Now this is the point when any seniõr dev will tell you:

Hey, but like, how often does this actually happen? Isn’t it enough to have it hard-coded and take the hit every once in a while? Also you have bed hair — you doing alright buddy?

To which i say:

Nein. My hair is my glory and we’re in a pandemic so give me a break. Also I love to learn and I feel bad about all the stupid setup you would need to do just to get this going, so we’re automating it. End of story!

So I guess we’re learning about more stuff; stuff like… Slack Commands, Interactive Messages and AWS DynamoDB.

Action Plan (third time) 👩‍👧‍👧

Ok so I wasn’t sure what it was going to look like in the final version but what I wanted to do is get the information about the channel <-> board relationship in some sort of storage.

A board can post to n channels, and we store many boards

So for each incoming request we want to look up the associated slack channels and send them a message.

A nice UX friendly mechanism would enable people to create these associations themselves. It would also be great if they had a way to delete them. Creating and deleting Board to Channel associations as they please, when they please.

So now there are a couple of things that the bot should do:

1. Receive webhooks from GitHub and forward them to Slack
2. Store associations for boards to channels (save / delete them)

What’s in a name?

First challenge is figuring out the real id of the project boards you want to reference. When you get a incoming webhook from GitHub you will get its content_url which is it’s absolute id in the GitHub infra. So for example:

https://github.com/github/roadmap/projects/1

corresponds to something like

https://api.github.com/projects/115684

Using the latter one can actually make calls to the GitHub API and get information about the board like name, etc.

It’s not very clear how one could get this information from actual board as there’s no button or such in the interface. What I used to do with the hard-coded solution was to inspect the GitHub payloads to find it.

“Do they speak english in WHAT?” 🕺

I wanted a seamless experience for my bot. None of this inspecting the payload nonsense. You could just ask it for repositories and it would give you options. One of the ways you can interact with a bot is making it respond to a Slash Command. In my experience it’s idiomatic to use slash commands to talk to this kind of bot so I made one.

Configuring a Slash Command in Slack

It’s quite easy to set up: chose a command and an API endpoint to call when it’s invoked in the chat. Then slack will call it up and expect a message object back.

I created a new endpoint for this /github/projects in my serverless.yml and got it to reply with a list of boards from the organisation. You can obtain the list of projects using GET /orgs/{org}/projects — see here

Additionally I setup an optional parameter which is the repository name. So you can list either the org level boards or repository level boards.

To get access to the text after the slash command, Slack will supply you with a text parameter in it’s request payload. More on this payload coming up.

Putting it together, we have the slack command + the github call in one gif

Listing projects on both org and repo level in response to a Slash Command

The Slack Payload

So the API has some quirks on this payload. It’s actually a POST call with a query string body that looks like so:

token=xxx
&team_id=yyy
&team_domain=idagio
&channel_id=G01D058PUUE
&channel_name=privategroup
&user_id=U0EK5NB47
&user_name=vladgoran
&command=%2Flist-projects
&text=idagio-android
&api_app_id=A01CXGDD4T0
&response_url=https%3A%2F%2Fhooks.slack.com%2Fcommands%2FT02LM9RJJ%2F1476885511601%2FfxmGFffJ887dqIzcCJrCTgw2
&trigger_id=1449515498503.2701331630.8801b0cf826b9ce42f9096e0f70f0124

I’ve introduced the line breaks so it’s easier to digest. But there’s some fun stuff in there.

First of all you can notice the text key, that corresponds to the repo name in our case. From the slack docs:

This is the part of the Slash Command after the command itself, and it can contain absolutely anything that the user might decide to type. It is common to use this text parameter to provide extra context for the command.

You also see the command , the channel_id and the user_id but the thing that captured my interest was the response_url

A temporary webhook URL that you can use to generate messages responses.

Which links to Handling user interaction in your Slack apps and I feel like I’m going down a rabbit hole, but I keep my eye on the prize for now: let’s make the thing subscribe to channels. 🐇

For this we set up another Slash Command and voilá:

Copy — paste subscribe

Of course this is made up of lies. We didn’t setup the database yet. Nor can we actually post to anything except this channel. But seems like progress.

I guess we need a database afterall

Ah yes, the dream has died. This was supposed to be a stateless jewel but the cruel gods of persistence were just taking a nap. 💎⚡️

What can we use? AWS DynamoDB of course

Amazon DynamoDB is a fully managed proprietary NoSQL database service that supports key-value and document data structures[2] and is offered by Amazon.com as part of the Amazon Web Services portfolio.[3] DynamoDB exposes a similar data model to and derives its name from Dynamo, but has a different underlying implementation.

Does it cost a lot of money?

Not really. In the Free Tier you get:

25 GB of data storage

2.5 million stream read requests from DynamoDB Streams

1 GB of data transfer out, aggregated across AWS services

Far beyond what my little bot will use with our Team.

Alright. We’re in the cool NoSQL crowd now. Here’s your basic serverless.yml database setup.

This plus a little bit of magic + aws sdk we are able to list, add, get and delete associations. Also we can create filters which is what we will need for the GitHub webhooks to run.

const store = new DynamoTable(process.env.ASSOC_TABLE_NAME);

An unnecessary detour ⚠️

While I was trying to figure how to add messages to any Slack channel I thought that I needed to install the app. Not so! And a big learning for me.

Unless you want to distribute your app you should NOT bother. You can just take oauth that Slack App Dashboard gives you and put it in an env variable. Installations will not generate new oauth tokens, just give you a secure way to obtain the one you already have.

Hours of my life wasted on oAuth

The reason why it was time wasted in this case is because if you wanted to distribute this slack app you would need to specify a ton of sensitive information that leads right into your org’s GitHub account.

I don’t want to be responsible for something that powerful and I don’t think people should trust me with it. That’s why I would recommend you install it in your own infrastructure. If you do want to distribute your app here’s what I came up with for the installation code. It works but in the end I ditched it.

Again, all you need in order to post is the oauth token. You send this in an Authorization: token <OAUTH_TOKEN> header and you can do whatever you set your app scope to.

Shoulders of giants

I should mention that a lot of the code I use to deal with Slack and DynamoDB comes from here: https://github.com/johnagan/serverless-slack-app

It is an excellent place to start understanding what the slack API can give you in terms of interactivity. It’s built as a Pub/Sub so you can do things like:

slack.on('slash_command', async (msg, bot) => {
  const { channel_id: channelId, text: repo, command } = msg;
  if (command === '/list-projects') {
    await onListProjects(channelId, repo, bot);
  }
});

Excellent!

I’ve changed it in a few ways

1. I’ve removed all the mentions of the app install code and the complicated logic for looking up oauth tokens based on the team id
2. I’ve streamlined the variables needed for initializing it and moved them into the constructor so i can easily use env varibles
3. I’ve upgraded the request logic to use the excellent got library
4. I’ve made it work with the latest Slack API (Authorization header)
5. I’ve introduced new methods (delete, list) in the DyanmoDB wrapper
6. I’ve quietly deleted the JSDoc comments as I don’t like them 💨

Big thank you to John Agan for putting it together! 🙇‍♂️

More interactivity!

Here we go back to the response_url You can use it to respond privately to a command. But it has a couple of nifty tricks as well:

1. If your app received an interaction payload after an interactive component was used inside of a message, you can use response_url to update that source message.
2. You can also delete a source message of an interaction entirely using response_url

Using the "replace_original": "true" parameter on your response to Slack you can actually have an entire conversation with the user. In my case I wanted to send the list of projects + a subscribe button that would turn into an unsubscribe button once the opration was complete.

See the following flowing flow diagram to understand it better.

Cool interactions!

So in the end this is what you get this kind of interaction.

You can notice here the bot also posts into the channel that a new subscription was added

Time for feedback ⏰

With this in place I’ve shared it with the rest of the IDAGIO Team to get some feedback. One of the first things we tweaked was the way the activity notifications looked. I expect more changes in the future!

Latest look of the notifications, with a similar look to the official integration

If you use it and want to give me feedback please do it give do it on the Issues in the github repo. I appreciate it!

Fin.

This was the last part of 3. Part 1. Part 2.

Lessons from Part 3:

1. Don’t underestimate your left-overs (it was another week that I spent on Part 3)
2. OAuth installation process for Slack apps is only necessary if you plan to distribute your app
3. You can build quite rich interactions on Slack using replace_orignal and delete_original
4. There’s nothing to be afraid of with DynamoDB + Serverless

Thank you for coming with me on this journey! 🙇‍♂️