Architecting a Slackbot for Developers

Hello from the IBM Extreme Blue Lab in Research Triangle Park. Extreme Blue is IBM’s global internship program focused on incubating new and emerging technologies.

Our team worked with IBM’s Whitewater Group this summer to build an engagement bot that promotes the use of agile/modern coding practices at IBM. MIYAGI (so named after one of our all-time favorite movies) works with developers to promote continuous integration and code review practices.

Why is this important?

As businesses adopt more agile processes to quickly deliver products that delight customers, developers are forced to adapt to shorter development cycles, making the waterfall development flow less viable. Teams need modern development practices to maintain code quality while responding to rapidly changing user needs. Learning all the new tools and processes needed to adopt modern practices is challenging.

Enter MIYAGI.

MIYAGI is an educational Slackbot that helps developers adopt modern coding practices like continuous integration and code review. It does this by tracking user education over time, reporting on key developer metrics, and connecting developers with code reviewers that broaden the developer’s knowledge. This enables development teams to grow from each other’s experience while reducing the number of bugs introduced into production.

So, how did we build Miyagi?

Languages, Microframeworks, Toolkits

• Working knowledge of Python, Node.JS, MongoDB, Javascript
  - IBM Bluemix (sign up for Bluemix here)
  - BotKit
  - Flask
  - IBM DevOps Insights API
  - GitHub API
  - Slack API

Architecture

Miyagi consists primarily of two separate services: the bot server and the analysis server. The bot server, written in Node.js, listens for and handles events, serving as the backbone of our user and test moderator interactions; the analysis server, written in Python, handles large scale data analysis and reporting. We wanted to separate the Miyagi’s user-facing components from those that handle analysis mostly for two reasons: performance and simplicity; Python has superior library support for natural language processing and generally demonstrates better performance than JavaScript when processing data. Unlike Python, JavaScript is asynchronous, so we opted for Node.js to handle the bulk of our requests and responses.

Analysis Server

The Analyzer modules (written in Python and using Flask) are responsible for generating different types of analysis to craft messages. Miyagi has two main Analyzer modules: The Quantitative Analyzer, which calculates or otherwise gathers quantitative metrics based on metadata, and the Team Qualitative Analyzer, which generates metrics from textual content and group interactions using natural language processing methods. From this Miyagi can do things like:

- Report on code quality metrics (i.e. Average Lines of Code, Code Coverage, Code Complexity, Bugs Introduced, Rubberstamping, Repo Health, and more!)

- Make recommendations for code reviewers based on their experience and skillset (i.e. selecting code reviewers who have deep technical knowledge in a specific field and code reviewers who are known for giving high-quality reviews)

External APIs

If any additional external data is required for analysis, it is retrieved from the External Data APIs. Miyagi requires additional pull request data for full analysis, and retrieves this through the Github API. Miyagi also gathers more complete analysis from IBM’s internal DevopsInsights API.

Database

For our persistent data storage, we chose to spin up a MongoDB instance. Our database stores user profiles as well as data for describing pull requests, repositories, user interactions, and any analyses on this data. Each of our collections stores data in a consistent model, so MongoDB was not necessarily the wisest choice of database framework for our project. Frankly, the reason we chose to use Mongo over other frameworks was because everyone on the team had used it before and we wanted to start building Miyagi as quickly as possible. If we had more time, we would have selected a database more optimized for querying than Mongo (e.g. ElasticSearch).

Bot Server

As mentioned earlier, the Bot Server handles all of Miyagi’s events and user interactions. Here are a few characteristics that we think make the Bot Server interesting:

Modular Conversations

Not one of us on the team had experience with conversational UI design, so rather than flying completely blind and making our UI decisions solely by intuition, we built Miyagi in a way that allowed us to test the efficacy of different conversational flows without rewriting the entire application.

To implement modular conversations, we store certain pertinent information on a per-user basis: statistics (how many good pull requests the user has made, when was her last one, etc.) and conversation configuration (the frequency at which the user receives certain messages, the contents of these messages, etc.). We pass this data into a function which creates a conversation object each of whose properties is only defined if a user’s configuration indicates that she should receive a message in response to a particular event.

Conversation modularity simplifies and expedites the process of testing Miyagi interactions with different content, attachments, and frequency to determine the optimal means of affecting our users’ behavior. In the future, Miyagi will incorporate Machine Learning models that fine tune its conversation flows to specific users, rather than entire test groups.

Built in A/B Testing Framework

We built Miyagi with an interface that allows test moderators to easily create new testing cohorts and assign users to them. Prior to testing on a new cohort, test moderators must define the configuration parameters for each group within the cohort as well as the frequency at which each group appears in the cohort. Then, moderators must decide which organizations (currently, a user’s organization is defined as his primary slack channel) will be the constituents of the new cohort. As part of our authorization flow, new users assign themselves to a testing cohort based on which organization they belong to within IBM, whereupon they are randomly assigned to a test group (A/B/C/…) defined by a test moderator. Test moderators can then access test results by cohort from the analysis server.

How the Bot Works

Response Files

Three types of files exists that handle responses to messages.

· response.js: Gathers the user data from the event, including their respective flow configuration file. It loads the data from that configuration and then sends a response message to the event.

· slack-response.js: A set of responses to interactive message events initiated by user interactions with Slack buttons.

· <event>-response.js : A module that provides an analysis of the event for the user.

Reactions to events

Miyagi sends messages based on the users’ actions. Any time the user executes an action on GitHub, such as submitting a pull request, the notification is sent to the /payload route on the Miyagi server.

The code parses the req.body to read the type of event that occurs, such as a pull request, and loads its respective module (in this case, pull-request-response.js).

Currently, Miyagi only examines pull request events when a brand new PR is created under `pull-request-response.js`. It will send a positive message with reviewer suggestions when the user’s PR is under 250 LOC, a negative message with reviewer suggestions when the PR is between 250 and 400 LOC, and solely a negative message if the PR is above 400 LOC.

Feedback

Miyagi asks to see if a message it sends is helpful and the user’s response (yes or no) is stored into the database. If the user responds with no, Miyagi asks for feedback in the form of a Survey Monkey link.

The survey asks for a user’s first and last name, their email, and any feedback they want to give about Miyagi.

Messaging

Miyagi utilizes Botkit and Slack API to send messages. A bot is spawned with Miyagi’s Bot token and sends messages on the Slack API route https://slack.com/api/chat.postMessage to the user. This message is sent through Botkit, as it contains a function to send the POST request using the request module (bot.api.chat.postMessage). This function streamlines the process of sending messages and allows us to send interactive buttons to the user, which are important to make interacting with the bot as easy as possible.

Any response to a button is sent to slack-response.js. Based on the message that was previously sent, the server processes the message to send a response, ask for feedback, and more.

At the end of each week, the server sends all members on the team a message that contains a link to a metrics dashboard.

Final Thoughts

This was a three month long project where our team ideated, designed and validated our ideas through user interviews and testing. Bots are all the rage right now so we hope this helps you better understand how you can build a helpful and extensible Slackbot.

Have fun!

Authored by Neil Davis, Kathryn Hite, Harris Lummis, Connie Yee