NATH for Redmine: NLP-based tickets classification

Automatic issues analyzer

Hi everyone, I am Damien Gilles, engineer for the R&D team at alter way and developer of the NATH (NLP Automated Ticket Handler) for Redmine. In this article, I will explain to you how we ended up using artificial intelligence to better qualify our customers issues through NATH Redmine implementation, and why we went from there to an experimental Gitlab implementation.

No business pain, no gain

When first discussing with the support team, we had a strong belief that a conversational UI along with a chatbot would make their life easier, by separating the customers issues expression with the actual tracking and resolving of the issue.

In fact, what appeared to be a painful task was to qualify the customer demand within the triage phase, first by distinguishing what was an anomaly or an evolution, then by determining its criticality and where it should stand in the support team backlog. So, we considered focusing on the current Redmine workflow rather than providing a yet to be developed interface.

Redmine and custom fields

Some examples of fields we fill for each ticket may include:

• tracker: the tracker in which the ticket will appear
• status: is the ticket open, close, ongoing…
• urgency: how fast a ticket must be solved
• importance: how big are the implications
• estimated time: how long will it take to solve

This classification helps us stay on track, but it also comes with multiple drawbacks:

• It’s time-consuming
• Clients may abuse fields such as “urgency”
• Agents don’t like to specify the estimated time
• It’s repetitive

Classification using AI

Thankfully, all those drawbacks can be overcome using AI. Indeed, all those fields can be deduced accurately using natural language understanding (NLU) or regression models, equivalent to the ones used for sentiment analysis. This approach allows us to save human resources and treat the tickets faster by automatically classifying them at their creation.

With modern NLU services like LUIS, Dialogflow or wit.ai you can create models for fields such as “importance” without any knowledge of machine learning. Our plugin’s flexible endpoint configuration allows us, to easily bind those service to Redmine. If you wish to explore this path we offer a quick walk-through to bind Redmine and LUIS.

However, it may be harder to find existing service to determine continuous fields, like the “estimated time”. For this one we chose a more classical approach: we built a linear classifier and published it on a local server. If you are interested in how to build such a classifier, this tutorial is a good starting point. We also provide a way to run our own model here. This simple approach can give surprisingly accurate results as illustrated by the following graph, where we compared human and machine results on 82 tickets:

After reviewing the few anomalies, we’ve concluded that they were mostly human errors and that the model’s prediction may be preferable to the human ones.

Redmine integration

We have managed to easily create some powerful IA but we still have to integrate them in our workflow and here comes NATH Redmine! Redmine does not offer an interface to configure webhooks but only provides hooks for plugins. So we’ve decided to build our own. The plugin allows you to call multiple webhooks on a ticket’s creation and to modify the value of a Redmine native or custom field.

Concretely, if you wish to automatize the custom field “estimated time” like in the example above, all you have to do is:

1. Go to the plugin configuration page
2. Provide the endpoint address for your service in the text box for “estimated time”
3. Specify where the information is in the JSON response in “response path”

This way you can bind your Redmine issue field with any web service with an HTTP GET endpoint returning JSON within minutes.

Toward a Gitlab implementation

As you may well know, Gitlab offers a rather complete tracking features, including binding issues status to commits labels. Following this trend, we considered implementing our algorithms within it, also in the idea of opening the platform to customers rather than only to developers.

While providing an easy to use webhooks system to trigger actions on the platform, the issue feature is lacking the possibility to implement custom fields… at all. Rather than implementing a beta feature out of our scope, we used labels to provide about the same level of functionality as our Redmine plugin does.

Follow up

We will Open Source the Gitlab plugin in the coming weeks, before proceeding with more advanced issues automation, such as automated pointers to existing solutions and in the best of our hopes, a first step toward automated resolution.

Use and Contribute

Meanwhile, if like us you wish to increase your reactivity when receiving tickets and likes to be ahead of time, feel free to add the plugin to your Redmine. The plugin is licensed under MIT and can be activated one project at a time, without disturbing your current workflow.

There is still room for improvement, so if you have any proposals be sure to share them with us on gitlab.

Thanks to Jonathan Rivalan for making this project possible.

Keep innovating!