Ticket Triaging with Natural Language Processing

By, Sean Sodha

What is Natural Language Processing?

Natural Language Processing (NLP) is a massive space within artificial intelligence (AI), which enterprises are integrating into their existing platforms more each day. As petabytes of textual data become available each day, companies can leverage NLP to retrieve deeper insights. Aspects such as entities, sentiment, emotion, and keywords can be extracted from textual data and enterprises can leverage this information to pivot, understand customer sentiment, and improve internal efficiency.

Watson Natural Language Understanding (NLU) and Watson Natural Language Classifier (NLC) are cutting-edge NLP technologies that provide deep insight into textual data.

Watson NLU provides insight such as entities, emotion, keywords, sentiment, and categories, while Watson NLC allows users to train a classification model in under 15 minutes and classify text. Whether the textual data is for a finance, healthcare, or real-estate use case, both of these tools can be trained on a specific domain to deliver more accurate results.

Ticket Triaging

Say for instance we are a medium-sized consumer bank in the U.S. called Bank Blue. With continued growth in mobile banking over the past few years, we’ve seen an uptick in the number of customer support tickets coming into our online portal each day. For example, we’ll receive tickets across a wide range of categories including incorrect user login ID, loan modifications, questions about existing debt, fraudulent payments, or customers even threatening legal action. A common challenge we face at Bank Blue is sorting through all of these support tickets as we receive hundreds of complaints coming into our system each day. We also have to take into account the level of severity from each support ticket, considering some tickets are more urgent than others.

What’s the logical next step and how can we best go about sorting through all of the support tickets in the queue? As customer service agents working for Bank Blue, we want to not only prioritize support tickets, but also place them into buckets based on category and level of severity. To help us accomplish our goal, we came across the Ticket Triaging system.

Ticket Triaging is a method that incorporates AI to sort customer support tickets by category and sentiment — in this case, customer anger level. To deploy our solution, we are using Watson NLU and NLC to handle this process.

1) Creating Watson NLU and NLC instances

___________________________________________________________________

import json

import pandas as pd

from watson_developer_cloud import NaturalLanguageUnderstandingV1

from watson_developer_cloud.natural_language_understanding_v1 import Features, EntitiesOptions, CategoriesOptions, ConceptsOptions, EmotionOptions, KeywordsOptions, MetadataOptions, RelationsOptions, SentimentOptions

natural_language_understanding = NaturalLanguageUnderstandingV1(

version=’2019–07–16',

iam_apikey=’NLU API KEY’

)

from watson_developer_cloud import NaturalLanguageClassifierV1

nlc = NaturalLanguageClassifierV1(

iam_apikey=’NLC API KEY’,

url=’https://gateway.watsonplatform.net/natural-language-classifier/api'

)

___________________________________________________________________

The first step is to import several libraries that we will need for our use case, as well as creating our instance of Watson NLU and NLC. In the iam_apikey field, we would want to input our API key for both services. We would retrieve this from our IBM Cloud Account.

2) Training our Watson NLC Model to Classify Texts

___________________________________________________________________

with open(‘Bank_Support_Tickets.csv’, ‘rb’) as training_data:

classifier = nlc.create_classifier(

training_data=training_data,

metadata=’{“name”: “Support Ticket Classification”,”language”: “en”}’

).get_result()

print(json.dumps(classifier, indent=2))

___________________________________________________________________

With Watson NLC, we can train a classifier model and create a solution in 15 minutes. In our example, say we only want to classify the customer support tickets that fall under the “Incorrect information on a credit report” and “Taking/threatening an illegal action” classifications. I have set up a .csv file that will represent our training data. Below is a snapshot of how the file is set up:

In our example, we took 150 unique customer support tickets for each classifier. Since we are only focusing on training two classifiers, our csv file has about 300 total entries. Typically, at IBM, we recommend you have about 5–10 entries per classifier when training the NLC model, but for this example, we wanted to make sure that our model had plenty of examples to accurately distinguish the two ticket classifiers.

3) Implementing Watson NLC to Classify and Watson NLU to Sort based on ‘anger score’

___________________________________________________________________

df = pd.read_csv(‘Support Tickets 07_20_2019.csv’)

length = len(df)

legal_action = list()

legal_action_anger = list()

incorrect_info = list()

incorrect_info_anger = list()

for i in range(length):

classification = nlc.classify(nlc_model_id, df.iloc[i,0]).get_result() #classifies which bucket the text should go into

classifier = classification[‘classes’][0][‘class_name’] #extract classifier from JSON output

extracted_text = classification[‘text’] #extract text from JSON output

#print(df.iloc[0,0])

NLU = natural_language_understanding.analyze(

text = df.iloc[i,0],

features = Features(emotion=EmotionOptions())).get_result()

anger_rating = NLU[‘emotion’][‘document’][‘emotion’][‘anger’]

#print(anger_rating)

if classifier == ‘Incorrect information on credit report’:

incorrect_info.append(extracted_text)

incorrect_info_anger.append(anger_rating)

else:

legal_action.append(extracted_text)

legal_action_anger.append(anger_rating)

___________________________________________________________________

This is where the magic begins to happen. Here, we have a file called Support Tickets 07_20_2019.csv which represents the number of support tickets received on a given day. For example, on July 20th 2019, Bank Blue received 60 customer support tickets. We used our newly trained NLC classifier model to sort each support ticket into our two trained classification buckets.

After sorting the 60 support tickets into two designated buckets, what’s next? How do we prioritize which tickets should be responded to first?

We can call Watson NLU’s Emotion API to retrieve the anger score from each ticket. Watson NLU’s Emotion API returns a sentiment score for sadness, joy, fear, disgust, and anger. In our example, as a customer service agent working for Bank Blue, we think it’s logical to prioritize tickets based on anger and sort each ticket based on its corresponding anger score. To find the ticket with the highest anger score, we initialized 4 lists. One list for holding comments related to incorrect information (named incorrect_info) and one list to hold the anger scores for each comment about incorrect information (named incorrect_info_anger). Similarly, we initialized a list for holding comments related to legal action (called legal_action) and one list to hold the anger scores for each comment about taking legal action (called legal_action_anger).

We placed this ticket triaging process into a loop, which iterates through the entire .csv file and correctly sorts each ticket. By the end of the process, we have 4 completed lists. Two of which hold the ticket-related content and the other two hold the anger scores associated with each comment per classification.

4) Extracting the ticket with the ‘highest anger score’

___________________________________________________________________

max_anger_value = max(incorrect_info_anger)

print(max_anger_value)

max_index = incorrect_info_anger.index(max_anger_value)

print(max_index)

angriest_comment = incorrect_info[max_index]

print(angriest_comment)

___________________________________________________________________

From the first print statement, we can see that the highest anger value that was measured was 0.824135, which means that the out-of-the-box emotion model within NLU was 82% confident that anger was detected. The customer support ticket below was given the highest anger score in our model:

“THERE ARE SEVERAL CREDIT INQUIRES XXXX, XXXX BANK XXXX, XXXX, XXXX XXXX, XXXX XXXX, XXXX, AND XXXX INQUIRES ARE ON MY CREDIT REPORT THAT HAVE BEEN DISPUTED SOME HAVE BEEN REMOVED BUT SOME REMAIN THAT DID NOT HAVE AUTHORIZATION TO PULL MY CREDIT REPORTS I REQUESTED VALIDATION TO SHOW WERE I AUTHORIZED THESES CREDIT INQUIRES AND IMPROPER RESPONSE WAS SENT BACK NOTHING SHOWING WERE I SIGNED TO HAVE MY CREDIT PROFILE PULLED”

It is worth noting that the ticket with the highest anger score was originally written in all caps. Our Emotion API is able to detect the difference between uppercase and lowercase characters and therefore, it will score the text differently. Given that the ticket above inherently depicts an angry tone, the anger score was increased since the ticket was submitted in all caps.

For simplicity’s sake, this use case simply analyzed two types of classifications and sorted each customer support ticket based on the anger score. When creating a fully rolled out ticket triaging system, using Watson NLC and NLU, we recommend creating additional classifications and API calls to surface different concepts, categories, sentiment, or emotion, to ultimately help you sort through and prioritize which tickets to handle first, based on your specific use case.

Get started today with Watson Natural Language Understanding and Watson Natural Language Classifier for free.