5 steps to improve intent recognition
Take a look at the following conversation:
User says: “My phone just died”
Chatbot says: “I’m sorry for your loss. You can end the contract of the deceased by e-mailing us. We’ll do our best to support you during this time”
Unfortunately, this has been a real conversation between our virtual assistant and one of our customers. We found the issue, corrected it immediately and thought it was time to take a closer look at our process of intent recognition.
Our team has been working on an IVR (interactive voice response system) and chatbots for several years now. As our team grew, so did the number of intents. Some new intents worked out great from the start and others needed more finetuning before doing well. During this process, we had discussions, breakdowns, deep-dives, failures, and successes.
All just to get the recognition right.
So where do I start?
When you set up a chatbot or IVR, it’s always hard to find a good starting point. What intents do your customers have and how can you answer their questions properly? I want to take a closer look at the 5-step process of setting up and mapping intents.
Any intent recognition journey starts with a big pile of user questions. You can gather these in many ways: user sessions, interviews, forms prior to live chats and many more. When you’ve collected your user questions, it’s time to set up intent recognition.
You will find that these user questions vary from smaller issues like “I lost my keys” to bigger and more recurring questions, such as “I have question about my latest invoice” or “Can I order product X?”… Can’t think of a good structure or don’t know where to start? Don’t worry, you’re not alone: virtually every conversation designer goes through this in the early stages of creating a chatbot.
1. Structuring the data
The first step is to get an overview of the largest and broadest subjects under which the user questions can be categorized, such as “Invoices” and “Ordering”.
Of course, you can’t plot everything in these categories, that’s okay. Keep in mind that you’re not designing your intents for the outliers, at least not at the outset of your intent recognition process. You don’t want to let these user questions go to waste; you can save these questions by filing them under a Fallback or Out of scope category. You don’t have to provide an answer in that case, but you can still recognize the intent. A common trick is to refer to the customer service or escalate the chat to an agent.
2. Set up clear guidelines.
Secondly, set up a definition for each group of user questions. What is the trait in this kind of question that bonds them together? How do you define the underlying pattern of questions?
If you’re the only one setting up your virtual assistant and NLU model, it’s easy to maintain the logic. Still, it’s good to define what type of question is plotted. Because you’re still dealing with big chunks, you can get away with definitions like “Has question about invoice” and “Wants to order”. After all, these are still the broad strokes of intent recognition. It’s important to keep a clear list of definitions of each question. This will help you when your assistant can answer and recognize more questions.
At this point, you can set up an answer with an explicit confirmation to check whether the intent has been recognized correctly. For example: “Would you like to order something?”
3. Refine, refine, refine.
The third step is the refinement. Those big chunks often contain valuable information about what the user wants, what they’ve tried, and in some cases, there might even be an underlying need they don’t specify. Some examples where you can refine your recognition:
1. “I have a question about my invoice, it’s too high”
2. “I have canceled my subscription, yet I still got an invoice”
3. “I need to pay the invoice, but I was wondering if I can pay it later”
4. “Invoice”
All four of these examples are about invoices and all are questions about the invoice. Yet, you can’t answer them all with the same response. This is when you start refining. Number 4 remains in the “Has question about invoice” category, since you can’t find a specific intent in that question. If your chatbot technology allows it, you can offer a user several options in cases like these. For instance, “You have a question about your invoice. Do you want an explanation of the costs, do you want to pay the invoice, or is it about something else?” This way, you can steer the conversation in a direction and proceed to a solution.
For the other three user questions, you could create a new intent, with a unique dialog for each question. In doing so, you can cater to their specific needs directly without having them to confirm their question. When you define these new intents, remember to update your list of definitions. This way, you can always go back and look at the thought process and definition of each unique intent. All done? Congratulations: you’ve set up your intents!
4. Working as a team
Everyone has a unique speech pattern and a unique way of processing language. As Terence said in Ancient times, “there are as many opinions as there are men,” and this still holds true today. When you’re working in a team, you’ll find that everyone reads a sentence differently and gives it a different meaning. In most cases, you’ll get the same result, but there are always edge cases. For those edge cases, it’s important to collaborate and discuss what goes where. A definition of an intent might not be clear enough, and you might end up creating a new intent.
To keep everyone aligned, and to keep your NLU model happy and healthy, you can set up a biweekly conversation review. During these review sessions, you’ll discuss any edge cases and go through real chatbot conversations together to give feedback on the recognition. Through conversation review, you can keep a closer eye on your chatbot’s performance and discuss this with your team.
5. Testing will make your hard work shine
The final step entails maintaining the sanity of your NLU model and checking if your setup still works.
This step is incredibly important to keep your chatbot relevant for your users. During sanity testing, you check up on every intent and basically start all over again. Don’t worry, you won’t have to start from scratch: this time around, you’ll map intents in a more confined setup. Instead of the broader questions, you start looking into the intents you defined. Does the chatbot match questions correctly to the intent? Are there false positives? Are there any unmatched questions? Do customers barge in with another intent you might not have answered? Do this regularly (and be critical!), and your digital assistant will stay relevant.
As companies change, so do users and their questions. The patterns you found 2 years ago and the intents you created might have changed or be gone entirely in the meantime. Make sure to do a check-up of your classification model once every year or two. We’ve found that a good way to do this is to export the sentences you labeled and split them between the colleagues. Everyone receives the set of definitions and relabels each question in the export. After every question is labeled again, check the outcome and discuss any differences, edge cases, or new patterns you might have found. It can be quite an undertaking, but it helps you and provides your users with the best possible experience.
If this takes too much time or if you feel like you can work smarter, you can look into self-testing. Some NLU models provide that in the solution itself. If yours does not, you can look into solutions like qbox.ai. Self-testing shows the patterns present in your NLU model and how you can subdue certain biases or over-trained intents.
Maintaining a healthy NLU model and a relevant virtual assistant is an ongoing process. There is no right or wrong in doing so. The only wrong way to go about it is to not do it at all. Always keep in mind that the environment you’re working in is always changing and that your chatbot needs close attention to give the best performance.
