Failing Gracefully with Rasa NLU

Chatbot fallback strategies and how to implement them

Since writing this tutorial we’ve released a complete open-source chatbot platform — built on Rasa NLU — called Articulate. Be sure to check it out!

Introducing Articulate

Getting Started

It’s inevitable: You’ve created the perfect chatbot for ordering pizza, but Joe Joker or Sally Smart Pants want to toy with it’s young, fragile, artificial mind.

“Hello! What kind of pizza can I prepare fresh for you today?” your bot says.

“Oh sweet pizza bot, I’d like to [Insert NSFW pizza related content]”, they reply.

Unless you’ve prepared for this occasion (and you should) you are likely going to get a pizza order with some very inappropriate toppings or be asked to deliver to inappropriate locations because if they can, they will.

A user toying with you bot is not the only request that could lead to failure. It could also be that your users desire your bot to be more capable than it is. How you handle that knowledge gap is very important. This conversation comes up very frequently on the Rasa NLU Gitter. Rasa NLU is an open source tool for Natural Language Understanding. Together with Rasa Core they provide all of tools needed to build any kind of intelligent agent. But neither of them provide an out of the box solution for handling fallbacks.

Fallback — An official alternative plan that may be used in an emergency. A reduction or retreat. In the chatbot space: an alternative action used when the user’s intent falls outside of the intents handle by the agent.

Why is this a problem?

Ignoring for a moment the societal desire to break chatbots; the problem really resides in the machine learning methods most commonly applied to natural language understanding. Most of the machine learning methods commonly applied to NLU return one of the labels that you provided during training regardless of its confidence. This means that no matter what a user asks your bot, no matter how badly it is misspelled, even if it is lorem ipsum, it will still get an intent classification.

What is the Solution?

In this tutorial we’ll work through three different methods of handling these sorts of unexpected inputs from the user. We will implement them in this order:

1. Design to minimize fallbacks — We will start by adding in more intents.
2. Collect the Garbage — We will train a special intent to collect the garbage.
3. Handle Confusion with Confidence — We will add logic to handle low confidence Classifications.

Let’s Get Started

All of this is necessary when building a chatbot and it is definitely true for intent classification in Rasa NLU. To give you an example of what I mean let’s spin up a bot and try out a few examples. The bot that we are going to interact with was the one we trained in Part 1 of my Rasa NLU tutorials. If you want to follow along with that click on the link below.

Build A Chatbot with Rasa NLU — Part One

The first thing we need to do is bring up the Docker containers and train the original examples.

☞ Open a terminal or command prompt

First, open a terminal or command prompt and enter the below commands.

git clone https://github.com/samtecspg/rasa-tutorials.git
cd rasa-tutorials
sudo docker-compose up

Docker is going to download the necessary components and start them. Once that’s all finished you should be able to do the below.

☞ Click this link to test that Rasa NLU is working

Or copy and paste this link into your browser. http://localhost:5000 if you get a message like the below then everything is working.

hello from Rasa NLU: 0.10.0a2

Now we’re going to use the training data generated in the first tutorial to train our bot.

☞ Execute the command below

Make sure you are still in the root of the rasa-tutorials folder when executing this command or you may have to change the path to the training data. You’ll need to do this in a new terminal or command prompt window, NOT the one that you used to start Docker.

curl -X POST -d "@./rasa-tutorial-training-data.json" "localhost:5000/train?name=rasaTutorialBot"

When you do this, there may not be an immediate response, but after some time you should see a line similar to this:

{"info": "new model trained: rasaTutorialBot"}

And that means that the model has finished training and we can query Rasa to classify intents and entities. So let’s give it a try.

☞ Execute the command below

Feel free to experiment with the query a bit and test what works and what doesn’t.

curl --request POST \
  --url http://localhost:5000/parse \
  --header 'content-type: application/json' \
  --data '{
 "q": "What does Chuck Norris say about love?",
 "model": "rasaTutorialBot"
}'

This will also take a little bit of time as Rasa NLU loads the model into memory, but once it is loaded we shouldn’t have this delay again.

From the above I got the below response:

{
 "entities": [
  {
   "start": 33,
   "extractor": "ner_crf",
   "end": 37,
   "value": "love",
   "entity": "query"
  }
 ],
 "intent": {
  "confidence": 0.5027919536750802,
  "name": "chuckNorris"
 },
 "text": "What does Chuck Norris say about love?",
}

Which is perfect. The intent was classified correctly and love was extracted as an entity. But this bot isn’t very hard to break. Have you ever asked a device like Siri or Alexa to marry you? Well give it a try with this bot:

☞ Execute the command below

Test what works and what doesn’t.

curl --request POST \
  --url http://localhost:5000/parse \
  --header 'content-type: application/json' \
  --data '{
 "q": "Will you marry me?",
 "model": "rasaTutorialBot"
}'

And in this case the bot labels this as the advice intent. At least it didn’t think my advances were related to a Chuck Norris joke, but it certainly wasn’t what I wanted.

This leads us to the first tip: Design to minimize fallbacks. A fallback is like putting a fence at the edge of a cliff. It stops the user from walking over the edge, but it doesn’t necessarily make them happy. With proper design out-of-scope requests and their responses can be used to humanize your bot and delight your users. Don’t stop with just the intents needed to fulfill the bot’s main purpose. Keep adding intents until all common interactions have useful or entertaining responses.

Expanding the Scope of our bot

In saying this I mean that it will be common for users to interact with your bot in a playful way. It will also be common for your users to ask slightly off angle questions. Let’s adapt our training data such that it handles two more intents: marriage proposals, and requests for knock knock jokes.

☞ Execute the command below

You can use the Rasa NLU Trainer or your favorite text editor to see the new examples I have added to the training data. Once you’ve reviewed those, let’s re-train our bot.

curl -X POST -d "@./rasa-expand-training-data.json" "localhost:5000/train?name=rasaTutorialBotExpanded"

Once that is finished you will again see a message similar to the below. Note that we changed the name that we gave the model in the above command.

{"info": "new model trained: rasaTutorialBotExpanded"}

And we can try to ask our bot to marry us once again.

☞ Execute the command below

Feel free to experiment and test what works and what doesn’t.

curl --request POST \
  --url http://localhost:5000/parse \
  --header 'content-type: application/json' \
  --data '{
 "q": "Will you marry me?",
 "model": "rasaTutorialBotExpanded"
}'

And this time we get an intent that we can use to customize the response.

"intent": {
  "confidence": 0.6080578129169885,
  "name": "marriageProposal"
 }

So how do we use this? At this point in time we’re going to jump into Node-RED. Which should be running on your computer if you used the docker-compose file from the tutorial.

☞ Click this Link

Or copy and paste http://localhost:1880 into your browser. This will take you to your local Node-RED interface. Once you get there you should notice that there are already several nodes on the screen.

If you need an introduction to Node-RED check out the tutorial I wrote for just that purpose.

A Fun Introduction to Node-RED

Now, you can either follow along with Part 2 of the Rasa chatbot tutorial series or you can use the greyed out tab in Node-RED labeled Part 2 Solution. If you want to follow along with part 2 get it here:

Build a Chatbot with Rasa NLU — Part Two

The instruction below is how to enable the solution flow in Node-RED assuming you didn’t follow along with the above link for Part 2.

☞ Disable the Rasa Chatbot Starter Tab

To do this, double click on the Rasa Chatbot Starter tab. In the pane that opens up, directly below the name field, click the status toggle to disable the flow.

Image 1 — Disabling the Rasa Chatbot Starter flow

Now we need to do the opposite to the Part 2 Solution tab.

☞ Enable the Part 2 Solution Tab

To do this, double click on the tab. In the pane that opens up, directly below the name field, click the status toggle to enable the flow.

☞ Deploy the Changes

Once you’ve done the above, click on the the red Deploy button in the top right hand corner to make the changes live.

If all went well you should be able to execute this command:

curl --request POST \
  --url http://localhost:1880/chat \
  --header 'content-type: application/json' \
  --data '{
 "q": "What does Chuck Norris say about love?",
 "model": "rasaTutorialBot"
}'

Note that we are now sending the request to localhost:1880/chat, which is the endpoint from our Node-RED flow. When I made the above request the response I got was:

Chuck Norris doesn’t need love, Chuck Norris is love.

But right now if you tried to propose marriage or ask for a knock knock joke, you wouldn’t get a reply. That is because the bot that I implemented in the 3 part series didn’t handle out of scope request or fallback. So let’s modify the Node-RED flow

☞ Double Click the Intent Switch Node

Double clicking the node will open the edit pane. Once that is open add two new switch values for marriageProposal and knockKnock. Then click the red Done button.

Image 2 — Modifying the Intent Switch

These two new intents will be similar to the greet and goodbye intents that we are already handling. So let’s copy one of those and modify it as needed. You’ll need to complete the below steps. An image of what my flow looked like at the very end is below.

☞ Copy the Goodbye Intent Flow Twice

You can drag your mouse over the comment, link, and function nodes. Press Ctrl-C and then Ctrl-V to copy and paste a bank of nodes.

☞ Use the Link In an Out nodes

To keep the same flow structure I am using the link in and link out nodes. Once you have the nodes on the screen, double click them to select which link node to connect them to. For help see Image 3 below, specifically elements 3 and 4.

☞ Edit the comment nodes

We use comment nodes to help keep track of what each line does in the overall flow.

☞ Connect the function nodes to the HTTP Response node

This is the final task for setting up the flow. Next we’ll modify the function nodes.

Image 3 — Wiring new intents into the flow

Now we need to modify the function nodes, highlight as 5 and 6 above so that they generate responses applicable to their intents.

☞ Modify the Function Nodes

The function nodes contain an array of possible responses and a small bit of javascript to choose a random response. To edit them double click on them and copy/paste the below text or enter your own custom responses.

For the Marriage Proposal function node, here is what I ended up with:

var intentResponse = [
    "Sorry I'm taken",
    "That would be nice, but what will your parents think!",
    "I don't think so wierdo."
]
var randomInt = Math.floor(Math.random() * intentResponse.length)
    
msg.payload = intentResponse[randomInt]
return msg;

And for the Knock Knock joke function node here is what I have:

var intentResponse = [
    "Sorry I don't know any knock knock jokes. I'm all about Chuck Norris and giving advice, but not giving advice to Chuck Norris.",
    "Move along, no knock knock jokes here."
]
var randomInt = Math.floor(Math.random() * intentResponse.length)
    
msg.payload = intentResponse[randomInt]
return msg;

☞ Deploy the Changes

Once you’ve done the above, click on the red Deploy button in the top right hand corner to make the changes live.

and we can test our two new intents:

☞ Execute the command below

Feel free to explore a bit and test what works and what doesn’t.

curl --request POST \
  --url http://localhost:1880/chat \
  --header 'content-type: application/json' \
  --data '{
 "q": "Will you marry me?",
 "model": "rasaTutorialBotExpanded"
}'

and I got

That would be nice, but what will your parents think?

or

curl --request POST \
  --url http://localhost:1880/chat \
  --header 'content-type: application/json' \
  --data '{
 "q": "Knock Knock",
 "model": "rasaTutorialBotExpanded"
}'

Which gave me

Sorry I don’t know any knock knock jokes. I’m all about Chuck Norris and giving advice, but not giving advice to Chuck Norris.

Great! With those changes we’ve increased the scope of our chatbot to where there are a few more user utterances that won’t cause unexpected results. But what happen if the user types in something completely meaningless?

For example try:

curl --request POST \
  --url http://localhost:1880/chat \
  --header 'content-type: application/json' \
  --data '{
 "q": "abcdefghijklmnopqrstuvwxyz",
 "model": "rasaTutorialBotExpanded"
}'

In this case the bot just told me Goodbye, which isn’t that bad of a response, but isn’t quite right either.

Vector Representation — Inside of Rasa all user utterances are represented as an array of numbers. So Will you Marry me may look like [123, 73, 63456, 234]. The numerical representation of those words comes from some machine learning itself. When the user makes a request similar to that Rasa will use it’s model to determine how similar the two arrays are. But garbage input, misspellings, and words not included in the dictionary used will all be labeled 0. So if someone asked Wil U Mry MEEE it is likely to get classified incorrectly.

This is the second tip for failing gracefully: Collect the Garbage. The easiest way to handle garbage input is to train a fallback intent that contains garbage itself.

Training a fallback intent to collect the Garbage

☞ Add some garbage examples to the training data set

You can use the NLU Trainer to add your own examples or upload the rasa-fallback-training-data.json file from the cloned folder to see the few examples that I added.

☞ Execute the command below

Now that we’ve modified the training data again, we need to train the bot.

curl -X POST -d "@./rasa-fallback-training-data.json" "localhost:5000/train?name=rasaTutorialBotFallback"

Once it is finished we need to modify Node-RED one more time to accept the fallback intent.

☞ Add fallback intent to Node-RED

Look back at the earlier step by step if you need help, but you need to: edit the intent switch, copy and paste the goodbye intent, link it to the switch and HTTP Response node, and then modify its function node. If you used the provided training data the name of the new intent is fallback

Here is what my flow looks like with the fallback intent added in:

Image 4 — The complete flow with the fallback intent added

And here is what I put in the function node for the fallback flow:

var intentResponse = [
    "I'm sorry I didn't understand that.",
    "Can you try to re-phrase that I don't know what you are after.",
    "You may be asking for something I can't do, all I know about are chuck norris jokes and giving advice."
]
var randomInt = Math.floor(Math.random() * intentResponse.length)
    
msg.payload = intentResponse[randomInt]
return msg;

Once you’ve got the new flow in place don’t forget to deploy the Node-RED changes.

☞ Deploy the Changes

Once you’ve done the above, click on the the red Deploy button in the top right hand corner to make the changes live.

Now let’s test with the same gibberish command from above.

☞ Execute the command below

Feel free to change it up a bit and test what works and what doesn’t.

curl --request POST \
  --url http://localhost:1880/chat \
  --header 'content-type: application/json' \
  --data '{
 "q": "abcdefghijklmnopqrstuvwxyz",
 "model": "rasaTutorialBotFallback"
}'

and I got a much more intelligent response than when the bot just said goodbye.

You may be asking for something I can’t do, all I know about are Chuck Norris jokes and giving advice.

Our bot is already much more robust than when we started. But there are still a few cases where we could get some unexpected results. What if a user asked for advice from Chuck Norris about Marriage Proposals? What do you think would happen?

☞ Execute the command below

Feel free to play around and test what works and what doesn’t.

curl --request POST \
  --url http://localhost:5000/parse \
  --header 'content-type: application/json' \
  --data '{
 "q": "Can I get some advice from Chuck Norris about marriage proposals?",
 "model": "rasaTutorialBotFallback"
}'

In my case the request was classified as chuck Norris, but it is important to notice that the confidence is lower than other requests we’ve been seeing. This is because the combination of advice, chuck Norris, and marriage in a single sentence means that the numerical representation we talked about earlier is closer to all three intents. In the Rasa NLU Parse results you can see the intent ranking:

  {
   "confidence": 0.4739587439428035,
   "name": "chuckNorris"
  },
  {
   "confidence": 0.14829240688491713,
   "name": "goodbye"
  },
  {
   "confidence": 0.10584108588416068,
   "name": "advice"
  }

This type of fallback can be handled with the third method for failing gracefully: Handle Confusion with Confidence.

Implementing a Confidence Score Check

This time we aren’t going to modify our training data, we’re just going to modify our Node-RED flow. Specifically, once we receive the results from the /parse request to Rasa, we will check the confidence. And if the confidence is low we will give a response.

☞ Add a Switch Node

This node will go between the Rasa Request node and the Intent Switch Node. See Image 6 if you need a visual.

☞ Edit the new Switch Node’s Properties

I changed the name to Confidence Switch. Then I changed the message property to payload.intent.confidence and created two switch values one for > .5 and one for otherwise Here is what my confidence switch node properties look like.

Image 5 — Editing the Confidence Switch

☞ Add a function node

Just like we used a function node to generate a response for the greet, goodbye, etc intents. We’re going to use one here to generate a response for low confidence. Below is the code I put into my function node.

var intentResponse = [
    "I'm not certain what you mean, can you try re-phrasing that.",
    "I'm just a young bot, all I know about are chuck norris jokes and giving advice."
]
var randomInt = Math.floor(Math.random() * intentResponse.length)
    
msg.payload = intentResponse[randomInt]
return msg;

Now we need to wire everything up.

☞ Connect the new nodes and Deploy

The Rasa Request node should now go into the confidence switch. The top output of the confidence switch goes to the intent switch, but the bottom one goes to our new function node. Then that new function node goes to the HTTP Response node. Don’t forget to deploy once you’ve made your changes. Whew, maybe a picture would be better:

Image 6 — The complete flow for failing gracefully!

Great now we have implemented all 3 methods for failing gracefully. Let’s retry that last confusing example.

☞ Execute the command below

Change is good. Try different options and test what works and what doesn’t.

curl --request POST \
  --url http://localhost:1880/chat \
  --header 'content-type: application/json' \
  --data '{
 "q": "Can I get some advice from Chuck Norris about marriage proposals?",
 "model": "rasaTutorialBotFallback"
}'

And here is the response that I got:

I’m just a young bot, all I know about are chuck norris jokes and giving advice.

With that we’ve got a chatbot that is a little bit harder to trip up and should guide the user back to handled capabilities.

What’s next

If you wanted to you could now take the solution that we’ve made and deploy it in the same way we did in part 3 of the Rasa NLU Chatbot tutorial series. If you want to check out those instructions then go here:

Build a Chatbot with Rasa NLU — Part 3

You could also expand the number of out of scope intents that the bot can handle. Some common examples of things to look into would be:

• Dissatisfaction — Often takes the form of calling the bot various terms of Stupid or profanities. The user may be beyond frustrated, this would be a good time to connect them with a real person.
• Gratitude — When the user thanks your bot you should respond in kind.
• Capabilities — Especially if the user knows they are speaking to a bot, it would be likely for them to ask what the bot is capable of or even examples of things to say to the bot.

Implement in the language of your choice. Node-RED is great for observing patterns in coding, but as we add more intents we’ll quickly overwhelm the simple interface. Maybe it’s time to take what you have learned and apply it in another language.

No matter what you choose to do next, just remember that if you build a chatbot it is important to fail gracefully. Here are the three methods that we implemented to do just that:

1. Design to minimize fallbacks— It’s common for users to confess their love or even propose marriage to your bots. Your bot should handle these common, but often un-expected user requests. Even if your bot only does pizza you should be able to respond to a request for steak.
2. Collect the Garbage — With the way classification works in Rasa NLU it is important for nonsensical input to be intentionally classified into a fallback intent. Otherwise it may end up classified in unexpected ways.
3. Handle Confusion with Confidence — When the user asks for something that is too complicated for your bot to understand, this will often be evident in the confidence score of the intent classification.

If you need any help with this tutorial feel free to comment below or get ahold of me on the Rasa NLU gitter.