How to create a NLU-powered chatbot
What is NLU? How to make chatbots understand what you want?
An introduction to Natural Language Understanding for chatbots
In this article I’ll introduce the concept of NLU, or Natural Language Understanding, as a tool for chatbot creation, and will guide you through the creation of a NLU-powered conversational agent step by step.
NLP, NLU, what is the difference?
We first have to distinguish NLP (Natural Language Processing) from NLU (Natural Language Understanding). NLP is a domain at the crossroads of computer science, linguistics and artificial intelligence. It is composed of a set of algorithms aiming at processing and analysing natural language textual data. Some examples of NLP algorithms are the following:
- Part-of-speech tagging (POS tagging): an algorithm that tags each word of a sentence based on its grammatical function (subject, verb, noun, pronoun, and so on)
- Named Entity Recognition (NER): an algorithm that extracts the named entities of a sentence, like locations, persons or companies for example.
- Stemming: treatment consisting of reducing words to their root form (“eating” → “eat”, “users” → “use)
- And many more!
NLU is a subset of NLP and its purpose is to extract the meaning from the text. For example, extracting the sentiment from a sentence, or the intention of the writer, or also the semantic of the sentence. There are many applications for these specific techniques, such as text paraphrasing, summarisation, question answering and more. You can therefore imagine that these techniques are loved by chatbot creators.
How does NLU services work?
In the chatbot world, NLU is usually a service that is called on every message received by a user that tries to understand the intention of this user. This will allow the automated agent to know what and how to reply. So let’s dive into the explanation of how a NLU service works. First we need to understand the basic vocabulary:
- Utterance: it’s simply the sentence sent by the user to the chatbot
- Intent: it’s the final goal of the sentence, that the NLU service will try to understand
- Entity: it’s like a parameter, to specify information for the intent
With an example, it’ll be extra clear, see below:
Each utterance will therefore be mapped to an intent, and each intent can have 0, 1, or multiple entities. It’s possible to create our own NLU service, or we can use existing systems, like Microsoft LUIS, Google Dialogflow, IBM Watson NLU, Amazon Lex, Rasa NLU and more.
What are the logical steps to create an assistant?
When creating a conversational agent with a NLU service, you have to follow different steps.
1. Define the intents and entities
The first step of course is to define the intents, and then the corresponding entities if needed. Examples of intents for a tourist office chatbot could be:
- ask_store_location: when the user asks for a store location (see example above)
- ask_store_opening_hours: when the user asks when a store opens or closes
- thanks: when the user says thank you
- ask_for_human: when the user asks to talk to a real human
- and so on…
Entities in this example could be store_name, city_name, activity_name, date, hour, email_address, and more.
2. Collect training sentences
In order for the NLU service to detect the different intents and entities correctly, it needs training. Thus you need to collect example sentences for each of the defined intents. You can write them yourself in the beginning. But there is a big chance that another person will talk differently to your bot than you imagined, leading in bad intent recognition.
It is, therefore, better to collect sentences from other persons. A service called Chatbotstrap will help you for this step. It allows you to create “scenarios” of conversations and ask people to reply to the bot in five different ways. You then have two ways to collect sentences. The first one is to share the link of these scenarios to whoever you want (friends, family, coworkers), and the second is to crowdsource the sentences on platforms like Amazon Mechanical Turk or Clickworker. Both methods have their advantages and disadvantages.
Chatbotstrap allows you then to manually validate the samples. We recommend to have at least 50 training sentences for each intent.
3. Train the NLU model
The third step is to train the NLU model and evaluate how it behaves. This step differs depending on the service you use. In the case of online services like Dialogflow, you just have to click on a button to train the model. Rasa, on another hand, is a bit different. It’s an open source software that you can self-host on your servers. This is particularly interesting if you don’t want your users’ data to pass through the servers of other companies. In the case of Rasa, you have to run a command to train your model and evaluate it.
Rasa offers a configurable pipeline for your NLU system. You can improve it and tune it to suit your needs. Rasa unveiled their new NLU model called DIET (Dual Intent and Entity Transformer) in March 2020. It is really interesting, customizable and powerful. If you are interested in its internal workings, I recommend the series of 3 videos that the Rasa team has put online:
- Part 1 — DIET architecture: How it works
- Part 2 — DIET architecture: Design Decisions
- Part 3 — DIET architecture: Benchmarking
4. Code your bot
And the last step is to code you bot, and the “intelligence” of it. This step varies a lot depending on the NLU service you chose. For example in Dialogflow, you can create a web service, or webhook, that will be called by Dialogflow with specific parameters when a specific intent is detected. With Rasa, you have different possibilities. You can use only the NLU service, make it available as a web service, and call it when needed with your code, or you can use Rasa Core to design the conversations (more on that on the Rasa website). And third option, you even have online services that allow you to create chatbots graphically, without coding. This part could be an article on its own, so I keep it short here.
The webhook code is the place where you get data from a database, call an API, fetch a predefined answer somewhere or even generate an answer. The dynamic generation of the reply is a big topic as well, and involves artificial intelligence, again. See NLG, or Natural Language Generation, on your favorite search engine (or on a future article) for more info.
5. Deploy the chatbot
And the last step is to deploy the conversational agent somewhere. Again, many possibilities await you: Facebook Messenger, Telegram, Slack, Discord, Twitter, your website or why not an e-mail or SMS bot. The possibilities are endless (I know, it’s a cliché to say this).
- Hint 1: deploying is good, but don’t forget to test your bot
- Hint 2: testing is good, but don’t forget to monitor your bot
- Hint 3: monitoring is good, but don’t forget to improve your bot with the previously unseen sentences that new users send to your bot
That’s it for today
Ok, I don’t know if you reached the bottom of this article. I hope you’ve learned something or at least enjoyed reading this little article about NLU and how to create a NLU-powered chatbot. Of course this was an introduction and I tried not to go into too much details.
Don’t hesitate to share your thoughts about the topic or to correct the mistakes I probably said. It’s been few years now that we, at the HumanTech Institute and at Empathic Labs, research and develop chatbots. I’d say that I admire the work they do at Rasa (so I may be biased when I recommend you something ahah, sorry).
Interested in other articles about chatbots at Empathic Labs? Here you go:
- “Make chatbot dead-end conversations more funny with movie quotes”, by Nicolas Crausaz
- “Evaluate datetime extraction algorithms in French conversations”, by Luke Perrottet
If you don’t know what Empathic Labs is, I suggest you read this introduction article. TL;DR → we’re a bunch of Swiss researchers, we love empathic interactions between humans and machines, and we say it out loud.
Jacky
PS: Please give me some claps ;)
