Natural Language Understanding for Chatbots
Natural language understanding (NLU) is a branch of artificial intelligence (AI) that deals with understanding human language.
The NLU is used to accomplish two main tasks: to identify the intent behind what a person is saying, and to generate a response based on the identified intent.
In simple terms, the intent is an action or task that the user wants to accomplish. Based on the users’ needs, the NLU software provides a text or a voice response. In any case, it should be tailored to the user’s needs.
If the user wants to “check” a movie’s rating, its response should be the movie’s rating (e.g. “The movie was rated as PG-13”).
Who is using NLU?
NLU software can be used in a variety of applications including customer service, sales, chatbots and virtual assistants, etc. Some NLU software solutions are designed to only identify the intent of the input while other software solutions are designed to also generate a response. Many companies are using NLU, e.g., Amazon uses it to help you complete purchases, Google uses it in its search engine and Netflix uses it to recommend movies or TV shows. NLU helps these companies provide a better user experience to their customers.
How does NLU work?
A naive NLU system takes a person’s speech or text as input, and tries to find the correct intent in its database. The database includes possible intents and corresponding responses that are prepared by the developer. The NLU system then compares the input with the sentences in the database and finds the best match and returns it.
NLU researchers and developers are trying to create a software that is capable of understanding language in the same way that humans understand it. While we have made major advancements in making machines understand context in natural language, we still have a long way to go.
Some of the most prominent use of NLU is in chatbots and virtual assistants where NLU has gained recent success. These systems are designed to understand the intent of the users through text or speech input.
What are chatbots and why do we need them?
Chatbots are gaining popularity day by day. Not just the ones you see popping up on various websites, but there are many bots working to improve business use cases like helping improve supply chain processes. One example is Amazon’s Seller Messaging Assistant.
However, it seems ridiculous to believe that any day bots will replace websites and apps. But, there are more than one reasons to believe so, be it ease of bots usage, no apps needed, no memory space needed, the same platform works for a lot of bots and so on.
The future of Web is definitely going to be bots and according to several tech reports, the bot internet traffic will be doubled by 2025.
Reasons why chatbots will replace websites and apps in near future
- Ease of Use: If we look at the basic points of chatbots, it’s hard to believe that it’s going to replace any website or app, but still there are good reasons to believe so. When we chat with anyone, the first thing we do is to understand if the person is a bot or a human, if it’s a bot, we don’t expect too much from it and we try to get the work done with the least possible effort, we try to keep the conversation as short as possible and we try to do it as fast as possible, so that we can move on to the next task. On the contrary, when we talk to human, we expect a lot from them, we try to keep the conversation long and we try to keep it interesting. This is the main difference between human and bots. Now, what happens when we visit a website, first impression matters a lot, it’s not like we’ll talk to the website or the app, but at least we expect a lot from it, like, we expect to get all the information we need at one place, as fast as possible, and we expect the website to be responsive. If the website fails to meet these expectations, we simply close it and try to find another website, it’s not that we talk to the website, but we talk to the service provider via website or app. If a website or app fails to keep us engaged and if it fails to provide a great user experience, we simply move on to another website or app.
- No app installation: When we talk about bots, they are not like normal applications, they don’t require any app installation, they don’t take any memory space and a lot of chatbots don’t need any Internet connection to work.
- Always available: Another important thing is that they are always available, and they never close, in short they are always there for you.
Components of a good chatbot
Let’s take an example of a chatbot for a music streaming application. When you ask your music assist chatbot to “Play Coldplay”, you are implicitly asking it to do two things:
1.Identify what is the intent behind the command (which in this case is to play something on a device).
2.Identify what to play (which is Coldplay in this case).
Here the bot has to identify the intent of the user. This is called Intent Classification. Then, it has to figure out what is Coldplay. Is it an Artist, Song, Playlist, etc. This is called Entity Extraction, where Coldplay is an Entity.
Intent Classification
Intent classification is the process of classifying the customer’s intent by analysing the language they use.
For instance, a customer who types “where is the nearest pharmacy?” is probably looking for a pharmacy, while a customer who types “where is the nearest restaurant?” is probably looking for a restaurant.
How to classify intents correctly?
There are two main methodologies for intent classification: rule-based and statistical.
1.Rule-based intent classification is based on expert-defined rules. This technique is cheaper and faster to build, and is flexible enough to be customised, but requires a large amount of human effort to maintain.
2. Statistical intent classification is based on Machine Learning algorithms. Statistical classification methods are faster to train, require less human effort to maintain, and are more accurate. However, they are more expensive and less flexible than rule-based classification.
How to do intent classification in a chatbot?
The first step in building a chatbot is to define the intents it will handle. Intents can be modelled as a hierarchical tree, where the topmost nodes are the broadest or highest-level intents. The lowest level intents are self-explanatory and are more catered to the specific task that we want to achieve.
For example a top level intent could be ‘ordering food’. The next level could be ‘ordering food of a specific cuisine’ At the last level, we will have specific dish names like ‘Chicken Biryani’.
Let’s see an example of intent classification in action:
Entity Extraction
The process of extracting targeted information from a piece of text is called NER. E.g., person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.
Let’s revisit our previous example where we asked our music assist bot to “play Coldplay”. An intuitive understanding from the given command is that the intent is to play somethings and entity is what to play. When we say “play Coldplay”, a chatbot would classify the intent as “play music”, and classify Coldplay as an entity, which is an Artist.
An entity is anything that exists in the real world, and can be a person, place, product, organization, or a concept.
Depending upon the application, there can be a large variety of entity types. For example, in news articles, entities could be people, places, companies, and organizations. In healthcare, entities could be diseases, drugs, and procedures. In the military, entities could be weapons, ships, and people.
How to extract entities from a given text?
There are many ways in which we can extract the important information from text.
It can range from a simple solution like rule based string matching to an extremely complex solution like understanding the implicit context behind the sentence and then extracting the entity based on the context. E.g., “Play Cricket”, and “Play Coldplay”.
A simple string / pattern matching example is identifying the number plates of the cars in a particular country. Since the pattern is fixed, we can write a regular expression to extract the pattern correctly from the sentence.
A complex sentence can be understood by identifying the implicit context behind the sentence. The entities can be extracted using the information stored in the context. The entity can be any from the list of : person, organization, place, event, product, service, time, quantity, etc .
Example: Apple announced the launch of their new device last Thursday that will compete with the likes of Samsung and Google. The device will be launched in India on the 11th of September .
Here, last Thursday refers to the last Thursday from the current date. To extract this information, we can use the information available in the context. That is, the current date, the day before yesterday, the day before that, etc.
But why do we need Entity extraction?
Entity Extraction is a very important task in the following situations:
1.Information Retrieval: to fetch relevant information from a large collection of documents. Example: When we say ‘Play Coldplay’ we need to ensure that Coldplay exists in our database and then extract the information present.
2. Knowledge Discovery: to find information about entities and their relations between each other. Example: Identifying Google as ORG and Alphabet as a parent company. Based on this relation, we can infer Alphabet to be an ORG.
3. Information Filtering: to recommend relevant documents based on the identified entity. Example: We can recommend Maroon 5 based on the interest in Coldplay.
4. Question Answering: to answer questions about entities (who, what, when, where, why). Example: When was Google started, who are the founders, their business model, mode of operation and so on.
5. Security: to identify and track suspicious activities. Example: To identify entities not present in DB and to categorize them. Identifying non-members of an organization to prevent them from accessing confidential information.
NLU will be required in every application that uses or contains text in natural language. From search engines or recommender systems to chatbots, NLU is being used everywhere. But, to use an NLU system efficiently, you need to understand what it can do and does it fit your use-case or not.
In this blog we have discussed basics about NLU and main components of a simple chatbot. In the next blog, we will discuss the entire development life cycle of a chatbot.
If you are interested in knowing how we are tackling the problem of low resource languages like Indian Languages, check out this blog:
Stay tuned to understand more about end-to-end NLU systems and how to choose the right one for your use-case.
If you have any questions, feel free to comment or find me on Twitter.
