Analytics Vidhya
Published in

Analytics Vidhya

Demystifying Chatbot NLU (Era of Chatbots)

There has been a huge hype in chatbots or digital assistants in recent years and in the post-pandemic era digital assistants are likely to become a new normal. In this blog, I will try to explain what are the major components of chatbots and how these components can collectively be used for various inferencing tasks.

Understanding the Conversation

The first and most important part of a chatbot is to understand the conversation. To understand a query better we first need to identify, “What the user is asking?”, “What information the user has provided?”, “What is the context?”, and “What is the tone or sentiment of the user?

The most important of these is “what the user is asking?”, or Intent (Intention of User).

What is an “Intent”?

An Intent is the bot user’s intention. It is used to understand what the user wants to do.

Suppose you are building a bot for the banking domain, the bot users are expected to check their account balance or find the nearest branch, these can be represented as intents like “check balance” or “find branch” respectively. These intentions along with the other entities, or traits extracted (which gives more context), are utilized by the inference engine to take appropriate action.

There are many ways to identify the Intent of a query,

Rule-Based Approach: Using Templates or Rules based on various Linguistic Features (like POS tags, lemmas) or regex patterns.

Machine learning Approach: Using machine learning techniques like SVM, NB, Logistic Regression, or any advanced neural network approaches, along with Embeddings to capture more context or meaning of the bot user.

Intent can be used to take different actions like Information Retrieval, Work Flow Execution, Chit Chat(Small talk).

NLU for Chatbots


What is an “Entity”?

Entities are the key pieces of information present in the bot user’s query, the process of detection and grouping these key information into pre-defined categories is called Named Entity Recognition(NER).

PC: Spacy NER Demo

These entities give more context to the Intent of the query, for example, a query like “find a restaurant which serves Chineese[Cuisine] food”, here the intent would be “find Restaurant” and entities would “cuisine” is “Chinese”. Once we know the intent and cuisine, this information can be used by the inference engine to retrieve restaurants serving Chinese cuisine.

There are multiple ways to extract entities like Regex Entities, Sequence-Based (using BILOU, or IBO techniques). There are lots of pre-defined entities in multiple domains available in multiple languages like names, organizations, locations, etc.


What is the context?

A conversation, unlike a search based retrieval, is usually in multiple dialogs, So it is important to keep a track of and understand the context the current dialog is currently in.

Context Management plays a vital role in the user experience and intelligence of the bot.

Suppose you have been inquiring about your car insurance, and the bot requires many details like car details, registration number, etc. which are the entities or in bot terms slots that need to be filled in order to do an action. These slots are usually extracted in the conversations between the bot user and the bot need to maintain these slots as well as the topic car insurance in context so the bot can take the necessary action.

Understanding and responding in the context is not the only important part of context management. It also needs to manage the context switching or, sudden off context dialogs by the user. Like how the bot should react to unexpected(off context) responses from users if it should clarify the context to the user or change context and prompt the user that he lost context earlier and if he wants to get back to it. This could make the bot user experience much more engaging.

What is the tone or sentiment of the user?

It is important to always analyze the sentiment and tone of the user, like if the user is taking positively or negatively, and if the user is angry, happy, sad, etc. So that the bot can control the conversation if the user gets frustrated or the user no able to get what he wants in the bots.

Tone analysis for query “I hate your customer support”. PC: Cogniwide NLP API’s

If the bot can track these traits then the dialog manager can be configured to take the necessary steps to control the bot's actions or to give more empathetical responses to the bot user.



This is a short non-technical overview of the various components for Language Understanding in Chatbots. All the above components are collectively used by the expert system to respond in a more human-like and empathetic way.

Check out some of the resources for the implementation of NLU in chatbots

Some Tools and Frameworks: Hugging Face, Spacy, NLTK, GATE, scikit-learn, Tensorflow, PyTorch

Interesting Blogs and channels: TrevTutor(Linguistics), StatQuest(Statistics), 3blue1brown, nlp-progress

Datasets and notebooks: Big Bad NLP Database, Kaggle

ChatBot Platforms: RASA, Kore AI, Google Dialog Flow, CogniAssist, IBM Watson, Wit ai

Below are some of the other main features which will be discussed later.

1: Driving the Conversation (dialog and context management)

2: Information Retrieval

3: Knowledge management (Domain Information and Documents)

4: Inference Engine / Expert Systems

5: Channel Support and Social Media Presence

6: Language Support and Cross-Lingual Understanding



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Athul Anand

Athul Anand

Consultant NLP/ML, Cogniwide Technologies | Conversational AI