Conversational Search in Knowledge Bases using NLP/NLU and Chatbots

Conversational AI becomes increasingly popular in the form of chatbots, especially as a means of quick information access.

In this post we want to showcase how you can easily build a natural language interface to any knowledge base by training a chatbot using the Rasa framework.

Photo by Gordon Tarpley on Flickr / CC BY 2.0

In this example we use Elasticsearch (ES), but this technique is applicable to other infrastructures as well. We will build a conversational search bot for books and movies using an IMDB movie dataset (incl. movie ratings) and a book dataset. Head over to our GitHub repo to dive into the complete code.

Our bot will support the following query types using natural language:

• Querying multiple indices (books/movies/ratings)
• Filtering by one or more attributes (e.g., only show action movies)
• Projection of individual attributes (e.g., show the summary of a movie)
• Range queries on attributes (e.g., show movies shot after 2020)
• Joining two indices (e.g., join a movie with its rating)

We want to be able to chat with our bot like shown below instead of writing complete ES JSON queries.

In chatbot development we model our use case with intents (what the user wants), entities (objects referred to in a message) and actions (triggered by intents and performed by the chatbot). Rasa includes a special (still experimental) action called action_query_knowledge_base, which does exactly what we need for this project. The values of the entities are stored in so called slots and guide the execution of actions.

We define intents and entities using YAML files (specifically nlu.yml for intent training data and domain.yml for declaring intents, entities and slots). For our search bot we define the intents query_books and query_movies and entities for all queryable attributes (e.g., author, actors, director). To be able to use Rasa’s knowledge base functionality we have to add an additional entity object_type (i.e., are we talking about books or movies).

We then wire up our intents with the actions in the rules.yml.

Note: It is also possible to only define one intent to trigger the knowledge base action for both books and movies, however we noticed that the entity prediction improved when we split the data into separate intents.

In the domain.yml we have to list all the intents we created. Here we can further improve the entity prediction by directing Rasa to ignore certain entities when a specific intent was recognised. This avoids scenarios like authors being misclassified as actors and can greatly improve your chatbot model!

To translate Rasa’s representation of the chat message into an ES query, we have to create a custom action class in python. We do this by inheriting from Rasa’s ActionQueryKnowledgeBase. We override the methods utter_objects and utter_attribute_value to customize the message output for object and attribute queries. But more importantly we have to create a knowledge base object in the action’s constructor. We therefore create a class ElasticsearchKnowledgeBase inheriting from Rasa’s KnowledgeBase interface and overriding the methods get_attributes_of_object, get_objects and get_object to perform ES queries.

We design ElasticsearchKnowledgeBase to allows for easy adaptation to any ES mapping. To achieve this we define the class DocumentType and classes for different types of attributes, which can be extended to reflect the knowledge base schema. The DocumentTypes hold an Attribute object for each of their attributes in a dictionary. Each attribute class overrides the generate_query method, which creates the ES query syntax.

The method get_objects in ElasticsearchKnowledgeBase then constructs a compound query out of all the queries generated by the attributes that are present in the context.

We tell Rasa to train the chatbot based on our training data using the command rasa train. It then uses the NLP pipeline defined in config.yml. As we are not dealing with domain-specific terms we chose the recommended pipeline based on Spacy word embeddings.

In order to start our chatbot we have to run the action server, which will be responsible for executing our custom knowledge base action, using

rasa run actions

To start up your bot you can either run the rasa shell command or use Rasa X.

Now we can start chatting with our bot like so:

But how does this work? We can use the command rasa shell nlu to get a debug output for our message:

As we can see Rasa’s NLU engine detected the word ‘films’ as object_type, ‘action’ as genre and ‘Brad Pitt’ as actor and filled the respective slots.

The object_type tells our custom action to look for objects in the ‘movies’ index. Because genres and actors are both attributes of that index, they are set as filter attributes. The ES query generated under the hood looks like this:

Query individual attributes of an object

But what if we want to know more about a film?

In order for this to work we need a way of referencing a specific movie in the conversation history. We do this by asking the bot for information about the ‘first’, ‘second’, ‘last’ and so on object in a list or ‘it’, if only one movie is in context. We introduce these keywords as an entity called mention. The knowledge base action by default supports ordinal mentions for indices from 1 to 10, ‘ANY’, ‘LAST’ as well the direct mention ‘IT’. It can automatically match these to objects in the conversation context.

Furthermore, we have to declare an entity called attribute, which takes on the values of all queryable attribute names in our knowledge base.

We put these into a separate intent know_more. Again, we are using a separate intent only because the prediction of the entities mention and attribute seems to perform better this way. This makes the training data easier to maintain as well.

Now, if the attribute slot is set, an attribute query is performed for the mentioned object.

Perform range queries on publication year

Until now the movies we get back from our queries are randomly scattered across time. We want to fix that and ask the bot something like this:

In order to achieve this we model the entity publication_year. But this is not enough as the year can take on different meanings depending on wether we want to query newer, older or films from this specific year.

Fortunately Rasa supports entity roles. With this feature we can train our model to not only detect values of entities but also their role in the sentence context. We introduce four roles: lt (lower than), gt (greater than), eq (equal) and NEW (movies from last or the current year).

It is worth noting that entity roles (at least for our use case) require quite a lot of training data to work properly. For us the role prediction improved when the term giving context to the publication year (e.g., ‘after’ or ‘before’) was included in the entity annotation. However, this has the drawback, that these words are sometimes interpreted as the year value as well. In general we noticed that in order for entity role classification to work well a lot of training sentences are required.

The RangeAttribute class then uses the value of the entity role to generate an ES range query:

Note: In order to be able to access the role here we had to override the Rasa method get_attribute_slots and retrieve the role from Tracker's entity list. The original method only returns attribute's names and values.

Perform join queries between movies and their ratings

Now we want to include an additional ES index in our knowledge base: movie ratings. These can be joined with movies via their id.

We can implement this by modeling ‘rating’ as another object type.Whenever a new object type (e.g. ‘rating’) appears in the conversation together with a mention of an object of a different kind (e.g. referring to a specific ‘movie’) we want to do a join query. ActionQueryKnowledgeBase’s run method does not support this functionality. So we override it in order to distinguish a new case.

The join method then simply finds the rating object with the same id as the mentioned movie object.

Testing our chatbot model

Our chatbot works now, great! But how can we test it to make sure it operates correctly?

The chatbot model can be tested by either providing a concrete train/test data split or using cross-validation. This returns testing metrics like F1-Score and confusion matrix for intent and entity classification.

rasa test nlu --cross-validation

Additionally we write test stories. They define a conversation flow including the user input, correct intents and entities and the slots that should be set.

To execute them we call:

rasa test nlu --stories tests/test_stories.yml

We can even include these tests in our CI workflow! If we add the flag —- fail-on-prediction-errors for test stories our pipeline will fail whenever something gets misclassified in the stories. This is great for regression testing!

With GitHub Actions we can even create a CI pipeline that gets triggered on the creation of pull requests and posts the current model testing results as a comment on that PR. Check out the model_ci.yml for details.

Thank you for reading this blogpost and good luck with your own chatbot projects!