Wikipedia Factoid Bot (4 of 6) : Train a factoid bot to classify the intent of a user’s query using Watson Conversation

This is the fourth of a six part series on training a bot to answer factoids

Post 1: Intro factoid bot demo plus download and configure code
Post 2: Identify famous people as entities using Alchemy Language
Post 3: Initialize the factoid bot’s connection to Watson Conversation
Post 4: You are here
Post 5: Extract answers from DBpedia (Wikipedia)
Post 6: Finalize the conversation flow

Working Demo: Factoid bot running on Bluemix
Source Code: NodeJS factoid bot on Github

The prior post presented steps 2 and 3 of the workflow below and showed how to identify famous people in a user’s query using Alchemy Language. Now we’ll use Watson Conversation to further analyze the user query’s intent in steps 4 and 5.

High Level Workflow for Factoid Conversation Bot

If you look closely at the workflow, you’ll see that two api calls are made to Watson Conversation. In step 4, the user’s query is passed and an intent is returned in step 5 (more on intents later). After contacting DBpedia in steps 6 and 7, a second api call is made to Watson in step 8 where we send all the info we’ve obtained and receive a formatted answer response in step 9. Steps 6–9 are covered in the last two posts in this series so now we’ll focus step 4 and 5 to classify our user’s query.

What is an Intent?

You can train Watson Conversation to classify a user query as having a specific intent, but what exactly is an “intent”? The best explanation is provided by exploring various user intents and seeing how they are represented in text. For example, simply asking the age of a famous person can be phrased in multiple ways:

On what day was Brad Pitt born?    What is Brad Pitt's birthday?       When was Brad Pitt born?           Do you know Brad Pitt's birthday?

All the above examples end with a question mark because each is phrased as a question, but users might also pose the same intent as a directive like this:

Tell me when Brad Pitt was born.       Tell me Brad Pitt's birthday.

Each of the six queries above represent the same user intent: a desire to know a person’s birthdate. But a user’s intent isn’t always to obtain information about something as their intent could also be to have a bot execute a desired action.

ACTION PHRASE             INTENT                 ENTITY
Call my sister.           #phone-call            @person.sister
Send a message to Ryan.   #sms-send              @person.ryan
Show me directions home.  #directions-display    @destination.home

The sentences or utterances from users could also be provided in a descriptive style such as “I wonder when Brad Pitt was born.” These aren’t requests or question but simply statements. Yet you could imagine an advanced bot that intuits a user’s intent simply from descriptive utterances like those below:

DESCRIPTIVE PHRASE             INTENT                    ENTITY
I need to wake up at 5am       #alarm-set                @time.5am
Wow, it's hot in here.         #temperature-turn_down
That is way too loud.          #music-audio-turn_down

As shown above, fulfilling a user’s intents is often tied to entities that are explicit parts of the user’s statement. But as shown in the last two examples of turning down the temperature or music, the point of interest could be implied instead. So yes, the realm of possible user utterances is broad so let’s go deeper into specific user intent when learning about famous people.

Training Watson Conversation to Identify a User’s Intent

If you haven’t already, then read more about defining and managing intents using Watson Conversation’s tooling as the following assumes you have a basic understanding using those tools. Since you loaded the factoid bot’s conversation workspace during the second post of this series, go to your tooling page and click on this tile:

Factoid Bot’s Workspace Tile

You’ll automatically land on the Intents screen which lists the previously defined intents such as #person-birthplace.

Intents Summary for Watson Conversation

Because our source for answers is DBpedia, our bot supports only queries for which answers can normally be found there. Here are examples of data that DBpedia provides about most famous people so our intents only cover these topics for now.

almaMater          birthDate          birthName          birthPlace
citizenship        deathDate          deathPlace         spouse

If you click on the row for the #person-birthdate intent, we’ll explore the user queries used to train that intent.

Example user queries that are of type #person-birthdate

You‘re likely curious why we put XYZXYZ for the famous person’s name and how this would affect training. This was done to avoid skewing the intent training towards any specific famous person. It also makes it easy at a glance to see review the variations in training examples. In a later post, I hope to perform an in-depth analysis of the impact of doing this.

Be Robust to Classification Error With Generic Replies

Look closely at the training examples for the #person-birthdate intent and you’ll notice that two types of queries have been grouped into this single intent: birthdate and age queries. This allows a single response to the user that has both of answers at once and simplifies training:

Brad Pitt is 53 years old and was born on December 18, 1963.

Why is this important? E.g. you could imagine a user asking How old is Brad Pitt and when was he born? or Tell me when Brad Pitt was born and how old he is. Since dual-yet-related queries often occur, we group them into a single intent rather than ignoring part of a user query or trying to somehow parse two intents from a single query.

So as a standard practice and rule #1, don’t over-engineer your conversation flow as no conversation flow survives confrontation with the user. Train your bot with general intents and return general responses when initially designing your conversation flow. Your bot will be more robust to intent misclassification when training examples overlap, and you’ll reduce complexity until more real user data is gathered. You can always segment your replies more granularly in the future.

Handling Intents With Minor Differences in Phrasing

Let’s reinforce this with a more challenging example of the intent: #person-spouse. The user might ask for the current spouse or a prior spouse. Only a single word would be different and thus difficult for intent classification to disambiguate.

CURRENT SPOUSE                 PRIOR SPOUSE
Who is Brad Pitt’s spouse?     Who was Brad Pitt’s spouse?
Who is Brad Pitt’s wife?       Who is Brad Pitt’s first wife?

By replying in a more general manner, we can handle both of these with a single response and be more robust to errors.

Brad Pitt is married to Angelina Jolie.  His prior spouse was Jennifer Aniston.

You’ll see how this is handled when we finalize the design of the conversation flow in the final post in this series. Go ahead and look through all the other intents that that factoid bot will recognize to see how the advice above was put into practice.

Submitting and Responding to User Queries

Watson Conversation is now trained on all our intents and ready to classify them. So ConversationUtils.processUserQuery() can submit the text of our user’s query plus context to Watson to see if it matches a supported intent.

See full code at javascript/conversation_utils.js
ConversationUtils.prototype.processUserQuery(...)
{
    var payload = internalThis.getDefaultConversationPayload()
    payload.input = req.body.input
    payload.context = req.body.context
    ...
    this.conversation.message(payload, function (err, message) {
        ...
   }
}

Watson will reply with JSON containing the identified intent. So if we passed input = "When was Brad Pitt born? then the response would be:

{
    "intents":[{
        "intent" = "person-birthdate",
        "confidence" = 0.826269897353014
    }],
    "entities":[],
    "input":{"When was Brad Pitt Born"},
"output":{
        "log_messages":[],
        "text":[],
        "nodes_visited":["node_9_1483510113199"]
    },
    "context":{
        "conversation_id":"c6f2cf1f-d5f4-4c47-83d8-68e3adf63dc5",
        "system":{
            "dialog_turn_counter":2,
            "dialog_stack": [{"dialog_node":"root"}],
            "dialog_request_counter":1
    }
}

But let’s walk through exactly how to configure Watson Conversation to be robust when classifying intents.

Handling Supported and Unsupported Intents

But in reality, providing feedback to users about unsupported queries is as important as answering valid ones. So we should ensure Watson Conversation is highly confident about a query’s intent else we should ask the user to restate their query.

Open your factoid bot’s workspace in the tooling and go to the Dialog screen to view the bot’s root-level conversation flow. We’ll now look at the second node as our first query will be submitted at dialog_turn_counter = 2.

Root level of the factoid bot’s dialog flow

Expanding that node shows how the child nodes perform a test whether confidence in the intent classification is ≤ 65% or not.

Testing confidence of intent classification

Expanding that node shows how Watson will respond for low confidence intent classifications.

Intent confidence ≤ 65%

You’ll notice a special formatting option being used in the response:

Sorry I don't understand your question. I can currently only answer questions about famous people such as their birthday, age, birthplace, or spouse. Click on any of the floating queries below to explore. Intent=<? intents[0].intent ?> and confidence=<? intents[0].confidence ?>

For debugging purposes, the top intent’s name <? intents[0].intent ?> and confidence <? intents[0].confidence ?> are appended to the reply. The full intent object can also be printed <? intents ?> when it’s useful to see more details.

Next let’s look at the confidence > 65% node to see what happens.

Intent confidence > 65%

The flow checks if a $person context object is available, but at this point we haven’t looked up the answers in DBpedia so $person == null. The anything_else node is evaluated next and simply returns the classified intent back to the factoid bot’s NodeJS code. And now we’re ready to lookup the intent’s answer in DBPedia.

Now Let’s Find an Answer to the User’s Query

So we’ve extracted the famous person as an entity which gave us a link to that person’s page on DBpedia, and now we’ve classified the user’s intent. Next we’ll make a SPARQL query to DBpedia for Brad Pitt’s age and then finish this series by extending our conversation flow to format the factoid answers properly.