When Things Go Wrong with NinjaChat (also: Localization)

This article is part three in a series of articles (previous parts #1, & #2) about how we set up a chatbot — NinjaChat — to serve our shippers and consignees here at Ninja Van. Here, I will cover how fallbacks were used to effectively ensure that input from our bot users were always intelligently managed (even if said input did not make sense in the context of the conversation). But just as importantly, I will also talk about how we implemented localization for our bot, which, as of right now, is available in four languages — English, Indonesian, Thai & Vietnamese.

Using Fallbacks to Recover From Errors

It would be foolish to only provide for happy paths in a conversation tree that deals with a free-text chatbot experience, yet nigh impossible to provide intents to handle the full range of input that users can conceivably provide. Luckily for us, Dialogflow provides fallback intents, which are intents that, like regular intents, match against a set of input contexts and activate a set of output contexts. But they do so when no other intent match can be found, given the user’s input and the training provided for each intent.

Dialogflow — Default Fallback Intent

In fact, when you create your very first agent (the entity that handles your conversation with your users and and stores all your intents), Dialogflow automatically provides a Default Fallback Intent with no input and output contexts, meaning that it is literally a catch-all for all scenarios where a user enters something that does not match any of your intents. However, relying solely on the default fallback provides a subpar chatbot experience, for the simple reason that it cannot control which flow-specific contexts to deactivate/re-activate when it is matched.

This means that, if a user were to be in the middle of entering a chosen pickup address and if the default fallback is repeatedly matched, the lifespan of the previous pickup contexts would eventually expire. This leaves the user in a sort of limbo where he/she is in a conversation with neither main menu contexts or flow (e.g. pickup) specific contexts. Any utterance from the user at this stage would not be matched according to what the user expects from the conversation, and may even trigger unexpected results.

For our chatbot implementation, we have deliberately constructed fallbacks for all critical states/pathways in the conversation tree. As a result, no matter where a user may be in a conversation, uttering garbage would bring the user back to where he was before. Consider this interaction:

//Intent: Shipper — Create Pickup was matched
Chatbot: Which pickup date would you like? <shows list of dates as options>

User: Please leave me alone.

//Intent: Shipper — Select Pickup Date (Fallback)was matched
Chatbot: I’m sorry, I did not catch that. <shows list of dates as options>

Here, we have set up a fallback that has the same input contexts as the intent that is supposedly to be matched had the user entered a valid date (e.g. say Shipper — Select Pickup Date). The benefits of having this specific fallback now matching garbage input are:

• We can have the specific fallback intent re-activate as output contexts the same contexts it was accepting as input, meaning it would effectively revert the conversation back to the state it was before the user entered garbage. If the intent Shipper — Create Pickup had shipper-sns and shipper-create-pickup as output contexts, the fallback intent would accept these intents as input, and set them as output contexts as well.
• We can configure an action that matches what the user had seen in the previous interaction, in this case something like DISPLAY_PICKUP_DATES, such that the fallback response (“I’m sorry, I did not catch that.”) is personalized and has the expected correct response from the user e.g. a date from a list of dates.

But what if we want to return the user back to their previous state despite having a valid intent match on Dialogflow? This was a scenario we encountered often.

Let’s use the same example as above. Although we’ve set up Shipper — Select Pickup Date to accept and respond to a list of dates, we actually want to restrict the date the user eventually selects to one that was presented as an option in the previous interaction (these are dates that have been validated against the shipper’s settings stored in our database). There is no easy way to build an intent that would match only to specific dates that vary for each shipper, so we need a way to effectively display an error and get the user to choose another date. That’s where events come in.

//Intent: Shipper — Create Pickup was matched
Chatbot: Which pickup date would you like? <shows list of dates as options>

User: 29th March 2020 please.

//Intent: Shipper — Create Pickup (Event — Invalid Date) was matched
Chatbot: Sorry, the date you have chosen is invalid. Please try again. <shows list of dates as options>

In addition to fallback intents set up, we also have regular intents that are matched by events with the purpose of resetting state at these junctures where post-intent match validation is performed.

Dialogflow — Event Section on Intent page

Here, we would have something like Shipper — Create Pickup (Event — Invalid Date) set up with TRIGGER_VALIDATION_FALLBACK_EVENT as its event. We call the same sessionsClient.detectIntent() in the API client but instead pass it in an event value instead. The only difference is that unlike a regular fallback intent, this validation intent has the same input contexts as the expected matched intent Shipper — Select Pickup Date, and it has output contexts similar to that of the previous state Shipper — Create Pickup, that is shipper-sns and shipper-create-pickup. Oh, and of course, the same action DISPLAY_PICKUP_DATES as well.

Using Responses to Localize Output

Before we had to accommodate non-English languages, many of our intents had the actual text responses (what we eventually show to the user) configured as part of the intent itself. Supporting localisation meant that we needed a workaround for controlling the actual responses displayed and this was done by integrating Lokalise, a translation tool, into our service.

Intents would be set up with the same responses for all supported languages, where the response itself is a message key that is mapped to a localized value in our messages.<language code> file e.g. messages.en-SG, messages.vi-VN, etc.

//Field from intent json config file
  "messages": [
    {
      "type": 0,
      "lang": "en",
      "condition": "",
      "speech": "df_select_order_enter_tracking_id"
    },
    {
      "type": 0,
      "lang": "id",
      "condition": "",
      "speech": "df_select_order_enter_tracking_id"
    },
    {
      "type": 0,
      "lang": "th",
      "condition": "",
      "speech": "df_select_order_enter_tracking_id"
    },
    {
      "type": 0,
      "lang": "vi",
      "condition": "",
      "speech": "df_select_order_enter_tracking_id"
    }
  ],

When building the project, a script is run that inserts all new message key value pairs found in a central messages.en file (which we use as the source of truth) to Lokalise, and also updates changes to existing pairs. Translators working on Lokalise routinely update the translations for other languages there, and these are downloaded and added to the project during the build. At runtime, all message keys are translated to their localized values by referring to these messages files. And with this, our translation woes were (mostly) resolved.

//Sample entries from messages.en
df_default_introduction=Thank you for using Ninja Van! How can I help you today?
df_other_options_prompt=How else can I help you?
df_inquire_prompt=What would you like to inquire?
df_customer_other_issues_no_matched_orders=It looks like you have no (matched) orders in our system. Would you like to speak to a live agent anyway?
df_live_agent_connect=Our live agent is connecting with you now. Please hold on for a moment!

Of course, translation has overall still been a herculean effort involving tons of resources and man hours, and we are still looking into improving our processes around handling translations whenever intents/message key changes are involved.

Using Events to Enforce Intent Matching

While we’re on the topic of localized message keys, I should mention that we also introduced a system of direct event triggers with Dialogflow during intent matching whenever a user had selected a predefined option shown on the platform as a button, quick reply, etc.

Simply put, here’s how it worked previously. For each intent that we have set up, if there is a menu option that was shown to the user which was meant to match with it, we submitted its localized text as a query string to Dialogflow.

Now, instead, we configured an event for this intent that corresponds to the message key of the option shown. We then made sure that the webhook that reaches us when an option was clicked would also contain the message key identifier, which we pass on directly to Dialogflow by triggering an event. This event, of course, would then unambiguously and very certainly match with the initial intent we had configured.

What good does this do? For starters, we can now guarantee an intent match for our agent setup as long as the user clicks a button while navigating our chatbot, as opposed to typing in free text (and from our research, they often choose to click buttons instead). Other ancillary benefits:

• Faster development times while working on a feature.
• QA will not be blocked by translations while testing our application.
• No more (who am I kidding, less) worrying about translations.

Using Priorities to Control Intent Matching

Dialogflow — Priorities for an Intent on the Console

Lastly, priorities allow us to preferentially match certain intents when a group of intents are more or less equally valid and share the same input contexts. A value between 0 and 1,000,000 can be assigned to the intent’s priority field, where a higher value denotes higher priority. (They can also be set to preset values on the console, as shown above.)

Some of the ways we made use of this feature includes:

• Deprioritizing the initial Start intent below all other intents such that it would always match an existing user-specific intent over it.
• Prioritizing intents that are trained to recognize return phrases (e.g. ‘Return to Main Menu’). This is helpful in flows where the next intent to be matched accepts @any input from the user such as an order barcode with which to search for details. To prevent the return phrase from being matched as a barcode, we make sure the return intent has a higher priority instead.
• Disabling a particular flow (e.g. a flow for customers to reschedule their deliveries) by setting the priority of the initial starting intent (Customer — Reschedule Delivery) that leads to all other intents in that pathway as -1. In the future, when that flow/feature is to be made available, we only need to reset this priority to a positive value to enable it as the backend handling code for those actions already exists in SNS.

This penultimate part wraps up most of the implementation details, and the final article will cover how we tested our bot, and what’s in store for NinjaChat in the future.