Why Chatbots still have a long way to go (Hint: It’s more than NLP Intent Inference)
Follow my publication at DickeySingh.com
To a business, a chatbot is a self-serve channel enabling its customers to access information and trigger task models interactively.
A chatbot is an exciting technology but, technology should never get in the way of solving a problem or providing a memorable experience to users. People want to “book a flight,” “return an item,” “play a playlist.” No one says, “I am going to book a flight using a bot.”
Chatbots that can get a task done with less resistance compared to traditional ways will have much higher engagement, acceptance, and success.
Certain tasks are best suited for chatbots whilst others are better suited for apps or websites. Further, chatbots may work great for a particular segment of the population, but not all.
For Chatbots to be successful, they need to convert natural language to intent effectively, convert intent to queries for backend systems and respond to users with engaging and helpful answers.
Natural language Processing and Intent Inference
There are numerous solutions for Natural Language Processing (NLP) to determine intent ranging from traditional rules-based to deep learning based. Examples include Stanford CoreNLP, Natural Language Toolkit (NTLK), Apache OpenNLP, Recurrent Networks or Recursive Neural Tensor Networks.
You can use a software library, a platform or a service for NLP. I won’t spend much time on this except to say that the new Recursive Neural Tensor Networks based NLP is significantly effective and efficient compared to traditional rule-based NLP.
Realtime, Engaging and Helpful Chatbot Responses
NLP technology aside, the key differentiators between chatbots is the ability to respond to users in realtime with helpful answers.
Let’s take an hypothetical question that a user may ask various implementations of an e-commerce chatbot. “When will my order arrive?”
When will my order arrive?
- Your order will arrive on 7/11/2017
This is a valid and to the point answer, but it is certainly not personalized. Personalized conversations increase engagement and retention. So let’s see some other possible answers.
2. Karen, your order M735232 will arrive on 3/28/2017
This answer is certainly personalized, as it used a name and the order number along with the answer. This is an improvement over the previous unpersonalized answer. However, “Karen, what would you like?” works great when the barista remembers your name, but not so much for a chatbot.
3. Your package with Philips Sonicare toothbrush and two more items will be delivered tomorrow by 7 PM via UPS Ground.
This answer is meaningful as humans remember orders by purchased items, versus order numbers. It is helpful as it is also giving a delivery timeline along with carrier details.
4. Your package with Philips Sonicare toothbrush and two more items will be delivered tomorrow by 7 PM via UPS Ground. Would you like to see UPS tracking details, set up a return or see accessories like travel charger or wall mount?
This conversation is concise (two sentences and about 40 words). The chatbot is keeping the user engaged with leading questions, and is helpful as it is solving a customer need. It is also upselling additional items making it win-win for both user and marketer.
It is also defining the scope of a chatbot in it’s answer and suggesting keywords that the chatbot understands.
The last response in the previous section is useful because of two reasons: access to all relevant data systems in an instantly consumable format, and meaningful personalization. These two make up the key differentiators between great chatbots from good ones.
Access to Systems
Chatbots cannot personalize with meaningful information and/or keep conversations engaging if they do not have access to data sources in a consumable manner.
The Chatbot needs to have access to the order management system, along with shipping and carrier details. Chatbots that are not integrated with payment systems cannot accept payments over chat.
User Profile
Marketers create user profiles based on usage, behavioral, psychographic, geographic, and attitudinal data for behavioral intelligence driven marketing.
Chatbots should have access to same user profiles, but in a chatbot consumable way.
Context
As in any intelligent system, context for a Chatbot can be derived from
- Current conversation: E.g. Julia may be interested in a telephoto lens
- Previous conversations: E.g. Julia owns a Nikon camera body
- User profile: E.g. Julia is a pro-consumer and an avid photographer derived from search and purchase history
Context from previous conversations is very useful for personalizing answers and filtering out incompatible products.
Julia: “I’m looking for a telephoto lens”
Chatbot: “Are you looking for the telephoto lens for your Nikon D5 camera?”
Up-selling is much easier when the context is well understood. For example, a professional photographer is more likely to buy a protective filter for their telephoto lens.
Fallback strategy
Chatbot technology is still not 100% accurate. Chatbots learn over time and cannot be perfect on day one. Chatbots should have a multi-level fall back strategy:
- Switch to a user interface to ask multiple choice questions
- Request a real human being to assist
- Offer to search based on keywords and known context
Outro
We expect chatbots to be conversational and communicate the way humans do. At the same time, our expectations from chatbots are much higher than humans as machines are assumed to answer faster and more accurately. Chatbots that have access to all systems in a consumable way can provide meaningful and engaging responses to users.
