How will your bot know it’s morning?
Taking your chatbot training to the next level
Let’s start with a little example. Check out Siri, Apple’s pride and joy, back in 2012:
Not very impressive.
But look at her now. This is from 2016:
Seriously, this really happened in Australia. Siri saved a baby girl’s life: The baby was in trouble, her mother ran to help, and yelled at Siri from the other side of the room. Call me an ambulance!
And Siri did just that.
So how did this happen? Was there a major technology breakthrough in language processing between 2012–2016? No. What probably happened is that Tim Cook called up one of his engineers and said “Look! they are making fun of Siri on the Internet!”, and the engineer just made sure that when you ask Siri for an ambulance, she’ll do the right thing.
And you know what, this is really good news. The technology field for creating chatbots is level. Everyone, including you, has access to almost the exact same technology, and that technology is quite limited. Bots still cannot have a natural, flowing conversation with their users straight out of the box, and a whole new field of conversation design has emerged from that need.
Building a great chatbot is a design challenge
So let’s talk about squeezing as much as possible from this tech. If you’re following our journey (you should), we’ve written about our conversation design experience building out scheduling assistant, Meekan. The underlying assumption was that we got the language processing part covered: either the bot understands (or at least, parses correctly) the user’s intent, and proceeds to do his job, or he doesn’t understand, and we try to talk our way out of it, back into the safe zone. But zoom in into the first part, and you discover that this game is about more than just parsing. Let’s see.
Basic training
The user just said something. Your first step is probably sending his words to a natural language engine (LUIS, wit.ai, api.ai or maybe even your own home-grown neural network). You feed your engine a sentence on one side:
…and if everything works properly, you get an intent and entities coming out from the other side:
In this case, the engine would identify the intent as “New meeting”. This is what the user is asking you to do. And the entities identified would be title:”team meeting”, day:tomorrow, time:morning.
If you’re dealing with something common, like saying “hello” or asking about the weather, you could be in luck and your engine may already be pre-trained for these. This means it’ll recognize them correctly straight out of the box, thanks to the work that was already done for you by the engine’s designers.
If this is not the case, that’s not so bad. Just come up with some more examples like this:
Schedule our Project Launch for early next week
I need to meet Jim before next Thursday
Quick coffee with the sales team early morningand start training your engine. Feed a sentence, watch the engine’s answer, and give it feedback: This is correct / This is a mistake.
After around 100 examples, your engine is going to get much better at doing this (depending on the quality of your engine, and the complexity of your topics). Now it’s time to start putting it in the hands of other people. Prepare to be shocked by the weird commands they’ll give your bot. Things you’ve never thought about. Relax. Just feed everything into your engine, and keep correcting its mistakes.
Eventually, when your robot is on the air, in front of real users, you can turn this into a daily process of checking what went wrong, and incrementally improving your bot’s language skills. At Meekan, we’re now at over 6,000 start-a-new-meeting sentences and still going. How many do you have?
Focus on a narrow domain
Really, it can take thousands of examples to train your bot on just one, seemingly simple, subject. So unless you have huge resources, or access to mountains of data, you should really stick to becoming an expert on one narrow subject.
As soon as you expand your bot’s mission to cover more types of tasks, you’ll find yourself constantly fine-tuning the engine’s intent parsing, with some of your training sometimes moving things backwards instead of improving.
For example, we gave Meekan the ability to find cheap flights. As soon as we started training the engine for this, we’d see the two intents clashing:
Meekan, Add a flight to New York to my calendar on Thursday This is obviously a “New meeting” intent — the user is trying to put this on her calendar, to make sure she’s blocked as busy for the flight day. But the engine would constantly insist that this is a “Find Flight” command, and would proceed to look for Cheap flights to New York.
If your engine is off-the-shelf, it might limit your ability to tailor it to your exact needs. And as you throw more tasks at it, you could eventually hit a wall.
Jack of all trades, master of none applies here. Don’t be Jack.
Keep context
Memory is a part of any conversation. Some sentences have no meaning, unless they are examined in the context of what was said before. Some bot engines have that capability built-in. In others, you’ll have to implement it yourself:
When can I meet Greg next week?You're both free on Tuesday. 10 am seems like the perfect time.Okay, book it
Here, “Okay book it” actually means: Add a new meeting for me and Greg next Tuesday at 10 am. If we didn’t remember the time, date and invitees we just discussed, the phrase “book it” alone would have no meaning.
What's the weather tomorrow?It's cloudy with a slight chance of rain.How about Friday?
“How about Friday” could possibly mean the user is asking for a new meeting, except the context was just set to the local weather.
Watch for local and cultural differences
Imagine half of your team is located in San Francisco, and the other half in Berlin. Now someone asks your bot:
Meekan, schedule a quick call tomorrow 3pmWho said that? If she’s from San Francisco, it actually means Midnight for the Berlin people. And SF’s “today” is sometimes their “tomorrow”.
It's 20 degrees outsideThat’s nice. In Berlin this means T-shirt. In the states, 20°F is below freezing!
How about some more subtle examples?
Next WednesdayIf today is Monday, when is next Wednesday? The day after tomorrow? Or the one next week, 9 days from now? It turns out that people from different countries will pick different answers to this question.
How do you handle these differences? Either you pick a side (eg. Be an American bot), or learn the location of every user you speak to, so you can guess what they mean.
Whatever you pick, always repeat what you understood, so the user can correct any confusion before it turns into disaster:
Set up a meeting next WednesdayOkay, new meeting on Wednesday next week, Sep 26No! Cancel. Set up a meeting on Sep 19
Bot nirvana: personalization
A word can mean different things to different people, even if they are from the same place, sitting in the same room together. Ever had an air-condition war at the office? So how should your language engine translate “Set the A/C to a nice, cool temp”? This could mean so many different things, depending on your location inside the office space, your preferences, and of course, your gender.
You could have your bot ask the user to specify an actual number for the temperature, or just decide something and inform the user about it.
Or, you can guess what the user meant. Make the best guess possible by collecting any piece of data you’re allowed to access. What’s the user’s gender? In which workgroup, IP address, workstation ID, floor, building. What’s the temperature outside right now? What time is it? Is the sun showing? Gather as many signals as you can, compare to the past decisions you made, and make a good guess. Hire a data scientist, they are particularly good at making guesses like this.
Look at this heatmap. It’s the average calendar of all the people we know in Rome, Italy. Blue means no meetings at this time (This is why the weekends are colored blue). Red hot means busy:
By asking Meekan users permission to analyze their calendar stats, we can understand what they mean without asking. If you have the classic Roman calendar, when you say you want to meet someone “in the morning”, we know you mean “9 or 10 am, 11 is okay. 8 am if you absolutely must, but never earlier than that please”.
We can describe groups of people (called “clusters”): People who like to have their meeting as early as possible. People who must have breaks between meetings (and those who like to bunch them together and get it over with). And if you fit into one of these groups, we can guess that you need a break after every meeting, and make sure you get one, without ever asking.
The perfect assistant knows you better than you know yourself. And this is where bots should aim to go.
A word about privacy
The data collected by bots should only be used to better serve their users. If you treat your user like they are the product, take the private insights you gathered about them, and sell them to the highest bidder, you are betraying their trust.
Bot makers must keep their user’s info private, anonymized and possibly encrypted at all times. By default, the user assumes you’re respecting their privacy, and abusing that trust affects the entire bot builder community.
We have this responsibility, let’s keep it clean.
—
This is a part of Meekan’s journey to create the digital office manager. Read more stories here, or check out meekan.com
Written with Eyal Yavor, Meekan’s CTO and co-founder, and helped by the crazy skills of Tim Nonner, and the Tamedia Data Analytics team. Heatmaps implementation by our own Ela Avrahami
