Cheating on the Turing test

8 Dirty tricks to make your chat robot appear more human

When we first set out to build our assistant robot at Meekan, we started by teaching him to carry out some basic scheduling tasks:

We thought, if you’re installing an assistant robot to schedule meetings, this is what you’d expect him to do. Work. But when we let people test him out, conversations soon started looking like this:

Conversations would escalate fast, people would call him names and insult him, and generally come out of the experience assured that SkyNet, together with the whole robot overlords thing, is not happening this century.

We realized that we need to make him smarter. Make our robot appear more human, so you’d feel almost like you’re actually talking to someone on the other side. Now, I say “almost”, because creating a robot that can carry a real conversation is quite difficult, and it’s not really what we’re trying to accomplish with Meekan. We want our robot to perform his assistant tasks well, learn and anticipate his master’s habits and preferences. Delighting his users with witty conversation is just a nice bonus.
So we designed and built in a few traits that will allow the robot to appear more intelligent without sacrificing our entire development resources for this. It’s not going to win us the Loebner Prize, but it’s surprising how well it works.

Get the basics right, don’t be an idiot

First thing’s first. Before adding the fun stuff, make sure the robot is doing his job. If someone asks “Meekan, arrange a team meeting tomorrow afternoon”, suggesting early 7:00am as a possible afternoon time will not seem very smart.
Or when your teammate John is in Beijing, 16 hours away, “Quick phone call with @john” should make sure we’re not going to wake him up at 3am his time.

Now you may respond to the unrelated things

Behind the scenes, when you’re talking with Meekan, he sends the entire line he got to his natural language understanding (NLU) engine, to figure out what you’re asking him to do. So if we take one of the examples from above, it will go like:

The engine chews up everything, and spits back the request’s main Intent, together with the data entities that came along with it.

So you got your work-related intents: “New Meeting”, “Find a Flight”, “Reschedule”. But now we can add some extra, less-obvious ones like “Greeting”, “Compliment”, and, well, “Insult”

Classic TJLYOM

That’s so much better than the earlier “I don’t understand” conversation killers. Now let’s take it up a notch.

Build a response bank to withdraw from

Giving a relevant, on-topic response is not worth it if you keep repeating it over and over again. Having just one response for every situation quickly reveals the mechanism underneath. Really, it takes just one repeat to break the illusion:

Now you broke it

So, for every intent the robot identifies, you need to have a whole list of responses, and let the robot choose one at random (or even better, pick the best one based on specific key words inside that intent). It’s not uncommon for us to write 20–30 different possible responses for the more popular intents. It’s a very quick way to appear intelligent in matters that are outside the core functions the robot should do.

Here, the robot gets two “Out of scope” requests, for things he can’t do, but picks a different response for each of them:

Use ambient information

Using a bit of basic information you have about your user, can go a long way in appearing knowledgeable and in context. If you know the user’s name, use it. Use knowledge about the local time or weather, whatever you can get easy access to:

It’s 5 AM

Maintain the illusion of a physical being

Wherever possible, pretend there is a real, physical humanoid robot sitting on the other side, and feverously typing away on his little clunky chrome keyboard. Refer to your robot as a robot (not “bot”), and as a “he” (not “it”) [We would rather present our assistant as a he-butler, even if just to avoid the annoying trend of giving virtual assistant a female name. But let’s get back to the topic].

When you answer back with a long response, don’t just send it all out in one go. Imagine that you’d have to type it line-by-line, and pace your answer to make it look as if it has to obey physical constraints (typing with fingers), and less automatic.

If it takes a bit of time to come up with an answer, you can also send a temporary “Meekan is typing…” message, the same one you get when a human on the other side is writing a long answer (Credit to Ben Brown for this one)

Stay in character

Your robot needs to have character, and stick to it. Mixing in multiple personalities will make it seem like you’re talking to a random customer service rep on the other side.

Part of the original character draft we wrote

When you have a good, solid description of the robot’s personality, it’s much easier to pick out which words he might use, and how he will respond to various situations. If you have a whole team that handles that, it’s the only way to make sure they all write for the same robot.

Know your limits

From time to time, the robot will be asked to perform a critical task. This could be something like

  • Buy stuff with the user’s money
  • Delete something important
  • Communicate with other people on the user’s behalf (and potentially make him look bad)

You should be 100% sure that you properly interpreted the user’s request. The best way we found, is to mirror the request back to the user, so that it’s clear what the robot is about to do, and wait for a confirmation.

Meekan got the right meeting, but still takes the safe side

If you’re introducing some new capabilities, and your language understanding is not fully trained yet, don’t be afraid to acknowledge that. Saying “Sorry, I’m kinda new at this, did you mean to say that…”.

This obviously comes with a cost: you’re now being overly verbose, and slower to act, but the trade-off is clear. If we cancel an important meeting because of a misspelled word, or some weird glitch in the language engine, the robot will appear not just plain stupid, but harmful stupid (read: uninstall-him-right-now stupid).

Set expectations

The nature of instant messaging forces some limitations. For example, since you can’t present a modal dialogue (you can’t put everything on hold until you receive the user input you’re waiting for), you’ll have to explicitly say when you’re expecting some special input (rather than present a plain, annoying “I don’t understand”)

This could actually get even worse. This is the nature of the medium. Your user may have went out into a long meeting, and now he comes back ready to start something completely different. Maybe an urgent notification breaks the conversation, needing immediate response.

We’ll talk about this more of these in the next post on this series, where we’ll discuss how to translate common UI patterns into a conversation.

Our robot, Meekan, is available for Slack and HipChat, go ahead and try him out. He’s not going to be our supreme overlord this year, but he’s pretty good at scheduling meetings.

More reading:

  • The Jack Principles: Outline for an Interactive Conversation Interface
  • B.A.S.A.A.P: Be As Smart As A Puppy. “Making smart things that don’t try to be too smart and fail, and indeed, by design, make endearing failures in their attempts to learn and improve. Like puppies”

This is a part of Meekan’s journey to create the digital office manager. Read more stories here
Written with Eyal Yavor, Meekan’s CTO and co-founder