Storytelling with Chatbots

Lessons in Content Mapping

Colten Jackson
Oct 6, 2017 · 8 min read

In Neal Stephenson’s Snow Crash, the protagonist opens a file so large it overwhelms his computer and says, “You must have half of the Library in here!” His compatriot replies, “And a librarian to boot.” The dream of a computer personality that helps us sort through massive troves of data is closer than ever, and Annalect Labs has been exploring the technology to program such a personality.

Our motivation was not original — we wanted to make data work accessible to a wide range of colleagues, without forcing them to learn SQL, explain how they can run commands in the terminal, securing credentials for databases and keeping dependencies up to date. We could have gone the standard route and built a point-and-click front end for running tasks, but it takes a lot of time to plan and design a GUI and roll out the training for every new tool that is programmed at Annalect, so we thought, our data-fetching programs are already a text interface, let’s take from Stephenson’s librarian and wrap the program in a personality.

Annalect Utility Bot Interpreter

We named our librarian AUBI for “Annalect Utility Bot Interpreter.” The main purpose of AUBI is to help users find the program that retrieves the data they want, and has the authority and credentials to run it on their behalf. Additionally, we wanted AUBI to provide documentation on request, and allow Annalect Data Scientists to give AUBI the ability to recommend and run their new tool without writing additional narratives and learn from what users are asking for.

Inspired by the far-future vision of building a data genie/oracle we prototyped some smart AIML scripts to create the illusion of intelligence, but our users were underwhelmed. We went back to the drawing board (and kept going back there many times), realizing that planning a response every user input is wouldn’t match the dream of building an intelligent data butler.

In the end, we learned that people can’t resist an adventure — a story with a narrator to help them along the way. So we wanted to share our mistakes and discoveries in hope to save time for bot masters and narrative designers and contribute to the dream of bringing data-librarians to other domains and industries. So here are ours “5 Lessons for better UI with chatbots”:

Lesson One: Pretending to be Intelligent Isn’t Helpful

Our first approach to writing a chatbot personality was to anticipate all the different questions someone might ask, and tell the chatbot what to say in response. We started with a primary goal of ‘appearing intelligent’ and the natural language engine we used1 had great tools to be clever about this. For example, an oft-used strategy for chatbots that didn’t understand an input is to answer with a question, or to change the subject entirely. This happens often in any conversation and we had fun getting unexpected responses from our early version of AUBI, who might reply with quibbles like “So nice that we agree” or “I’m not a walking encyclopedia you know” — the bot had surprisingly relevant responses in the course of a normal conversation.
However, we abandoned this strategy for a couple of reasons:

  • Users who came to the bot knowing what they wanted to do were annoyed when AUBI changed the subject. Maybe it seemed clever, but it was also ‘easily distracted’ — not a great partner for data science.
  • Users who didn’t know what they wanted were met with a blank input box with little guidance on what they could say. If they asked questions we didn’t anticipate in our script, AUBI would quibble, and the user wouldn’t get much guidance on what they could ask.
  • Adding new utilities to this ‘answer every question you can think of’ chatbot necessitates writing more narrative for every new capability, and we didn’t intend on hiring someone to write chatbot narratives full time.

Lesson Two: Adventures Need Walls

So AUBI was able to have the right answer to a selection of questions within its expertise, but wasn’t great at expressing what its expertise was. In brainstorming what kind of conversation flow could be an improvement, we were inspired by the old choose-your-own-adventure game Zork (ca. 1980). Now, Zork has no illusions of intelligence. If your input wasn’t anticipated, the program just repeats itself until you say something that makes sense (go west, open the box, etc.).

These conversational ‘walls’ keep you on the timeline of the game and make it obvious what the program does and doesn’t understand. Instead of being clever, the blunt interaction trains the user what they can and can’t say.
Because of the extremely limited set of valid inputs, choose-your-own-adventure games take advantage of leading questions or exposition that implies the next step.

We decided AUBI should follow these footsteps — we re-wrote the narrative so that every message from AUBI implied or explained what you could say next. This had three benefits:

  • New users understood what the tool could be used for at a glance
  • We could turn keywords into links that short-circuit to other parts of the narrative, kind of like seeing that you can ‘Turn to page 22’ for one option or another
  • The author of the chatbot is given the power to direct a user through the pre-programmed expertise. The user can walk through gentle transitions in order to navigate to the part of the bot’s expertise most relevant to their needs.

Lesson Three: Allow Walking in Circles

In a good story there is beginning, a climax, and a finale. “Choose your Own Adventure” books and programs are allowed to have multiple attendant plots (middles) and hence multiple endings. What we found out is that there is no-ending to a chatbot. We had to come-up with multiple transitions, to allow users to navigate to “plot-intersections”. If they were done with their adventure, they would simple close our chatbot. However, if they wanted to keep exploring we had to give them the ability to loop though. The caveat was not to navigate to the beginning, but navigate them to a certain intersection of the narrative. For AUBI, this manifests as the option to run the same tool again, or dig into documentation for that same tool, to choose a related tool, or to choose a different goal altogether. This improved on early adventures which came to an end and required a user to restart the application to get to a different part of the narrative. Instead, we allow users to walk all over the narrative and visit every possibility without starting from the beginning every time.

Lesson Four: Computers Let You Undo

Our users did not have any illusions that they are talking to a human, so unlike a regular human conversation, they would type in commands like “Go Back” and “Start Over” if the narrative story did not unroll as they expected it to. The “Back” button on your browser or “Undo” buttons in the software trained users for many years to easily get back if they did not like the output or accidentally made a mistake. It looks like chatbots, while able to handle flexible input from the user, are still pieces of software and people expect that they can undo their actions.

Lesson Five: There’s an Algo for That

At this point we had built a narrative that allowed for exploration and execution of a few tools that people were already using, but the original intention of building a text-based conversational interface was to allow quickly adding new capabilities. We were able to refactor our conversation to include fill-in-the-blank templates that updated themselves from a spreadsheet hosted online. This way, content producers and programmers could edit the tools and their descriptions without having to edit and debug AUBI’s narrative program.

But this left us with the challenge of writing a narrative that could recommend tools that we didn’t know about yet. For this, we explored the use of semantic space, which is one of many strategies for the age-old problem of ‘document retrieval.’ In our case, the document was the description of a program, but still we didn’t have to re-invent the wheel for this.

The result is a hybrid of a pre-written narrative with limited pathways and an AI engine that tries its best to suggest a relevant point in the narrative to jump to. We are able to accommodate the different approaches we noticed people take with our AUBI prototypes:

  • Return users who can click their way through keywords to execute the tool they want right away
  • Users who want to follow the narrative to explore its expertise and abilities
  • Users who have an intention they want to fulfill, but don’t know the name of what they want

The ‘document retrieval’ which addresses the last point is the component that stands to be most improved through training the bot over the course of many conversations. Users are encouraged to give thumbs up and downs to AUBI’s suggestions, so that it can have more information to compare to new input when deciding what to suggest.

Our adventure of building AUBI has us imagining chatbots as a new form of communication, but it is the same old story of information design. A successful chatbot is first of all a good storyteller:

  • He can tell you what you want to hear but at the same time take you “information- places” that you didn’t know existed
  • allows you to participate in the conversation (instead of sounding as monotonous orator), while still being in control.
  • Leads you to end destination of the story — he understands that the user came not from boredom, but because he/she wants to accomplish something

And while the chatbot industry can debate which algorithm solves better for user input, it is our core belief that a good chatbot design starts with a good story.

AUBI was created at Annalect Labs. Annalect Labs is a rapid prototyping and experimentation department at Annalect, with a purpose to build fast, stimulate experimentation and push forward r&d within the company.

Annalect is the Data, Technology, Analytics and Consulting division of Omnicom Media Group (OMG), sitting as a foundation of expertise. Annalect manages all technology development, data management and deep technical expertise to support client embedded teams in the OMG agencies and on individual agency client projects.

Colten Jackson

Written by

Software Developer in Urbana, IL