A6: Behavioral Prototype

Scenario

Your challenge is to build and test a behavioral prototype. Design a new chatbot application that uses text recognition from users chatting on a mobile device or computer to accomplish a task.

Prototype

My team decided to prototype a chatbot using Facebook Messenger that would help people find a place to eat for a date. Our idea is inspired by a Buzzfeed video about people testing out an app that allows a person to have a fake significant other who would sent them periodic texts throughout the day. We decided to use Facebook Messenger because there are actual chatbots implemented on this platform and it would be easy to run a user test, as the “wizard” could be on the other end of the chat interaction with the user.

To simulate the chat experience, we developed a set of questions, some free response and some with multiple choice options. Sometimes the next question would be prompted depending on their previous response. To make the experience more realistic and computerized, the wizard and copy and pasted the questions into the chat to speed up the process so there wasn’t much typing done on the chatbot’s end.

As for the evaluation session, we decided to have the facilitator accompany the user while there would be two wizards in the background chatting with the user and preparing a list of restaurant suggestions. The suggestions we sent were the top search results from Yelp based on the user’s responses.

Design

We designed our chatbot to be simple and decided that it would be able to make some assumptions depending on the user’s responses. Also, it would be able to interpret the user’s responses, even if the grammar and spelling may not be 100% correct. To keep it simple, we limited our chat interaction to 10 questions and provided a users with an “almost there” message at the halfway point to ensure them that they are almost finished.

We conducted 2 user tests with college students and gathered some feedback regarding how their experience went. Overall, it went smoothly with a few points of confusion such as whether or not they should respond when the chatbot makes a statement. They felt that the questions were the right length and right amount. The experience seemed to be simulated realistically, as the chatbot responded quickly and it almost seemed computerized.

Chatbot messages and interactions compiled behind-the-scenes.
Example of the chat interaction in Facebook Messenger.
Sketches for how the video of user testing will look like.

Analysis

We conducted 2 user tests and asked them for feedback regarding their experience. In addition, we presented our chatbot to the class and received some feedback. Here are our findings:

What worked well:

  • Straight-forward information
  • Clear directions
  • Chatbot provided quick responses
  • Chatbot was able to recognize information in lots of text
  • Restaurant suggestions with reviews were helpful

What needs improvement:

  • making it clear whether the user should respond or not
  • reducing the creepiness factor
  • not showing the testing messages in the chat
  • make it more clear when the bot is typing
One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.