Last April, Facebook announced Messenger, its latest platform. There is nothing like a good make-away to try new tools and techniques. Apurva Shah shares that enthusiasm and we set out to create a bot over a weekend.
We paced around Apurva’s dining room table, a place where we have anguished and hacked on many products together. The blank whiteboard taunted us. We realized that to test behavioral products, one needs to operate within their day-to-day life. We decided that we would treat our early prototype simply as a ‘science experiment.’
By thinking in terms of experimentation it provided us the lightness to consider a core problem and develop a hypothesis rather than foreshadow the complexity of a real product experience. This insight unlocked the rest of our session, the work began to flow more easily. In the same spirit of following the scientific method, we focused on answering a key question — we call our “Minimum Viable Question” our MVQ.
Are people’s financial decisions able to be influenced by providing a single, timely piece of data everyday?
Now that we have a MVQ, what experiment could we craft from this question and who would be our guinea pigs? We decided to focus on people like us — millennial minded, tech savvy, financial luddites. Messenger as a communication channel is already woven into the daily life of these users. We felt this simple notification would be a “low cognitive load” method for users to quickly understand their past spending and have their allowance amount top of mind as they go through their day.
Ben Bot, named after Ben Franklin, is a Messenger Bot who provides lightweight, daily financial advice during your busy day. He is the Fitbit for your daily allowance.
We imagined a scenario where when the user wakes in the morning, he would get a notification from BenBot. “Morning Jason, today, you have $32.00 to spend. You are doing a great job saving!”
Chatbots do not offer an opportunity for visual design. The entire app experience is trapped inside bubbles. During our brainstorm we began to wonder how design could impact the creation of a bot. We decided design for a bot meant character design. In this light, we wondered who should our bot personify? What if the bot took the tone of a founding father? Ben Franklin began to feel like the right fit as we began playing with some of his more memorable quotes.
Collecting Live Data
Once BenBot’s first feature was complete we took him out for a test drive and began inviting friends to try it out. This phase of the process really highlighted just how good a chat interface is for experiments. Users could tell BenBot their income, attach a credit card and then receive their daily allowance. This feature already felt like magic compared to the other tools we had tried for budgeting.
Once testing was underway we committed to refocusing ourselves to a single conversation we could craft that would have a big impact on someone’s life. It was with this idea that we arrived at starting with sharing someone’s daily allowance. After all, we had a fully functional prototype in users’ hands after a few hours of work. This wasn’t a mere Wizard of Oz or clickable prototype — BenBot is the real deal!
One of the nice parts about using Messenger as a prototyping tool is seeing the conversations in real-time as our users were conversing. I could even interject as Ben from the chat interface. This allowed me to aid in user’s first time experience. For example, I assisted a few users in lowering their allowance as they seemed too high for a daily spendable amount.
We believe you cannot test behavioral products without operating prototypes in customers’ daily lives. It is a common design research process to try and test prototypes with users, but sometimes it is difficult to conduct research in their homes, offices or along their commutes. So product teams settle for testing in our offices or research labs. But to truly affect someone’s behavior we realized we must operate within their daily life. Chatbots provide a glimpse into how we can test real product elements in the form of flexible conversations.
We ran the test for about two months with twenty users. It was exciting to see that a one-day make away yielded such a rich data set. Compared to their pre-BenBot behavior, we saw on average 10% drop in spending. More importantly, we saw a tighter variance around their daily spending allowance. We attributed this variance to the daily notification they received each morning. We are excited with these early results and hope to continue further explorations.
It isn’t a big leap to see that new tools can be developed within chat interfaces to evolve what is considered a conversational interface. Today, Messenger allows chat blocks to be composed of image or text. Imagine adding new content blocks and keyboards for sharing new types of data for our bots to shape our conversations.
The bottom-line is we believe that a thoughtful product development process starts with a kernel of an idea focused around a minimum viable question. This MVQ will be on the path to being answered with an experiment. In all likelihood, it will be several experiments that each bring with them answers and more questions as teams of makers dive down the rabbit hole leading them bit-by-bit to real understanding of a problem domain. This process is iterative and truly based on “test and learn” with minimal confirmation bias.
We hope you found the story of our BenBot experiment useful. Have experiments, methods or design tools you’d like to share with us? Please leave us a comment.