Conversation Driven Design

With Botsociety and OpenAI, a new conversation design approach

Ever since the guys at Rasa started to talk about Conversation Driven Development, we started to ask ourselves: Is Conversation Driven Design possible? Up until today, the answer was: Not really.
In particular, what you needed to do was something along the lines of:

• Design a conversation
• Build and deploy it
• Give your chatbot or voice interface to real users
• Grab feedback
• Back to square 1

Even if you accelerate your workflow using Botsociety for step 1, and say you release an unfinished version on step 2 just to test it out, it still takes say half a day to complete the whole thing. You can’t rely on this process to make high-level design decisions, like Which use cases should we tackle first? Or Is this persona effective?

This is different than what you are used to in the UI world. For example, designing the first web page and then showing it to someone on Slack. That takes say 20 minutes, not an entire day. That speed allows you to try designs faster, iterate more, and opens up a lot of possibilities.

In the same way, a real Conversation Driven Design has to be fast enough to allow designers to:

• Design only one simple path, maybe even only the happy path
• Run a test
• Gather test data
• Design another path, iterate

And it has to take around 20 minutes as well. But how do you do that? You have two major problems:

1. No engine The Natural Language Processing engine is not there yet, since you just started designing and you have no data.
2. Time Giving feedback on a conversation design takes a lot of time compared to a UI. Every tester needs to experience the whole conversation multiple times to give proper feedback

To overcome the first problem, designers usually rely on the Wizard of Oz test. That is: The designer acts as the bot, making the choices that the Natural Processing Engine can’t do — because it does not exist yet.

Wizard of Oz it’s ok, but makes problem 2 (Time) even bigger. Now you not only have to wait for the testers to try your design, but you even have to attend every testing session to act as Oz. You can’t run a bunch of tests at once, in parallel. Etc.

The solution: Auto-testing with Botsociety

This is how it works:

• You start designing
• You click a button
• We ask OpenAI to act as the human and test your design
• You gather the feedback and keep designing

And now, this takes less than 5 minutes. So you can really proceed in an iterative way, and test every step of your design multiple times, basically while you are designing it. Plus, you can always bring a human tester on board to get even a better feeling.
Once the data is in, you can review it in the context of your design and:
Add the testing sentence as a new utterance
Use the testing sentence as a new path (or use case)
Discard the data

So now all of sudden you can do stuff like:

• Design a happy path and decide on dedicated paths based on the testing
• Design two use cases and then decide the priority based on the testing
• Design the same interaction with two different personas and see what behaves best
• Not write any utterance and just generate test data to populate your utterances

If you want to try it out yourself, sign up on Botsociety here.
Also, we are having a workshop in February on this topic, you can RSVP here.