Wizard of Oz testing voice apps with Tortu
While designing a voice application you should understand how real people will use it. The sooner you make a user testing, the better you will be granted to find the weak points of your application and it gonna be easier to correct them.
If you aren’t experiencing failure, then you are making a far worse mistake. You are being driven by the desire to avoid it.
I will tell how Tortu helps to do WoZ testing with medium-fidelity prototype before you have written the first code line.
Wizard of Oz
Wizard of Oz (WOz) testing occurs when the thing being tested does not yet actually exist, and a human is “behind the curtain” to give the illusion of a fully working system.
Cathy Pearl, Designing Voice User Interfaces (2016)
We will pretend the application is absolutely ready when we actually have only the medium-fidelity prototype. Then user will speak to it being sure it’s working. Also we gonna simulate the work of the app.
Overall the process will looks like we are starting the prototype, the user speaking to it, he can hear the answers and also giving his answers and we are choosing what the prototype should say.
Before the test
Let’s suppose we have already designed the basic flows. Then we decided to invite some users to try our application. We already have the examples of the dialogs, we drew a flowchart and started the work on prompt lists.
Considering that everything except the dialog examples are inside of Tortu, we can only recheck everything and start the prototype.
The prototype possibilities:
- Steps voicing. It’s more than 10 languages, 20+ voices, speed and pitch tuning.
- Adding new steps right at executing prototype in case you’ve found unaccounted scenarios.
- If you have variety of phrases you can choose any of them.
- Undo/Redo. Do you want to come back to another dialog branch? You can rewind the conversation back.
- Support of SSML.
We are planning to add:
- [DONE] Support of audio files.
- Support of slots.
Before the start we must have a look at how the prototype works to prevent the unexpected occasions at testing time.
During the test
So, we have completed the preparation and the user is sitting in front of us. We’ve already spoken to him and told what is gonna be next. So let’s start.
Step 1. Set the user the way he won’t be able to watch your screen, but have a possibility to hear the sound of your computer. So this way he may be thinking the system works itself.
Step 2. Start the application’s prototype. Choose the language, set the voice and tune the pitch and voice speed if needed.
Step 3. Play wizard. Listen to the user and choose the necessary variants at the prototype. The application’s text will be voiced.
In the prototype process there may occur the incidents you didn’t accounted some details at your application. And it’s clear if you want to repair them right now. Tortu have a possibility of adding new dialog steps right in the prototyping process.
After the test
Test every scenario this way you planed to do, don’t forget to analyse the results.
It’s well if you have an opportunity of video recording your test session. So you can not only analyse the user-system dialogs but also users behaviour.
I’m sure the user testing is an important part of the VUI-applications designing, however much people avoiding it cause of its difficulty. In Tortu we tried to simplify as much as possible the process of WoZ testing.
We always looking for the ideas how to improve Tortu, that is why every feedback is much appreciated. I’ll be glad to speak to you in person at p@tortu.io.
P.S. I recommend you to read “Designing Voice User Interfaces” by Cathy Pearl. There is at Chapter 6 a specified analyse of user-testing process of voice applications and not only with WoZ methods.
