A6 — Netflix Wizard of Oz Prototyping
Wizard of Oz Prototyping or Behavioral Prototyping is technique used by designers to test a concept where implementation might be too costly or infeasible.
In WoZ Prototyping, participants are immersed in an experience mimicking the intended post-implementation experience and are made to believe that their experience is genuine. Meanwhile, a “wizard” acts as a human computer, controlling the product by reacting to the participants behavior. For example, in a concept test of a voice translation system, a participant might speak into a computer whereby the trained wizard manually translates their speech back to them via computer. This eliminates the need for costly implementation and tests the basic concept in early stages. The participant remains unaware of the human component and is made to believe the computer is the one actually translating their speech. In remaining unaware, the participant is able to give genuine feedback. Like the Wizard of Oz, the wizards are the true mastermind behind a believable, livable product experience in WoZ prototyping. The goal of WoZ Prototyping is ultimately to identity areas of strength and weakness before hi-fidelity products are made and mass distributed.
Prototyping
In our class assignment, I worked in a group to imagine a gesture recognition and control system for Netflix TV. Our task was to create a…
Gestural user interface for an Apple TV or similar system that allows interaction through physical motions. An example prototype would be controlling basic video function controls (play, pause, stop, fast forward, rewind, etc.). The gestural UI can be via a 2D (tablet touch) or a 3D (camera sensor, like Kinect) system.
We decided to evaluate the intuitiveness and usability of our proposed hand gesture controls, including scrolling in all direction, changing the volume, making a selection, fast-forwarding and rewinding, and playing and pausing the video.
Team Roles
We had four roles: facilitator, wizard, scribe, and documentarian. Facilitator walked the participant through the introduction and tasks, wizard controlled the Netflix system hidden from view, scribe noted the happenings of the test, and documentarian filmed the tasks.
Participants
Two participants were invited to test of the behavioral prototype. They were prior acquaintances with members on our team and non-design UW students. In recruiting them, details pertaining to the setup and true goal of the assignment were hidden from them.
Wizard of Oz Setup
A projector was placed on the TV to act as a pseudo motion-detection system and watch battery stickers were placed on the participant’s hands as pseudo motion sensors. The user performed 3 tasks: (1) selecting a show, (2) turning the volume up and down, and (3) fast forwarding.
Our Wizardry
The prototyping test would be conducted in facilitator’s home and with her TV. The facilitator, scribe, documentarian, and participant would stay in the living room with the TV. The facilitator would read from a script and introduce the gesture control concept.
Meanwhile, a tablet device would be angled so that the video camera captures the living room space and allows the Wizard to view the video from the kitchen. The same video was also used by the documentarian to record the actions of the participant. Meanwhile, the wizard will control the TV with a remote from the blind spot in the kitchen area. The wizard would control the TV based on the commands they see the participant performing via from the tablet.
Criteria for Success
We considered a mix of qualitative and quantitive data to critique the success of our prototype. The following list includes the behaviors we looked at and questions our facilitator asked the participant.
Quantitative Data
- Number of times user successfully completes action
- Number of times user makes a mistake
- Ratio of success to mistakes
- Number of times confusion is expressed
- Rating (1 hard - 10 easy) of task
Qualitative Data
- Questions the user asks
- Thoughts the user expresses
- Emotions the user displays, facial expressions
We will also know if the prototype was successful if our user fully believes in the scenario and prototype.
Analysis
After evaluating the prototype and yes, revealing the wizard to the participants, we looked back on our data to summarize our key findings. Below are discussions of the prototype’s successes, shortcomings, and in-class critique comments.
What Worked Well
Overall, the designed gestures were intuitive and easy to use. Participants quickly learned and remembered to bring their hand forward to play/pause, move their hand left/right for scrolling/fast forwarding/rewinding, turn their hand clockwise to turn volume up, and turn counterclockwise to turn volume down. When asked to rate each task, participants rated all three tasks between 8 to 10 (10 being the easiest). They showed little signs of confusion, except when on-boarding and displayed facial signs of content after a few minutes into the activity. They also appreciated the “responsiveness” of the system.
What Needed Improvement
One participant experienced confusion with the scrolling during the on-boarding. Similar to how on personal laptops, people prefer different settings in scrolling up or down on the mousepad, the first participant was unaccustomed to the scroll direction setting we designed. In our control system, a swipe from up to down meant “scroll up”. He expected the motion to mean “scroll down”. It look him a few minutes to reorient himself to the direction of the scrolling. While he mentioned this issue did not detract from the usability of the system, his expectation is something worth considering. One solution could be to allow users to change their scroll settings to what is most intuitive to the user like how personal laptops allow.
Another issue brought up by both participants was the reactivity of the gestures. For consistency, we decided that no matter the intensity or scale of the gesture, each gesture would only perform one action at a time. For example, one turn to the right would turn the volume up by one. To turn the volume up by 10 would involve 10 separate turns to the right. Even more frustratingly, one swipe would only fast forward or rewind by 10 seconds. It would be exhausting to fast forward into 20 minutes of the show. For volume, one solution could be to count the degrees the user turns their hand. For X number of degrees they turn, the volume would turn up by 1. In one turn, perhaps volume could turn up by 5. For fast forwarding/rewinding, one solution could be to measure the speed of the swipe. The faster the swipe would reflect the speed of the forwarding/rewinding.
In-Class Video Critique
The main concern within the critique group was that the video does not show the TV interactions. Our peers explained that it would have been helpful to view the participants’ gesture and the TV response side-by-side. Because our camera primarily positioned to aid the wizard, we did not film the TV monitor. In retrospect, we could have positioned a second camera that captures the participant’s and TV activity together. The video of the second camera may have showcased the evaluation of the prototype better.
Conclusions — Effectiveness of Design
In conclusion, the gestures we designed successfully met our standards. The gestures were intuitive and memorable according to the Wizard of Oz tests we performed with two participants. Going forward, I think we would continue to use these basic gestures of pressing, swiping, and turning. However, we would also need to implement some sort of protocol or feature to respond to the scale of users’ gestures in changing volume and scrubbing through video.
