Apple TV Gestural UI Redesign
Behavioral Prototype, HCID 521 Prototyping Studio, t University of Washington
Background
The Apple TV’s current means of control are either an IR remote control or an application on an iPhone using wi-fi for communication. Our team explored the realm of gesture-based controls. We utilized the “Wizard of Oz” (WOZ) technique in order to explore how gesture based controls could work on the Apple TV. Our team developed a set of gestures that we felt would be intuitive to the users, then tested our gestures with two users using the WOZ technique. In this paper we discuss the gestures, prototype setup, user evaluation and finally we analyze the behavior prototyping process and results from our evaluations.
My team members: Jeremy, and Gwen.
Prototype
To design our prototype we initially discussed the existing Apple TV function controls and the design principles of a gestural user interface. Our discussion revolved around the following questions and considerations:
- How precise does each gesture need to be?
- What gestures make most sense to us?
- Can we use the same gesture for multiple functions?
- Are finger motions too small to be recognized by our system?
- What gestures do other motion-activated systems employ?
- Should we avoid use of two-handed gestures?
- How do we avoid activating the system with hand gestures used in conversation? Do we need an ‘activate system’ gesture?
- How do we ensure the system works smoothly for left- and right-handed people?
Consideration of the above points revealed that we had to set down our design principles before brainstorming each gesture.
Design Principles
Our design principles were as follows:
- Intuitive: Design intuitive gestures, keeping in mind that “intuitive” is mainly based on a user’s mental model and previous experience.
- Short learning curve: If intuitive gestures can’t be found and users must learn them, make the learning process easy and fast.
- Minimal physical effort: Design gestures that don’t lead to muscle exhaustion.
- Differentiable: If a certain gesture represents more than one function control the context of use must be differentiable.
- Universal design: Wherever possible only one limb should be required and left- or right-handedness should not make a difference.
Apple TV Gestural UI
For a visual reference of the gestures, please refer to the film recording of user testing. The below image shows the result of our whiteboard session, brainstorming and sketching our gestures.
Navigation
- Up — arm/wrist swipe up
- Down — arm/wrist swipe down
- Left — swipe left
- Right — swipe right
- Back to previous screen — counter-clockwise turn of the wrist
- Enter/Play — flat hand
Video/Audio playback control
- Forward — arm/wrist swipe right
- Reverse — arm/wrist swipe left
- Play/Pause/Back/Stop — flat hand
User Testing
The discussion around user test planning included questions and considerations such as:
- Should we ask users to simply guess the gestures to reveal what is intuitive to them, or inform them of the gestures?
- Is an instruction manual required? How much instruction is acceptable, keeping in mind ‘knowledge in the head’ versus ‘knowledge in the world’?
- How can we create a realistic test context in which the user believes the gestures they are making are controlling the Apple TV?
User Testing Set Up and Process
Our WOZ set up included one person controlling the Apple TV with its native remote control out of the line of sight of the user. We set up a camera to record the session, which was placed behind the user to capture both user gestures and what was happening on the TV. One team member facilitated the user testing sessions and encouraged users to think aloud and feedback on the spot.
Following extensive practice rounds coordinating our WOZ technique, we began user testing with a brief introduction for each of our two users. We did not initially direct the users, but allowed them to test what was intuitive to them. When a user made a gesture that the system didn’t recognize, we used a friendly “uh oh” sound to let them know their gesture had been perceived but wasn’t in the system’s “vocabulary”, or that they were attempting an impossible action (such as scrolling beyond the last of the items on the screen).
Questions we asked the users throughout testing were those such as:
- What was your thought process as you completed scenario x?
- Can you explain why you used gesture x to execute that function?
- What was your overall impression of the interface?
- How tired are you, having engaged physically with the system for this time
- Having experienced the system, would you prefer to use gestures or the native remote, and why?
User Testing Scenarios
- You are at the home screen. You want to navigate to and watch an episode of Cougar Town.
- You watched some of an episode and want to fast forward at speed level 1 and then play.
- You fast forwarded too quickly and you want to rewind at speed level 1 and play.
- You want to watch The Big Bang Theory instead of Cougar Town, so you return to the menu.
At the end of the sessions we asked users for their feedback so they had an opportunity to communicate anything that didn’t come up during testing.
User testing in progress
Evaluation
The ability to immediately iterate on our design rendered this exercise different from other prototyping exercises in which we have had to invest more time in the creation and construction of the prototype. Less investment in the prototype itself enabled users to experiment with a gesture, consider it, suggest a change, and then experience their suggested improvement immediately. The result was that we were able to learn what worked and what didn’t work instantly and improve both the design and our user testing methods rapidly. Our impression is that behavioral prototyping is highly cost effective (i.e. rich feedback is obtained very quickly and techniques like Wizard of Oz mean investment in the prototype is minimal), and likely most effective in the very early stages of design before any development begins.
At the beginning of the tests we did not instruct the users on how to perform the gestures we designed; rather we asked them to do what they felt was intuitive. We found that what we hypothesized to be intuitive gestures ahead of time turned out to be intuitive to our users as well. We encountered some deviations with our two users according to their mental models, and pre-existing knowledge of the Apple TV interface and other gestural UIs such as Kinect. For example, our second user demonstrated broad sweeping gestures using her arm, whereas our first user expressed a preference for subtle wrist motions (the latter being in line with our initial hypothesis). When questioned, the second user said that her prior knowledge of motion-activated interfaces suggested that subtle gestures wouldn’t be recognized. She said, “if you hadn’t mentioned flicks [of the wrist] I wouldn’t have considered it, and had already decided I wouldn’t ever use this product [because it was tiring]”.
We found some of our users’ suggestions surprising. For example, our first user wanted to skip the step of navigating to an icon on the screen, and wanted a pointing gesture to immediately open what he was pointing at. We hadn’t considered this possibility in advance as we focused mainly on mimicking existing Apple TV navigation with the remote, which moves sequentially through items on the screen. We recognized, however, that pointing directly at the target was extremely sensical as a gesture even though it wasn’t a repeatable action with the remote. Step by step navigation was still important in some contexts however, especially when navigation to menu items off screen was required.
Reflection
Although we repeatedly practiced our design gestures and the coordination of our prototype and user testing, we still found that we underestimated just how much coordination and practice was required ahead of time to conduct an accurate test. We found that we were focused on reacting to a user’s motions to such an extent that exercising judgment on the accuracy of the user’s gesture was an afterthought. As a result, we weren’t able to test some of the more subtle gestures that we planned wouldn’t have an impact on the system. Additionally, we found that pre-recorded “error” sounds and other audio feedback would have been useful in creating a more realistic experience, and to communicate to the user when they were making a gesture that had been perceived by the system but that wasn’t in the system’s “vocabulary” or when a user wanted to scroll further to the right, for example, but the end of items on the screen had been reached.
In addition to audio feedback, both users expressed a desire for the system to show where their gestures were landing on the screen. Integrating a snap-to-grid function with a visual cursor was suggested, to help users feel more confident in using gestures, and so they wouldn’t feel like they had to hold their hands or arms at a certain level which they both found tiring.
Finally, we found that the tests took longer than expected and we didn’t end up testing whether we would need to incorporate a gesture that activates the system before it starts registering gestures. We recognize that we would need to expand the tests to include multiple people in front of the TV, to gauge whether random hand-motions people make when in conversation, for example, would end up influencing the system (on a side note we question whether a behavioral prototype would be effective for determining the answer to what the technology would register in a group setting).
