What if you could enjoy the nostalgia of your favorite circa 2008 motion-controlled racing game, without the encumbrance of these plastic missiles?
Potentially controversial opinion: these things are the worst. From batteries that quickly die to uncomfortable wrist straps, from ugly plastic grips to semi-unreachable buttons, I have to say that I’m not a fan of the Wiimote.
Also, the name “Wiimote.” The early 2000’s truly had no rules.
To solve this pressing issue, my project partner and I found a way that we could eliminate them. Also, we had an HCI project to complete.
The Solution: Leap Motion & Gesture Interface
This project is based on the hands-only motion design which the Leap Motion controller utilizes. To give a very reduced summary, the Leap Motion is a sensor that tracks the user’s hands, and then uses algorithms to interpret gestures via relative position and motion. It can only “see” in one direction, and so must be positioned below the hands on a surface, or mounted in front of the user. This puts both hands in view, and avoids one being obscured by the other.
The Leap Motion allows for the things you can see in this video: drawing, scrolling, selecting, and more, with only the user’s hands. However, its limitations demand very specialized gesture design. Otherwise, the intuitive quality is lost, or the controller fails altogether. This is the story of how one of my peers and I attempted this specialized design challenge.
Step 1: The Decision
When tasked with finding a program that benefit from a gesture interface, my project partner and I immediately turned toward driving or steering-based games. We wanted something which would use:
- Minimal point-and-click screen interaction, to prevent finicky issues with aim
- Little/no need to switch back to keyboard
- Lots of on-screen movement and physical actions, to make gesture inputs feel as realistic and responsive as possible
In short, we didn’t want to create a gesture interface for something better served by something else. With these considerations, we settled on Mario Kart Wii- a game which already uses motion as its primary input, and almost never requires use of language or other highly specialized input.
A Quick Caveat: Wizardry
With limited resources and time, this project is restricted to gesture/motion design only. We were not yet able to implement this using a real Leap Motion controller, and so our user testing took place via Wizard-of-Oz-ing. This is a practice by which a user tests the appropriate gestures, but in reality another user is responding to them and controlling the program “from behind the curtain.” This will become clear in our user testing videos. This practice, while imperfect due to delays and human error, still effectively tests whether gestures are intutive and memorable to users.
Intended demographic for this design:
Experienced Mario Kart players who are looking for a change. We chose this demographic partially as a matter of ease, because all our available testers are Mario Kart owners who play frequently.
Alternatively, people who don’t like Wiimotes.
Step 2: Gesture Brainstorming
We started with a brain dump from our knowledge of the game, and then supplemented it by researching all the necessary controls.
For each control, we tried to find a motion that corresponded in position, movement, tension, and/or anology to existing gestures. I believed that these more specific considerations would accomplish our broader goal of intuitiveness and memorability.
We decided that the Leap Motion would sit on a surface underneath the user’s hands, and look up at them. This would allow for easier steering than a front view.
We began with menu screen interaction. The Wiimote allows for point-and-click, but we quickly decided against that due to complexity. The Leap Motion would likely work very well for this, especially based on demo videos, but guessing where exactly someone is pointing seemed like it’d be nearly impossible to Wizard-of-Oz. At the very least, there was an easier way.
The Wii also allows for the use of arrow keys to switch between options onscreen, and we felt this system would lend itself better to gesture. This is because it is limited to the simple directionality of up, down, left, right, and also only moves by one “option” at a time instead of jumping across the screen like a cursor.
For our first iteration, we settled on users pointing one finger forward, flicking it in the direction they want the selection to move, and curling and re-extending the finger in a “poke” motion to select. For the “back” button, we had an open hand swipe, like a waving motion, with the same hand.
Then we moved to the driving and other in-game controls. The obvious first choice was a mimed “hands on the wheel” position, but this soon seemed complex and inefficient. The physical wheel had been eliminated, so this motion was almost meaningless. Additionally, occupying both hands is unnecessarily inhibiting to other gestures.
Instead, we decided on the dominant hand controlling the car, and the other coming into play to throw powerups. This mimics the control of a real life scenario- one hand on the wheel, and the other free to throw a strategic banana peel out the window.
To control the car, we came up with a position which aims the hand forward and allows for easy directional pointing. This alignment mimics forward motion, functions while stationary, and is relatively relaxed. The latter two are important because forward motion is the default position. Additionally, this position is reminiscent of a rudder, fin, or other moving/steering device, which makes it easily memorable.
Note- this is the position only for forward motion. For brake and reverse (which use the same button on the Wiimote, stopping you first and then sending you backwards) we took this same position and rotated it 90 degrees so the palm is down. We decided this would be an easily distinguishable shape for the Leap Motion, and is intuitive enough since putting one’s palm down implies resting or friction. This still allows for steering in reverse. We acknowledge that it’s a bit harder, but it’s also harder to steer in reverse in Mario Kart already.
The non-dominant hand controls power-ups, and only comes into play when needed. To throw a power-up forward, the user opens their hand forward as though they are throwing something. To throw one backwards, they do the same motion but down toward the Leap Motion, as though they are dropping the item on the road as they go. The video below is of a backwards power-up throw.
Some additional in game controls include pausing the game (two fists out), doing a trick jump or wheelie (flicking the driving hand upwards), revving up at the start of a race(clenching the dominant hand in a fist, with the other hand out of view), and drifting*.
The wheelie/jump gesture is an example of prioritizing motion in the design. This flick of the wrist is very similar to how one would shake a Wiimote to get the same outcome in the regular Mario Kart, and the quick upward motion implies a liftoff of some kind.
The “rev up” motion is meant to imply tension without motion, thus the clenched fist, which then springs open for the “driving forward” position. This function is about pressure which then pushes the player forward, and the hand motions are meant to mimic that.
“Pause game” needed to be easy to do, but not easy to do on accident; thus the two fists. The non-dominant hand will likely not be in the Leap Motion’s field of view for most of the game, so it’s very unlikely that the user would accidentally have both hands in fists and in view of the controller.
Step 3: First User Test
Our first user test was instrumental in helping us rework our menu/selection gestures. Our participant, Evan, noted almost immediately that it was hard to gesture repeatedly with one finger, and that having to re-extend the finger to “select” was tiring. Additionally, these gestures were small and hard to distinguish for the “wizard” (myself). We quickly changed this to waving the full hand to change selection, and poking one finger forward just for selection. This helped distinguish the gestures from one another, and according to Evan, made them even more intuitive. See the result of this change below.
User Test 1.5
In addition, we changed the “back” motion to include the non-dominant hand, helping even more with distinction and adding some balance. This way, one hand interacts with the current screen, and the other hand can get the user away from it.
It’s worth mentioning at this point that our idea would include an initial calibration that lets the user choose their dominant hand. If they’re a lefty, all our gestures would simply be mirror opposites.
This took a few practice rounds, but Evan became quite skilled at this before I was able to keep up as the wizard. User testing proved difficult because the learning curve was much steeper and quicker for users than for wizards. This wasn’t because of the game’s own difficulty, but because of the unfamiliar need to watch another person’s hands and unlearn old Mario Kart habits.
Evan’s main feedback was:
- Fix the menu/selection gestures.
- Holding the dominant arm out is tiring.
- He wishes we could play on manual, not automatic (allows for intentional drifting). We accounted for this in gesture design, but found that it was too complex for us to actually test*.
- Edge cases- wheelies, power-ups that require repeated fire, etc.
To consider this feedback, we did a few things. We fixed the menu and selection gestures before his user test was over, and arrived at something he not only found easier but also “fun.” We acknowledge that this isn’t easy on the arms, but we couldn’t actually find a way around that without rigging the Leap Motion above the user, which creates another host of problems. In truth, we left this one alone, and conceded to remind users that they can rest their elbows if they want, and can also tilt their hands from the elbow to avoid wrist fatigue. The issue of playing on manual came to mind when we were brainstorming gestures, and we attempted manual in our first practice run before filming, but couldn’t even finish the race. A real Leap Motion could probably pick this up without an issue, but I’m not that good at Mario Kart. In terms of edge cases, we went through the power-ups and made sure each was appropriate for our gestures. We also specified that wheelies are executed in the same manner as tricks, and the gesture to end a wheelie is to bring the hand down to neutral forward position. With that, we were ready for our next user.
Step 4: Second & Final User Test
We are infinitely grateful that several of our friends have Mario Kart Wii and some free time.
Our second user, Ed, was even more skilled at Mario Kart than Evan. This proved extra difficult for effective wizarding. Ed had no issue with menu selection and didn’t require any practice at all. After a few moments of instruction on the possible gestures, he began interacting with the game, even going so far as to start a race without the controller in his hand, before I even taught him how to drive. I took this as a victory, because he forgot that he wasn’t controlling the screen, and felt like he knew what to do.
He relied more on wheelies than I anticipated, and his main feedback involved wanting a more effective way to start and end wheelies. Unfortunately, this came down to me being a Not Ideal Mario Kart Wizard, and having never executed a wheelie before this project. If I were to change the gesture for this going forward, I would probably incorporate the palm-down brake gesture as a method of quickly exiting a wheelie, since it’s easier to distinguish than just lowering the hand, and implies a downward/ground-hitting motion.
Ed had far less feedback about arm tiredness, possibly because he naturally used less exaggerated gestures. See his test below.
Note- skip to the second race for a slightly better view of the screen.
Ed commented that while our chosen driving motion wasn’t the first thing he would’ve thought of, it made sense pretty quickly. He had an easy time implementing it, and luckily didn’t have any issue with forgetting controls.
Since Ed had so little feedback and did so well, we decided to treat this as our final user test, and get to writing.
Step 5: Reflective
If I were to take this project further and create an actual Leap Motion interface with Mario Kart, I would seriously consider arm tiredness as a factor. This would be easier to test with an actual Leap Motion controller, so that we could test if users could rest their elbows, prop their arms somehow, etc. Additionally, I would make wheelie and trick two different motions, since I could see this as a potential point of confusion especially if the two coincide during a race.
I would also do more rounds of user testing with more options for the “drive forward” position, and see which one people like best. The one we used definitely worked and was easy to use and remember, but I don’t know if steering as though the hand itself is the car is the most intuitive option. We never changed it because we never got negative feedback about it, but it’s the only major gesture that doesn’t have me convinced. It seems too similar to an animal or a rudder, not a car. In the future, I would test gestures that tilt, not just curve.
Also, I would user test with less experienced Mario Kart players. Unfortunately, the users we had access to were people who own the game, which skewed our demographic.
In the end, the most important thing we learned is that it’s just fun. Even my user testers, who don’t dislike Wiimotes the way I do, rated this no-remote design as being highly enjoyable. No one said that they preferred this to the regular Mario Kart, but there was also no one who liked it worse. To be perfectly honest, I wish I had written down a list of interview questions to ask during testing, in order to get a better grasp of what parts of this design were better or worse than the regular game. However, Evan’s commentary on this issue may stand on its own: “It’s not necessarily better or worse, it’s just a different game now.”
I enjoyed this project thoroughly because it was creative, and involved humans and interaction more than computers. However, more important than whether I enjoyed it, it gave me a crash course on motion design. It’s difficult to come up with motions that the Leap Motion (or any other system) will likely understand and always be able to see, while also taking into account intuitiveness, ease, and fun. This was real practice at balancing a system’s limitations against the need for an effective UI design.
My takeaway from this project is that there’s always a “wizard,” whether it’s a clever algorithm or me sitting in a corner with a Wiimote. However, the point of good design is to make people forget that they aren’t directly controlling the interface. I don’t know yet if our project has completely accomplished this, so I might keep on iterating. Hands-free Mario Kart is just that good.
*Note: Drifting and “manual” mode ended up being outside the scope of our design, purely because nobody was skilled enough at Mario Kart to convincingly act as the wizard. Because this aspect couldn’t get tested and therefore wasn’t applicable to our final design, I didn’t feel like it made sense to include it in the main section of this post. However, we did create gestures that would account for this function, and considered how the Leap Motion would interpret them.
We decided that since manually drifting creates a tension similar to revving up (this is expressed in the sound and visual feedback of the game, as well as a slight speed boost), it would be make sense for this to also involve a fist. However, drifting isn’t a stationary activity, and requires steering to continue. See our solution below.
While the Leap Motion can also figure out direction from the users wrist motion from one side to another, a more precise way of discerning direction could come from the flat part of the fist between the first and second knuckles. This area of the hand creates a straight line, and on a straight arm this faces directly forward. This gives us enough information that the Leap Motion could look at the line between these knuckles, and establish direction as orthogonal to that line.
Gesturing this way is slightly more difficult and tiring when making a fist, but drifing only happens for short bursts and also requires easy transition back and forth from neutral driving position. As a result, without the ability to user test very rigorously, this is the best solution we found.
Thanks for reading! Here’s a video of something called the Matilda Challenge, which is very similar to our project in both wizardry and fun pointing.