Adding Another Dimension

Our needs and wants as users evolve quickly and warrant the need for different ways to interact with our devices and apps. While voice and 3D motion gesture control are picking up, they don’t fit many contexts of our lives today, making touch still the primary mechanism. One way we’ve addressed this desire for expanded functionality is by creating other distinct touch gestures and combinations, such as adding a hold or finger in the mix. But we only have so many fingers and can only remember so many long sequences of combinations (think Tony Hawk or Mortal Kombat — left trigger, X, X, and wait what? Ugh, fail). Not to mention, even if we have enough fingers or can remember the combos, many are super awkward and inconvenient, in light of the way we hold and use our devices.

Over a year ago, I had dinner with friends at JailbreakCon. One of them was Grant Paul — he’s known for his involvement in early iOS jailbreak exploits and has developed many tweaks and apps for the Cydia Store. We were discussing his project Zephyr, which allows people to quickly access the app switcher with a swipe up (amongst other things). The problem at hand was how to rebuild it, now that Control Center was part of iOS. Other people around the table made comments like, “What if you move the gesture somewhere else, add more fingers, or place Control Center in other spot?” There was even talk of using “hot corners” so opposite sides of the screen did different things. These have been popular suggestions online too, and other designers and developers have taken this approach in their own projects. He politefully disagreed, replying that he felt these weren’t right. I agreed with him. Compartmentalizing the screen or extending gesture combinations didn’t feel right to me either. We both discussed this at great length to no avail. It’s tugged at me ever since.

My “Duh” Moment

There has to be a better way to do this. I have had this thought ever since that conversation and it continues to poke at me. I have bent my mind over a zillion options with no luck, trying to come up with a new way to do this — not just for Zephyr’s sake, but because the issue of gestural conflict is important, and maybe I could come up with a more general, relevant solution to problems like this. The biggest issues at hand were timeliness and efficiency. Interaction happens quick and you can’t afford to wait for more than a second to recognize the distinction. Otherwise, you can end up creating a jagged or unpleasant experience in a number of different ways: delay in action, choppy animations due to failing to properly register, etc.

It wasn’t until recently, while pursuing other projects, that it dawned on me. I’ve been doing this all wrong. Maybe there is that “better solution” out there, but what if it’s right in front of me, hidden in what’s already here? What if I were to investigate how people already use gestures and see if there’s an opportunity to expand? Or in another words, reclaim a gesture back and re-purpose it. In hindsight, I recognize that this is a trap many of us fall into. Not everything is a new-new discovery–most realizations are made through reevaluations of what already exists. A great example of this is Apple’s introduction of Force Touch. A touch is still a touch, but adding pressure as another dimension factors in intentionality; a deliberate action that is different from all the rest. So, I embarked on a simple research project.

I built a prototype with Framer.js and Parse to see how people behaved when asked to perform a swipe and drag gesture. I wanted to know if there were variations in behavior and intention when using said gestures. This seems like an obvious yes, but let me explain. I have always asserted that if swipe and drag gestures could be recognized correctly, one could have access to different sets of features and options. Swipe and drag combinations are often tied together — a swipe back on a page in iBooks takes you one page back, while a drag does the exact same thing except in a fluid, realtime manner. In some circumstances, the combination makes sense, but I would argue that swipe and drag gestures have become a parent-child relationship because it has been so difficult to distinguish the two upfront in a timely and efficient fashion that it’s not worth the risk of guessing. But what if this could be different? What if swiping back was one page, while dragging allowed me to page through multiple pages? This is incredibly difficult in most circumstances because swipe and drag gestures are often triggered based on distance. A swipe is assumed until a threshold is passed and that’s why swipe and drag gestures in the same space can be tricky. But, distance isn’t the only ingredient to swipe and drag interactions. What’s the other piece? Speed. That’s what I was after. Not only did my experiment look or variations in distance, but in the speed at which a user performed said gesture. Needing velocity calculations for swipe and drag gestures is not a unique thing, however the results are often applied in accel/decelerating already-moving objects, not necessarily in early gesture detection. This is key to why I felt this experiment was valid and worth a shot–I was interested in another dimension of said gestures that maybe people hadn’t looked at in the same way before. Or maybe I’m just a crazy fool.

The Experiment

I don’t claim to be an expert mathematician or researcher. Could my experiment have been designed better? Absolutely. Is there an opportunity to expand further as well as include a greater diverse group of participants? Definitely. I plan to do so and also welcome any feedback on this piece (I’ve gotten a lot already from people I’ve chatted with about this). But I think the results of this very low-fidelity experiment are pretty fascinating. Let me walk you through it.

I had 40 random respondents, all of which opened my prototype link from Facebook or Twitter and performed the same thing with the circle shown on screen:

• Swipe up the circle five times
• Drag up and drop the circle five times

No explanation of what I meant by each gesture or how the mechanics of the interaction worked were given to ensure that my experiment was as blind as possible. From each respondent, I extracted which device they were using, the orientation of the device, the average distance (touch start to touch end, average of the 5 capture numbers), and the average velocity (calculated every 100 ms, average of the 5 captured numbers). Unfortunately, I screwed up on the API for getting event touches for Android devices so the two Android users that participated had to be thrown out (leaving me exactly 40).

The results

• The average swipe distance was 449.278947 pixels
• The average drag distance was 670.036842 pixels
• The average velocity of a swipe gesture was 5.17261068 pixels per second (calculated in 100 ms intervals)
• The average velocity of a drag gesture was .61688938 pixels per second (calculated in 100 ms intervals)
• The average swipe gesture took 0.08685729 seconds to finish
• The average drag gesture took 1.08615395 seconds to finish

The differences the between a swipe and a drag are pretty pronounced — awesome! My hypothesis has been validated to an extent.

What Now?

These results are important because they showcase a key point: people not only understand the difference between a swipe and drag gesture, they behave very differently for each, even from the beginning of starting the interaction–there’s clear intention involved. With this knowledge, one could look into new ways to incorporate swipe and drag gestures to access two distinctly-different features or options, whether its tweaking some of the values here or running other experiments. This doesn’t mean that this type of interaction applies in all contexts, but there is definitely opportunity to pursue this further. I plan to do so.

For Grant’s sake, I decided to apply my findings (using the geometric mean of the velocities and mean of the mean distances) to bring gesture activation back to the iOS app switcher. No hot corners or funky gestures ☺. Further experimenting and testing could make this even more accurate, at least in this context. Although I demonstrate each gesture very deliberately in the video below, once it locks into one gesture or the other (say it recognizes a drag), you can move as fast as you want. It’s just in the initial motion that it really hinges on velocity, which is the best part! Your movement speed obviously can change how fast the app scales in the case of opening the app switcher. This prototype was build with Framer.js (note that all elements are images).

If you want to peek at my data or play around with my Framer.js iOS app switcher concept, you can view it all on my Github.

Have questions? Want to debate this further or give me feedback? I’m on Twitter.