Gestural communication in VR

Gestures play an important role in how we communicate and interact with others. Gestural interfaces were already part of VR experiences with controllers. But now with hand tracking on Oculus Quest and devices like Leap Motion, gestural communication is something worth exploring even further within VR.

Been playing with gestures and haptics previously, in this post, I decided to look into theory behind gestural interfaces — and more specifically into different categorisations of gestural communication that can apply in human-computer interactions.

A gesture is a form of non-verbal communication in which visible bodily actions communicate particular messages, either in place of, or in conjunction with speech. Gestures include movement of the hands, face, or other parts of the body. [Wikipedia]

Gestural communication can be divided into postures and gestures [Doug Bowman, Ivan P. Poupyrev, others].

1. Gestures are a movement of the hand, like waving or drawing.
2. Postures are a static configuration of the hand. A good example here would be forming a V shape with fingers for “peace”, a thumb up for “great”.

Postures and gestures in Leap Motion Paint

Gestural communication can deliver four types of information [Hummels & Stappers]:

1. Spatial information, which is the spatial relationship that a gesture refers to. Such gestures can manipulate (push/pull), indicate (point), describe form (show size), describe functionality (twisting motion to describe twisting a screw), or use objects.
2. Symbolic information, which is the sign that a gesture refers to. Such gestural commands can be concepts like making a V sign, waving to say hello or goodbye, etc.
3. Pathic information, which is the process of thinking and doing that a gesture is used with (like subconsciously talking with one’s hands).
4. Affective information, which is the emotion a gesture refers to. Such gestures are more typically body gestures that convey mood such as distressed, relaxed, or enthusiastic.

Even though pathic and affective communication can enhance virtual experiences, due to direct influence on objects and immediate delivery of information, spatial and symbolic communication can be most effective and useful in human-computer interaction.

Cat Explorer

Designing VR experiences with gestures

Controllers like Oculus Touch do offer a number of gestural commands. However, gestural commands with hands have a number of advantages which include more flexibility, not having to hold any devices in the hands, not having to see or directly look at the hands, etc.

At the same time, due to such flexibility, there can be an overwhelmingly large amount of gestures that the user will have to learn and remember.

On-boarding tutorial in Paint where you need to give thumbs up to go to the next step

When designing VR experiences with gestural commands, one should consider:

• Gesture commands require learning, so limit their amount, and provide references where user can learn those from;
• To teach gestures, the user will need clear signifiers — often visual, like icons, video, or mimicking motion;
• Gestural commands need to be simple and easy to repeat and recognise;
• Feedback should be provided to let users know if they have done the command correctly and understood everything;
• Gesture commands on their own are not enough, and should co-exist with other commands and cues;
• When designing gestural experiences, consider the target audience and cultural references to gestures;
• It is important to also consider possible accessibility issues and how gestural commands can be adjusted accordingly.

Racket Fury with the combination of spatial, symbolic and affective communication

Which experiences in VR do you know that offer interesting gestural communication? Let me know in comments :)