Why is eye tracking important for Virtual Reality?

By: Chris Madott, Technical Lead at Metavrse

Let’s briefly discuss what eye tracking is. Eye tracking is the art and science of measuring what a user is looking at. This data can help researchers better understand consumers. Virtual reality can use eye tracking to fundamentally improve the user experience. VR can use eye tracking for intuitive user input, secure authentication, improved graphics, and expressive social interaction.

1) User input

Our eyes move remarkably fast. It takes just a few milliseconds for our eyes to move from object to object. By measuring our eye movements we can quickly and reliably provide user input in VR. Imagine navigating a context menu or shooting down a plane using just your eye movements. This may seem like technology from a science fiction movie, but in fact it exists today. Companies such as SMI and FOVE have built very capable eye tracking systems that you will likely use in VR.

The speed at which you can provide user input borders on telepathy.

This form of user input is inherently intuitive just like the touch screen on your smart phone. Since everyone knows how to move their eyes, the learning curve is eliminated. To select an item with eye tracking the user can just look at the object then blink or press a button on a controller. This input method is much faster then moving your hand to tap on your smartphone.

Eye tracking is also a hands free input method for VR. This is highly important for mobile VR where users may have their hands full. Mobile VR is all about convenience, so freeing up the users hands is highly desirable. Moreover, mobile users of VR may be unwilling to carry around a separate controller. Eye tracking that is integrated in the head mounted display (HMD) is by far the most convenient input method for mobile VR.

2) User Authentication

Many smart phone users authenticate themselves through a fingerprint scanner integrated into the devices body. This feature improves the security of the device, as the device will not work for users other then the owner. After a user is authenticated he or she is free to do e-commerce knowing that their banking information is secure. Eye tracking technology can be used to seamlessly authenticate a user as well. The same camera that tracks the users eye movements can be used to scan the users retina for authentication. From the users point of view authentication happens automatically once the HMD is in use.

Eye scanning is also wonderfully secure if implemented properly. Your eye scan data will likely be stored in a hardware-encrypted chip located on the HMD (similar to Apple’s “secure enclave” used to store fingerprint data). It is much more difficult to imitate the unique features of the eye compared to a fingerprint. Thus, user authentication through eye scanning is more secure then the existing systems we use for e-commerce.

3) Improved graphics

Buy it, use it, break it, fix it, trash it, mail, now upgrade it. Every time we buy a new device the system generally gets faster, higher pixel count and longer battery life. Eye tracking enables all of these benefits through a process called foveated rendering. By drawing only the part of a scene that you are looking at, foveated rendering can improve graphics quality while lowering the workload on the Graphics Processing Unit (GPU).

Furthermore, foveated rendering is key to achieving higher resolution screens. Both the HTC Vive and the Oculus Rift use screens with a pixel count of 2.6 million. This pixel count is relatively low compared to what our ideal HMD would have. Michael Abrash, chief scientist at Oculus has said that ideally we have 16K screens or about 132 million pixels. The only way that computers will be able to generate high fidelity graphics for such high res screens will be to draw just where your eyes are looking. Thus, only a small fraction of the screen needs to be rendered at high quality.

https://www.youtube.com/watch?v=2HS2p2BmVsk

Foveated rendering can result in less eye strain as well. When you look at the real world your eyes focus on one object and put the rest of the world out of focus. Currently all HMDs keep the entire scene in focus regardless of what you are looking at. Foveated rendering can place only what you are looking at in focus and put the surrounding area into a soft focus, lowering eye strain.

4) Social Interaction

Emotions will be the New Emoji

Often times cartoon characters have disproportionately large eyes relative to their heads. This design decision is done on purpose to make the characters more expressive. We use our eyes in a variety of different ways to convey social information to one another. For example we roll our eyes to express frustration, open our eyelids wide to show surprise, and water flows out of our eyes when we are sad. Eye tracking can capture all of the subtleties with which we move our eyes and convey that information in social VR applications.

We use Emoji because it’s a quick and low bandwidth way of expressing ourselves over messaging apps. With the coming age of social VR we can replace our Emoji icons with real emotions that contain more depth and individuality. Thus, emotions are the new Emoji. Eye tracking is a key to helping users fully express themselves in natural ways. However full body tracking and facial expressions will be need for truly expressive social interaction.

Eye tracking is the key enabling technology that will push VR into the mainstream.

Looking back through the lens of history we can see that the mouse enabled the personal computer revolution. The scroll wheel on the iPod enabled the personal music player revolution. Lastly, the multi-touch screen enabled the smartphone revolution. All of these examples are key user interface paradigms that pushed their respective technology into the main stream. Similarly, eye tracking is the key enabling technology that will push VR into the main stream. Eye tracking enables fundamental improvements in the way users express themselves, do e-commerce and see high fidelity graphics in Virtual reality. For theses reasons, eye tracking HMDs are inevitable as virtual reality matures into a mass-market product.

Chris Madott is Technical Lead at Metavrse, a full service Virtual and Augmented Reality company focused on developing winning Mixed Reality experiences and strategies for companies across a broad range of industries.