Is Virtual Reality the Future of Productivity?
How machine learning algorithms can improve virtual reality—and our ability to WFH.
Recently, Facebook — a leader in virtual reality systems — teased a video of a virtual reality headset emulating a new work-from-home experience. In the 8 second long video, Facebook showed off their vision for how people could work from home using technologies such as Oculus and Passthrough. Experiences such as this can define a new age of technology, one with an emphasis on machine interaction in a human environment.
Machine learning can have a large impact on our usage of VR. Models such as neural nets and image classification can help increase the quality of our virtual reality systems to make them more realistic and reliable. Combined with world-class hardware, using machine learning in virtual reality systems can help us stay more productive at work.
How VR enhances productivity
If we use a virtual reality headset, image classification algorithms can detect areas in our workspace to inject elements we can interact with such as meeting toggles or markdown features. Instead of shifting focus from the task at hand, we can easily mute our microphone or switch tabs. These interactions can also become more subconscious.
If virtual reality integrates eye-tracking software to measure a vector of where the user’s eyes are looking at, then we can create even more areas where people can interact with their surroundings artificially.
Examples of this could include instantiating more informational elements around a user’s workspace as their eyes shift from place to place. These elements could automatically recede into the background or fade away when a user’s attention is shifted.
While there are many places to integrate machine learning into VR, there are three main areas to focus on. The first is user body tracking, including eye and hand movement recognition as well as measuring the distance a user moves around a given area. Second, we also need to see the distance between a user’s hand or eyes and a target such as a flat surface which can be converted into an interactive tile. The final piece to the puzzle is voice commands to directly contact a virtual reality system for explicit directions from a user.
Tracking a user’s body movements can lead to more subconscious and intricate interactions. Detecting motion activity can allow for virtual reality systems to automatically switch scenes, providing more content for the user to explore. Body movement detection has a high potential to create more nuanced changes in a user’s workspace, giving them more flexibility in their virtual interactions.
Odometry is the process of collecting data from motion sensors to estimate displacement over time. This can be used in virtual reality software to detect movements of users over time, particularly for actions that need to be executed with precise data. When the margin of error needs to be less than 1 inch, in terms of distance, then odometry can help us distinguish between trivial movements and distinct hand coordinations.
As shown in the image above, a neural net can be used to determine an output such as a steering angle. We can adapt this graphic, meant for robotics, into terms applicable to virtual reality. Instead, we can measure the distance from a target by comparing the angle between a user’s hand and that specific target in relation to the user’s workspace.
If a user wants to deliberately change an aspect of their virtual reality workspace, they may do so with their voice. The case of integrating voice commands into a virtual reality system lays in the ability to adapt a user’s commands into the environment around them: How do you distinguish what the user is referring to.
We already have voice assistants that can execute requests but these systems are often mapped to internet functions such as finding directions to the nearest pizza parlor or booking an appointment at a hair salon. Mapping these functions to virtual reality is an entirely new dimension.
Applying voice commands to virtual reality settings can be done with the help of standard voice recognition technologies and by utilizing hand and eye movement to detect a user’s location. Spatially applying commands by dividing a virtual reality set into multiple parts can help ML algorithms detect what a user is referring to.
An example of this is when you ask a virtual reality system to remove all tabs and modals in your workspace. Do you mean just the ones on your desktop (such as a clock or calendar) or even the ones that you are actively interacting with (virtual internet browsers or VR communication apps)?
Integrating ML into virtual reality
Algorithms have to take into account what you are looking at (an indicator of active interaction) as well as your location relative to the virtual environment.
The technical aspect of integrating various different ML models into a virtual reality system is very interesting. Most image and video recognition softwares use convolutional neural networks to detect objects or patterns. Convolutional neural nets are designed for sequential data which makes them perfect for finding interact-able areas in a user’s workspace. A user action can, in turn, change the output or layers of the neural network to adapt to the user’s needs.
Knowledge graphs — a visualization technique for mapping large quantities of knowledge and information using nodes and edges to represent relation — can be used for a VR system to intuitively supply information based on a user’s needs. This information can be displayed on tiles and modals resting on a user’s workspace to provide them with information based on the time of day or a user’s body motions.
This mapping technique can be controlled very easily because we can set parameters to limit how relevant information retrieved is. If we tell our ML model to only retrieve knowledge up to two relations, denoted by edges in a knowledge graph, away, then we can easily provide users with up-to-date and important information.
If we properly utilize models that are suitable for machine learning, then we can expand the capabilities of VR even further. Integrating algorithms such as natural language processors can help make user interactions more direct. Applying odometry to a virtual reality system can create more subconscious-based actions. Taking breakthrough machine learning models and adapting them to virtual reality can have a big impact on a user’s interaction with their workspace and occupation.
