Harnessing the Power of Brain Waves in Virtual Reality: Applications for Gaming and Medicine
As children, we all dreamt of lifting things and controlling them with our minds. Years have passed and we struggle to summon the right information in our exams, let alone manipulate objects with ‘mind control’. What if we told you that we could achieve these supernatural mind powers, not in the real world but in the ‘reel’ world. The sum of your brain waves and virtual reality could mean that the technologies in movies like ‘Ready Player One’ may be just around the corner.
Pioneering advances in the ability to harness brain waves have made the almost impossible notion of moving and controlling objects with the mind possible. Neuroscientists are now trying to implement the same ‘mind-control’ using a type of virtual reality called neural VR. Neural VR is a new type of VR that is more immersive and real for the user such that it allows them to control the virtual environment using their thoughts. The applications of this technology are starting to sprout in video games, healthcare, and many other fields.
The last decade has seen video games feature cutting-edge experiences for users: sensor-based computing in Nintendo’s Wii, gesture-recognition in Microsoft’s Kinect and Sony Playstation’s VR (Virtual Reality) Headset are a few that have hit the markets and caused a spur. Even in the medical field, the application of neural VR has seen exponential results when it comes to helping patients overcome mental health issues and other brain-related illnesses. These inventions are only the beginning; the future of neurotech in VR is very exciting.
Users use a headset provided with VR goggles that are attached to sensors that can read and process and translate their brainwaves into actual movement in the VR world. Boston-based startup Neurable is focused on deciphering brain activity to determine a person’s intention, particularly in virtual and augmented reality. This is the first step in a giant leap towards controlling VR games with only your thoughts.
Before we get into the specifics on neural VR, we need to understand basic VR headsets function. For the VR headsets to work, a console (computer or smartphone) has to be installed. Video is sent from the console to the VR headset. The display inside the VR headset has two LCD display screens. The headset creates a 3D environment by angling several 2D images at a minimum frame rate of at least 60 frames per second. Another important feature is the head-tracking. Headsets use degrees of freedom (DoF). This allows the headset to look at your head’s position in terms of the X, Y, and Z axes to measure your head movements. Additional equipment for head tracking includes a gyroscope and an accelerometer. The gyro sensors constantly send information to the display driver to change the orientation of the screen according to the user’s action, allowing movement in multi-directions. The accelerometer’s primarily function is to let the system know which way you are facing, as well as also measures the acceleration along a particular axis.
Audio is incorporated in order to immerse the user. Developers use 3-D audio that manipulates the in-game sound to give users the feeling that sound is coming from all directions. This method tricks your brain by using the ears and the auditory nerves, pretending to play different sounds in different 3-D locations, even though the sounds are only produced by a couple of speakers. By carefully executing the timing, direction, and echo characteristics of each sound that is played, engineers are able to create this 3-D audio environment. Thus, users will receive sounds from all directions. This is a major leap forward from the current dual-stereo headphones, which only project sound from the left and right, that we currently use.
Along with the aforementioned features, A neural VR headset is able to capture, decipher, and use brain waves from the user.
Our brains are networks of neurons, nerve cells that transmit information using electrical and chemical signals. Every motion that we perform in the ‘real’ world produces a signal or brain wave. This means that every time our neurons fire up and communicate with each other to receive and transmit information, they emit specific frequencies and we capture these as brain waves. To record brain waves, we use a method that uses non-invasive EEG (Electroencephalography) machines wherein a set of electrodes are placed on the subject’s scalp. The signals are picked up by the headset and the corresponding action is performed in the ‘reel’ world.
Once we know what each brain signal means, we can code that into the game so we get the expected in-game output. In short, once the gamer thinks of something they want to do, the headset obtains the brain signal, sends it to the computer and the computer executes the right code.
Recently, there have been several examples of neural VR growing in the tech arena.
MyndPlay is a UK based company with a passion for the mind and tech; creators of the revolutionary VR ready MyndBand EEG Brainwave headset and the MyndPlayer interactive mind-controlled video platform which gives users an interface to control, influence, and interact with video games, apps and movies using only their mind and emotions.
The first neurogames — “Throw Trucks With Your Mind” (which allows users to pick up and throw objects by mentally blocking distractions) and NeuroMage, which allows users to use a “relax the mind” technique to learn new spells and levitate the Millennium Falcon. (Watch the trailer video for the game: https://www.youtube.com/watch?v=Ij22vP3q6Y4)
Looxid Labs took a few more steps with this technology. Their system allows you to create your own VR environment with your thoughts. They achieved this using EEG sensors that we mentioned before. You don’t even need a gaming console,you can use your phone. They also implemented an eye-tracking feature that moves the game environment as your eyes move.
Neurorehabilitation is another promising area where Neural VR can be applied. A study was done by Duke University that used VR and brain-computer interfaces to treat physically-impaired patients. The 12-month study with 8 patients showed that VR can help aid in restoring ability with patients who suffer from a chronic spinal cord injury. They used a brain-machine interface and a VR headset to simulate neurological functions that allowed them to move their lower limbs. Scientists hope to make use of VR games to assist patients with illnesses ranging from Alzheimer’s to attention deficit hyperactivity disorder (ADHD).
MindMaze is a neurorehabilitation company that is making waves in healthcare industries around the world. Combining neuroscience and virtual reality, the company develops a platform called MindMotion Pro that works to fix broken connections in the brain and help a patient’s brain and body bounce back from traumatic injuries and acute and chronic strokes. MindMotion Pro uses real-time motion capture sensors to map a patient’s movements onto a 3D avatar. The platform then guides the patient through customized exercise regimes based on neurorehabilitation principles in order to restore damaged neural pathways and form new ones. The interactive nature of MindMotion Pro is key to maximizing the benefits for stroke recovery patients. MindMotion Pro’s virtual-reality based games increase patient motivation and prolongs engagement in exercise repetition up to 15 times more than standard rehabilitation programs. The real-time multisensory feedback allows therapists to better customize treatment plans to patient performance.
The neuro VR industry still has plenty of challenges to overcome. Even as hardware keeps getting more powerful, the technology required to bring ‘mind control’ in gaming is still in a nascent stage. A big hurdle is the comfort of the headsets. Users need to buy into the idea of wearing big clunky headsets until we can come up with a more compact version. Furthermore, these headsets raise privacy issues with regard to the security of our thoughts. These are some areas where technology must be reined before we continue developing this fantastic technology.
This article was written by Shreyash Sridhar and Josephine Ta, and was edited by Jwalin Joshi.
Shreyash is an undergraduate at UC Berkeley, studying Electrical Engineering and Computer Science.
Josephine is an undergraduate at UC Berkeley, studying Neurobiology.
Jwalin is an undergraduate at UC Berkeley, studying Applied Math and Computer Science.
Contact Neurotech@Berkeley for a list of sources.