Brain-Computer Interfaces x Virtual Reality — A Match Made in Heaven
I’m sure most of you have heard of virtual reality (VR). But if you somehow haven’t, it’s a fully immersive computer-simulated environment that gives the user the feeling of being in a different environment. Essentially, it’s an alternate environment and completely digital. There are countless applications of virtual reality from immersive gaming, immersive entertainment, and potentially even virtual environments where you can interact with other people, similar to reality! But virtual reality is still not fully immersive since it doesn’t stimulate all of our senses. Is there a way to further immerse ourselves in a digital environment?
Enter BCIs.
Brain-Computer Interfaces (BCIs) are not as popular or well-known as VR. However, they are just as cool, if not, even cooler than VR. A BCI is a system that connects your brain to a computer and allows it to interact with your brain. Using their thoughts, the user can perform various tasks on their computer. These actions can range from changing the volume to controlling video game characters. However, there are so many more insanely cool things we will be able to do in the future. Using BCIs, you will be much more immersed in a VR experience because you will be controlling everything with your brain instead of using VR controllers. Trust me, this is not a joke.
So how do they work?
BCIs rely on signals that your brain emits called action potentials. Action potentials tell parts of your body to perform a function and are converted into waves which a computer can then interpret Based on the frequency of the waves, the computer can then perform the associated task. Basically, you are communicating with the computer, but you need to program the computer first to understand you.
There are three different types of BCIs:
- Invasive
- Semi-Invasive
- Non-Invasive
Going inside your brain…
Invasive BCIs require electrodes to be implanted into the brain during neurosurgery.
I know it sounds crazy. You’re probably thinking “Why would anyone willingly put things inside their brain?”, and I would be in the same boat as you.
But hear me out, these BCIs have massive potential to completely change our lives.
Within invasive BCIs, there are single unit BCIs, which detect signals from a small area of neurons, while multiunit BCIs detect signals from multiple areas. Since the electrodes are inside your brain, invasive BCIs can obtain the highest quality signal out of all other types of BCIs. The signal is taken directly from the cortex and transmitted to the computer without any noise or interference.
These BCIs can also stimulate your brain to make you feel something or to control your actions with neurofeedback, which is the process of sending signals to your brain. With nearly endless possible applications in the future, BCIs can potentially help with restoring vision, neurorehabilitation, and could even give us the ability to upload information to our brain🤯.
However, as you can expect with any invasive technology, there are some problems. When normal tissue is destroyed by either disease or surgery, there is a high risk of scar tissue forming in your brain, which puts pressure on your nerves. The surgery is also very expensive and requires extreme precision.
So far, this is the least developed and least researched type of BCI, for obvious reasons, so it is not ready to be integrated with VR yet.
How about the surface of your brain?
Semi-invasive BCIs are less extreme than invasive BCIs and don’t require electrodes to be implanted into your brain, but rather on the surface of your brain. So they still require an operation, but instead of neurosurgery, they require a craniotomy, which is a process where your head is opened.
This sounds crazy too, but it’s way easier than neurosurgery.
The most common type of semi-invasive BCI is electrocorticography (ECoG). ECoG requires electrodes to be implanted on the surface of your brain, lowering the risk that fully invasive BCIs propose.
ECoG signals have a high spatial resolution, which you can think of as the clarity of signals. It’s similar to the clarity of a picture; 480p (bad) vs 1440p (great). There is also no background noise from eye or muscle movements, resulting in high-quality signals.
Compared to invasive BCIs, ECoG has a lower clinical risk over time and is cheaper, which is a more appealing option to most people.
Semi-invasive BCIs have the best of both worlds, having close to the quality and clarity of the signals that invasive BCIs have but produce less risk and are cheaper, similar to non-invasive BCIs.
They also have a high potential for merging with VR since they produce high-quality signals without having to do a full surgery but right now, it’s still a developing science.
No surgery, thank you.
Right now, non-invasive BCIs are the most common and used for most consumer products that track brain activity. As you can guess, non-invasive BCIs are not invasive, which means they measure brain activity from outside your head.
By far, the most common non-invasive BCI is electroencephalography (EEG). EEG consists of electrodes being placed on your head, which is obviously safer and a much easier process than EcoG or invasive BCIs. EEG is quite easy to access as well because most consumer products that measure brain activity utilize EEG.
However, EEGs have limited use in more advanced BCIs because of their low signal quality. EEG devices produce low-quality signals with a lot of interference because they are further away from the brain. This is a major problem with future applications of BCIs which require the utmost precision and accuracy. For our current needs, however, non-invasive BCIs have the most value.
When it comes to VR, EEG devices are the easiest to integrate. The EEG electrodes can be attached to the VR headset, making it extremely simple to record your brain data while using the VR headset.
Creating a BCI x VR game
Combining BCI technology and VR seems so far away, but there are actually several games that currently exist which combine BCI and VR. One of these games is “Throw Trucks With Your Mind” (it sounds so fun) and it was created by the company Neurosky. Let’s dive into how a game like this is created.
Generally, the process of building a BCI is as follows: signal production, signal detection, signal processing, and signal output.
Signal Production
You need someone to actively produce brain signals for your BCI; the user.
Signal Detection
There are many methods to detect brain signals, but EEG is currently the most commonly used method. You use EEG electrodes and you would either place them on your head or on a muscle.
Signal Processing
Signal processing is a process of filtering out any noise interference in the EEG data, generally including your heart, muscle, or eye movement. It is produced by P300 waves, which represent involuntary brain activity. For this reason, we use filters and AI algorithms and filter the EEG data to extract the data we need.
Signal Transduction
Signal transduction is the step where you tell the computer what to do with your EEG data. You tell it how to interpret it and which task to perform. For example, you can program your computer to change the volume every time you flex your bicep. Essentially, you are assigning a physical activity to a computer task.
Since “Throw Trucks With Your Mind” relies on the user’s emotions. EEG data is tracked and classified more specifically. Neural oscillations are tracked as a means for measuring your excitement levels. Neural oscillations are brain waves that are classified based on what excitement level they are generated at and are listed as follows:
- Gamma (32–100 Hz): Emitted during a life or death situation.
- Beta (13–32 Hz): Emitted doing work hours and problem-solving.
- Alpha (8–13 Hz): Emitted while relaxing or closing eyes.
- Theta (4–8 Hz): Emitted during sleep, daydreaming, and meditation.
- Delta (0.5–4 Hz): Emitted during deep sleep and meditation.
Using these neural oscillations, Neurosky tracks your emotions and excitement levels during the game.
The final step to create the game is to change the input commands from peripherals to the EEG data. The actions within the game are based on your concentration levels and calmness, which are measured through your brain waves. If you emit beta or gamma waves, the game recognizes that you are concentrated on one action, which lets you access all abilities in the game. If you are not focused on one action, you emit theta or delta waves. The game decides what to do with alpha waves based on other factors.
But the essence of the game is that the more focused and you are, the more abilities you have that you can use with ease. If you lose that concentration, the game becomes more difficult.
I know this seems complicated, but we’re only scratching the surface of BCI integration with VR. Once we can make semi-invasive and invasive BCIs mainstream, VR technology will take a huge leap and I’m totally ready for it.
Current Developments
Similar to Neurosky, there are some companies working on BCI gaming:
Valve
Valve is one of the largest video game developers in the world. Some of their games which you may recognize are CS: GO, Half-Life (yes, the entire series), and Portal (1 and 2). Most of their games were released in the 2000s or the early 2010s, but my point still stands. Currently, they are working with OpenBCI to build a fully-immersive gaming experience. The thing that sets this idea apart is that Valve plans to use neurofeedback to allow the user to receive sensory information from the virtual world. This would completely alter the field of gaming and could lead to a completely new and unforeseeable future.
Neurable
Neurable is one of the biggest companies in the BCI industry and plans to advance this technology by VR gaming. Neurable currently sells headphones to track concentration and help with focusing, but they are researching using BCIs with VR. With the idea of making the UI/UX of VR more accessible and easy to use, while also focusing on gaming, they are using event-related potentials for their BCI because they show what the user is intentional about. They have already developed a prototype of this at SIGGRAPH, a tech conference in Singapore:
TL;DR
- Brain-computer interfaces (BCI) can make VR much more immersive
BCIs rely on signals called action potentials, which your brain generates to perform a function
- There are 3 types of BCIs: invasive, semi-invasive, and non-invasive
- Invasive BCIs produce the highest quality signals, but require neurosurgery and are very expense
- Semi-invasive BCIs produce signals with no interference but require a craniotomy
- Non-invasive BCIs produce signals with a lot of interference and low clarity, but they are cheap and easy to use
- The steps to create a BCI x VR game are signal production, signal detection, signal processing, and signal transduction
- A company called NeuroSky created a BCIx VR game called “Throw Trucks With Your Brain” and it tracks neural oscillations
- Neural oscillations are brain waves classified based on the frequency of the waves and the task or scenario that produces those waves and they are divided into Delta, Theta, Alpha, Beta, Gamma
- Semi-invasive and invasive BCIs have a lot of potential in the future to be integrated with VR
- Some of the largest companies working on BCI x VR integration are Valve and Neurable
There you have it! These are the basics of BCI integration with VR. It seems so futuristic, but we already developed several products in this field. Nonetheless, We are just scratching the surface of what we could do in this fascinating field.
If you enjoyed this article, it’d be great if you can share it with your friends and family and give me a clap!
For more content like this, consider following me on Medium and connecting with me on LinkedIn!
