The World [of Augmented Reality] Is Not Enough
If you’ve been reading this blog, you know that I’m a big fan of virtual and augmented reality. They are the next progression in tech interfaces, promising a plethora of new ways to perceive and interact with information.
However, most VR and AR tools seem to be focused on gestural control as the primary way users interact with the virtual environment. Just like we use keyboards and mice to give input to our laptops, VR/AR headsets let us use our body movement and gestures.
While this is an incredibly meaningful step, we don’t necessarily use wide, sweeping gestures when visualizing ideas. Perhaps mixed reality headsets could be combined with hi-res motion trackers such as the one being developed by Google X’s Project Soli, and expand the gesture library to include all kinds of micro-movements. But even these movements only represent a small part of the inputs that VR/AR tools could potentially incorporate.
Please, sir…can I have some more inputs?
A Bay Area company called Eyefluence is looking ahead— literally. It’s working on using eye-tracking as an input for head-mounted displays. With a product that can be integrated into existing headsets, Eyefluence aims to use the direction of our gaze to improve what we see and how it is presented.
This is actually quite intuitive. I’ll tell you why — but first, recall the name of your 5th grade teacher. Chances are, your eyes drifted off screen, a common phenomenon when we are trying to recall memories. Our pupils naturally make movements and change in size (dilate) depending on a variety of factors. Incorporating ocular input offers a new layer of user commands for mixed reality systems. However, this still requires the development of an eye-based language, something which Eyefluence is heavily working on.
Is there something even more natural, an input that doesn’t require us to learn a new language of gestures or eye movements? Perhaps we can go straight to the source with brain-computer interfaces. Wouldn’t it be incredible if VR/AR headsets were able to decipher brainwaves and turn them into action? Simply thinking about a calendar, for instance, would open up your calendar app and allow you to interact with your schedule. Or you could think of a memory and a videorecording of the scene would pop up, allowing you to actually see what you’re thinking, and maybe even share it with others.
While such goals are far off, some are starting to explore this intersection. The most popular way to capture information directly from the brain is by recording rhythmic electrical activity from the brain’s surface, or cortex — a technique known as EEG. Portable, wireless EEG headsets for consumers have become increasingly popular over the past several years, with applications such as controlling a video game avatar, flying a quadcopter, and measuring concentration and mindfulness.
“You Think, Therefore You Can” -Emotiv
Combining neurotech headsets with mixed reality systems may actually create a symbiotic relationship, where each technology benefits from improvements to the other. VR/AR displays provide a valuable testing ground for neural interfaces, and EEG headsets may provide information about how the brain perceives or focuses on objects in virtual spaces.
There have been a few notable forays into this space. I personally like this project by Judith Amores, a masters student in the Fluid Interfaces group at the MIT Media Lab. She uses concentration measurements from a MUSE EEG headband to control actions in a virtual space viewed with an Oculus Rift.
The use case is simple but the potential is very impressive. There is much room for exploration:
- She uses a Muse headband, which only has 4–6 recording electrodes, and therefore limited data from the brain. What sort of actions could one code for with a more robust headset, such as the 14-channel Emotiv EPOC+, or even a clinical grade brain-computer interface?
- Recording quality is simply weaker without direct scalp-electrode contact and the use of a conductive gel — how could this experimental protocol improve the data and the resultant virtual experience?
Another group potentially attempting to combine neural interfaces with augmented reality is a company called DAQRI, which acquired consumer EEG headband maker Melon in 2015. Melon’s product aimed to track and improve focus and prioritized an aesthetic form factor, something people would be likely to wear. They made a minimalist headband which looked good but did not necessarily contain as much information as a more robust recording device. DAQRI has yet to release plans for a combined headset.
You Get Out What You Put In
So far, we have discussed inputs from the body that could create a more seamless interaction with virtual and augmented reality spaces. However, simply tracking may not be enough. Remember, mixed reality systems are also stimulating the body — they are sending sensory stimuli such as sight and sound into the body, beaming light into your retinas and projecting sound into your ears. What if we use peripheral nerve stimulation (i.e. tiny electrical signals on the skin) to stimulate our somatosensory receptors, to make us perceive the sense of touch? To make it feel as if we are touching an object which is actually virtual? This is the subject of haptics research, which aims to provide tactile feedback to the body to enhance a digital experience. (It’s actually quite common already — think about rumble packs on old video game controllers, or the vibration feedback our smartphones give us when we type on the keyboard, or the Force Touch feature on the iPhone 6s).
Haptics gloves are an inevitable development, being the most direct way to deliver tactile information to the hands. The whole point is sending information more intuitively to the user. Giving virtual objects the ability to be touched tells us information such as weight, texture, hardness, stickiness, deformation tendencies, and other material properties.
However, gloves are not ideal, and limit some of our somatosensory ability. How can the sense of touch be relayed better, and to other parts of the body? Current sent through the air to a particular point on the skin? What about a synthetic biological material that allows us to switch between real and virtual interactions seamlessly? This is a big question which requires much more thought. (For more innovation in this space, see the Tangible Media Group at the MIT Media Lab.)
What about our sense of taste? Stimulating the nerves on our tongue is even easier than stimulating the skin, because saliva makes for a good conductor. Smell? Olfactory memory is incredibly powerful, more than most people realize— this could be huge for increasing memory encoding and recall. Proprioception, aka our sense of spatial position/orientation? We can fake this well enough with varied visual and auditory inputs. But our brains can be tough to fool; maybe stimulation of the inner ear/vestibular system could truly replicate the disoriented feeling of being on a roller coaster, or flying, or floating in space, or being underwater. The Mayo Clinic/vMocion and Samsung are already underway with this tech, called Galvanic Vestibular Stimulation (GVS).
Why stick to only stimulating our peripheral nervous system? What about a more direct form of neurostimulation? Transcranial Direct Current Stimulation (tDCS) is gaining incredible popularity. It’s a technique where electrical current is delivered through the scalp to the brain’s surface/cortex. (Other forms of stimulation such as magnetic, intracranial, etc. are available, but tDCS is the most popular and accessible to consumers at the moment.) In this way, the brain’s auditory, motor, gustatory (taste), olfactory (smell), and visual cortices could all be stimulated directly.
Research groups, companies, and amateurs are finding fascinating correlations between tDCS and mood states and enhanced motor and cognitive learning. Startup Halo Neuroscience has uncovered fascinating research on the effects of tDCS for motor learning, showing that neurostimulation may actually help people learn motor skills faster. Research groups have found early evidence of cognitive benefits as well, such as Professor Kadosh’s group from Oxford which performed studies in which tDCS improved number learning and memory and a variant called tRNS improved math scores in stimulated students.
What if we used bidirectional stimulating and recording electrodes to both record from the brain and stimulate it, allowing us to not only perceive and interact with digital information more intuitively, but also help us learn new tasks, remember more information, and become better versions of ourselves?
The Upshot
Although this is all very exciting and futuristic, wouldn’t it take something away from the human experience? Distract us from the natural wonder of the world?
Well, we’re already constantly distracted by the small light-filled rectangles in our hands. Augmented reality tools would allow us to engage more with reality, to better integrate our digital evolution with the real world. There’s a reason they are called heads-up displays.
As we have done countless times before, we will adapt and find a balance between the virtual and physical worlds, both equally real. (I hesitate to say physical world or natural world, because as technology advances, the virtual will start to feel physical, natural, and undeniably real. What, then, will be the difference?)
This future isn’t taking away from our natural state of being — it’s enhancing it, by minimizing computing, giving more physical space to nature, and making tech more integratable with our natural environments so we’re not choosing screen or sunrise, but can have both.
Bionic means being extraordinary, having superhuman capabilities. Better human-computer interfaces can turn us into better humans. What we do with that power is up to us.
Thanks for reading! I’d love to hear your thoughts and ideas — email me at mallick.skm@gmail.com or find me on Twitter @shmallick.
