Magic Leap XR Design: Mercury, Gemini, and Apollo — Part One
I would like to share a bit about my philosophy for XR/spatial computing, and how that thinking found its way into the Magic Leap One and now the Magic Leap Two. These are my personal thoughts, ideas, and musings (shaped by many fun debates and late nights with the early pioneers in XR and Magic Leap).
I wrote a small piece on Neurologically True Reality, which is a good backdrop for what I am writing about here.
Mercury Days
In the very early days of Magic Leap we had a small, ragtag group of artists, musicians, and scientists: comic book writers, science-fiction authors, NASA rocket engineers, system engineers, Caltech physicists, some good coders, and optics/neuroscience PhD types. To test out some of our ideas regarding Neurologically True Reality (NTR), we decided to build this thing:
The thing had a few names: The Beast, The Big Bench, The Benchtop, and the very nerdy Digital Lightfield Signal Generator (DLSG). We wanted to hack around with different signals that would go into the brain, to see if our hypothesis of NTR had any legs. To get anywhere we needed to plug the thing into actual brains (non-invasively of course), so all of us pretty much took turns hooking ourselves up to the machine. There were many, many failures and things not working at all — but I think we all felt like the Wright Brothers at Kitty Hawk, North Carolina — explorers of a new terrain, Inner Space. Many of us had the feeling that Inner Space travel was just as valid and exciting as Outer Space travel. The 100 trillion neural connections in the brain, connecting 100 billion neurons (each of which may be an amazing analog quantum computer), felt like an immense, endless place to explore (it still does). Given the intense nerd/geek density of the initial garage team and the fact that we had an actual NASA rocket scientist (who became employee #0001) led me down the path to frame it all as our Mercury, Gemini, and Apollo phases and overall mission.
The Beast was the first blip of our work in our Mercury phase, a wonderful testbed, but we needed a lot more. The Cheesehead came soon after, as an initial way to explore mobility and what we may need from computer vision.
The Cheesehead sucked — and it barely worked. It sat in the corner as a vague dunce hat most of the time (an engineer who really screwed up maybe would have to wear it, and no one wanted that).
The beautiful thing about the Beast was that it could produce something I called The Pop. We had an amazing, brilliant engineer (lets call her Yan) who could tune the Beast like a concert violinist. The Beast also sucked for many weeks, maybe some months. I remember the day Yan locked herself in the lab room where the Beast resided, and I vaguely recall her vowing not to leave until the Beast somehow yielded its magic to us. No one else was allowed in the room, and as the day became evening and then night, we began to worry about Yan and just what the heck may be happening in there.
At some point in the night Yan emerged from the lab room (where the Beast was kept) — triumphant and joyful. She had slain some metaphysical-physics-optics-neuro-compute dragon. The Beast was in tune and singing and beautiful. Each of us strapped ourselves into the framework of the Beast (it looked like a mad fusion of Clockwork Orange meets Brainstorm) and we had a moment of blissful transcendence. Objects would appear clear and sharp, and as they moved closer and closer to the eyes they would manifest greater detail at resolutions that seemed to be far past all of our hacked up gear. They would appear present and full, and they seemed to contain an existential thereness that is very hard to describe. As we moved the objects father away they would deflate and appear thinner. They would lose detail until we brought them back very close to our eyes again and they would inflate and become incredible and detailed. They would Pop.
You can experience the same thing with real objects. As you bring something closer and closer you can experience its presence and detail and vivid richness. As it is moved farther and farther back (a few feet, then a few yards, and then many dozens of yards) you can sense the object deflating and flattening — until the point where it basically looks like a 2D cutout. Screens do not work this way. As you get closer and closer to a screen, the resolution sucks and things get blurry. This was not the case with the Beast and its Pop. We were not in Kansas anymore.
We talked about the Pop, and our working theory was that our brains and sensory systems evolved in a very specific way. Spatial sound helps us identify threats from all around with pretty good accuracy, while our far-field vision could locate food (a deer, apples) a threat (an attacking lion) at good distances away-without being wasting too much brain compute power. This brain efficiency may tie into how and why we focus on things (visual focus is very likely a cue to the brain to use more or less compute power at various distances). At a far distance we do not need to render the lion beyond a 2D object- so our brain sips in just a wee bit from the available lightfield. But as the light gets closer and closer, our brain summons up more and more compute, rendering that lion in great detail. The lion about to eat you surely Pops (as did our digital objects). This was not holography (which is its own discipline). This was something else — the beginning of a dynamic, living interaction with the human brain, a symbiosis between computing and biology. Whatever this was, it was really cool and exciting.
The Beast was massive, but it showed us a true path to the inner stars. The Cheesehead sucked, and it showed us what a long, long road it would be (perhaps an endless road). Next came the WDx series (Wearable Demo 1,2,3,x).
The WDx series did not suck, but it was big, and it had a Matrix style giant hose that connected you to some serious workstations. The WDx did not have the Pop of the Beast, but it is where we began to learn how to make a whole bunch of systems (including computer vision) come together. The Beast, The Cheesehead, and the WDx series were all part of our early Mercury Phase. We knew all of that know-how needed to culiminate in a first thing that we could give to developers and creators and early partners. That first thing would evolve into the Magic Leap One, which I considered to be the key milestone of the Mercury launch. A few of us at the beginning, those who best understood the Pop and would it could mean, could see what our Apollo could be, one day. This vision excited us deeply and fueled the idea that we could one day put it all together: Mercury, Gemini, and then Apollo. Each step growing almost exponentially in system complexity. Each step needing 5–10 startup companies worth of breakthroughs in design, software, physics, and engineering, and overall invention. We all knew that what everyone in the world ultimately wanted was Apollo, which would be able to summon NTR in a seamless and wonderful way. To reach the top of the mountain one must climb in stages, make base camps, and avoid avalanches (and yetis) when and where possible.
A lot of people ask: why? who cares?
I think of it this way: the exploration of Inner Space is one of neuroscience, imagination, consciousness, physics, and adventure. The connected Inner Space of billions of human minds is rich and boundless-and may yield discoveries that far exceed our travels to the farthest edges of the known Universe. In NTR what we perceive externally is all internally created and generated (meaning — that is how everything is). The Mars of Outer Space may be just a dead planet that once had water-and perhaps a pretty terrible way to spend years of your life (stuck in a tiny mobile home thing in a cold desert). The Mars of Inner Space can be full of life and have anything we want in it-and can be shared with billions. You could travel to it in fractions of a second, and then be back home. NTR could harness and unleash a Renaissance of creativity and imagination. It could be a way for the human race to work on improving itself, and to connect in incredibly cool ways.
It may also be that the Universe has placed a hard limit on our ability to create our own NTRs. The deeper you go in this field, the more you see this maybe true. Perhaps we can only create sub-NTRs that are good, but never as good as the Prime NTR. That is ok. I think we can live with that. The Universe, our World 0, is the gold standard. We can be happy mining the silver and bronze.
When I think about what is possible and what still lays ahead, we are now perhaps leaving the Mercury Phase and about to begin Gemini. But we still have a a lot of mountain to climb until we reach the Apollo of NTR (which perhaps can never be reached, but we chase infinities anyway). We (all of us) may be in the Gemini phase forever. Or we may yet find a wormhole to the Apollo level-maybe when we are ready as people, as a human race (one could easily argue that we are far from being ready).
There is a lot more to discuss, but that will be for another time :-)
R. Abovitz
March 11th, 2022
Earth
