Inventing the Looking Glass

A look into the magical chaos of R&D, performed in pursuit of commercial holographic display

Originally published at https://blog.lookingglassfactory.com/announcements/inventing-the-looking-glass/

Magic is sometimes hidden in the chaos

Is it possible to perfectly replicate reality? Can light truly represent three-dimensional objects, people, and places? Are the holograms of Blade Runner and Star Wars five years away. Or 50. Or 500?

Back in 2017, no one in the world knew the answers to these questions — least of all us. ‘Us’ in this case was me and a small group of about a dozen engineers in a company I co-founded called Looking Glass — the pre-eminent and possibly only holographic display company in the world at that point. Definitely the only one in Greenpoint, Brooklyn.

As 2017 rolled around, the Looking Glass team and I had already spent a couple years building hundreds of prototypes and running thousands of experiments on all manner of systems that might yield a glasses and goggle-free view of a three-dimensional world. We made flapping systems, spinning whirly-gigs, re-imaging optical circuits — all sorts of stuff that didn’t quite work, but always led us to the next experiment. We believed, as Edison did, that “I have not failed. I’ve just found 10,000 ways that won’t work”.

A brief history of holographic display in Looking Glass Factory

March 2015: A prototype we codenamed the “Hypertube” using an array of rapidly spinning and flashing LEDs. Was hard to imagine this scaling so we moved onto other ideas.

2015 Prototype: Hard to tell, but that’s me in there! Encouraging — no moving parts! — but felt too much like the lo-fi videophone prototypes of the 1960s — far from good enough.

2016 Prototype called Volume running an early version one of the first games we made (“Valley Racer” by Kyle Appelgate), in which the 3d effect is achieved by scattering light off of a dozen light guides. Closer, but it weighed over 100lbs. p.s.

2016 Prototype “Volume” showing a depth video recording of my co-founder Alex Hornstein. Like the previous Volume prototypes, this one also contained an index-matching fluid — but it was hard to imagine displays of the future being filled with liquid, so we moved onto other approaches.

2017 Prototype called “Holoplayer One”. We thought this was it. Turns out, it was close, but too complex with too many resolution tradeoffs and we shelved the system after a few months of testing. However, the software we developed for this prototype was the key to reaching the final destination.

We weren’t the only ones obsessed with this. The past century of work on cracking the problem of perfect three-dimensional display — a technological holy grail to many — had consumed the professional lives of thousands of inventors, engineers, and researchers. These ingenious creators had worked out countless brilliant techniques to reach this holy grail — igniting the air with lasers in 3d matrixes, firing high speed projectors at wildly oscillating platens, and modulating pixels on the scale of the wavelength of light with soundwaves. Even a partial catalog of the attempts in this 2014 paper is truly astounding. But, as of 2017, all that humankind had to show for this effort was a vast assortment of “maybe-this-could-work-one-day” experiments that were either too complex to manufacture or simply not good enough.

The closest analogy I can think of is the pursuit of heavier-than-air human flight back at the turn of the 19th century. This was something that had been chased for millennia. And while everyone knew birds could fly, no one knew if people could — there was a very real chance it just wasn’t possible. Inventors, scientists, and entire government labs made thousands of flying machines that felt like they *should* work, but mostly didn’t.

And that’s the situation we were in with the pursuit of holographic display back in 2017. We could get crude three-dimensional images to glow through the fog of finely tuned optical arrangements or deafening machines of our own design, but nothing was good enough to get off of the lab bench and into the world.

And then —we had a breakthrough. In 2018 we finally crossed the threshold with a system we initially called the “Holopad”, but quickly and blissfully gave our company’s name instead — the Looking Glass.

The 2018 breakthough: The Looking Glass is born. Recording by Ikuo Nakamura.

The key to cracking the problem of group-viewable, no headset-required, genuinely three-dimensional dynamic display that feels completely real was to add one ingredient to what everybody already knew with two-dimensional displays. That ingredient turned out to be directionality. By controlling the color, intensity, and directionality of millions, and in some cases hundreds of millions of points of light, a perfect reproduction of any three-dimensional content, real or imagined, became suddenly possible. The requirements for complete realism are substantial, in that they include not only stereopsis at a variety of view points, but also the ability to render specular highlights, occlusions, and realistic opacity — everything from reality. And that was finally achieved in a product that could be manufactured at scale.

I’m leaving out considerable nuance here, and perhaps the details are where the magic really is. But that’s another post for another day. Needless to say, with an overnight success that was a century in the making, all of the other primary methods of approach on this goal, like scattering light off of physical material or two-stage excitation of a gas or any of the other things we’d been trying collectively as a species over the decades — all of this was wiped away. It was now clear that controlling the directionality of light in a very particular way was the key.

2019: Shown here is our 32" Looking Glass 8K running a depth video recorded by team member Missy Senteio, in which 100 million points of light combine to create synthetic light fields that can update at 60 times a second in full color.

After that core invention problem — answering the question of if this was even possible — was solved, holographic displays powered by our light field technology started down an exponentially decreasing cost and increasing quality curve that they are very much still on today.

This is thanks to three tail winds.

1. the ever decreasing cost of compute power — specifically GPUs;
2. the vast decrease in the cost of dense pixel arrays, a result of the past 10 years of cell phone and tablet evolution and;
3. the vast availability of 3D content and game engines like Unity and Unreal that let millions of people build three-dimensional holographic-capable apps.

Nowadays thousands of people are actively using holographic displays and the flywheel of creation is entering a new phase. Researchers who have built applications for VR or AR headsets like the Hololens 2, for instance, can now run many of those applications in their own holographic display without needing to gear up. What’s more, over the last few months we’ve seen an increasing number of developers push the boundaries further and faster than we ever dreamed possible with holographic applications that are helping to connect distributed teams in fields like telemedicine, drug design, medical applications — any and all ways to replace those ever-present 2D Zoom walls.

https://twitter.com/LKGGlass/status/1287803087646842881

If you work in R&D yourself and are curious to learn more about what we’ve developed in the field of holographic display — and what you can develop on top of this new type of interface — we’re hosting a webinar entitled “The State of Holographic Displays in Research and Development Today” on October 6 at 1pm ET. Sign up here today! https://bit.ly/2Eb0wt7.

To the future!

— — — —

Inspired by movies in the 80’s and 90’s, the author Shawn Frayne has been reaching towards the dream of the hologram for over 20 years. Shawn got his start with a classic laser interference pattern holographic studio he built in high school, followed by training in advanced holographic film techniques under holography pioneer Steve Benton at MIT. Shawn currently works between Brooklyn and Hong Kong where he serves as co-founder and CEO of Looking Glass Factory.