The Dawn of Holographic Telepresence has Arrived

Holographic Telepresence as envisioned in “Revenge of the Sith”

Science fiction often precedes science fact. Princess Leia’s holographic projection from R2D2 predicted human holographic recording. The Jedi council from “Revenge of the Sith” envisioned real-time holographic communication. While the display technology for free-floating holographic apparitions remains elusive, a pipeline for the capture, reconstruction, transmission, and head-mounted display of live human holograms has been achieved. While at Microsoft Research, I began a project four years ago that culminated in Holoportation, a widely viewed YouTube video that demonstrated holographic telepresence.

Video chat systems like Skype combine voice and video to connect distant parties. While seeing and hearing someone remotely is compelling, the spatial constraints imposed by 2D screens and cameras limit the effectiveness of the shared experience. Holographic telepresence, on the other hand, is all about sharing a sense of presence — proximity, intimacy, and a sense of interpersonal space, can be achieved remotely.

Every advance in the sensory enhancement of human-to-human communication has been widely adopted and soon taken for granted; writing, printing, the telegraph, the telephone, radio, photography, movies, television, video conferencing. Holographic telepresence is the next step; one whose time is at hand.

Several recent breakthroughs are behind making holographic telepresence a reality.

1. Real-Time Depth Acquisition. Digital cameras record a color at each pixel of a digital image. A depth image stores the distance along an invisible ray, from the focal point of a camera through a pixel to the nearest object hit by the ray. “Real-time” means that a new depth image can be acquired at least 30 times per second. This technology became widely available with the introduction of Microsoft’s Kinect device in 2010. Since then, several similar “depth camera” devices have appeared, including the Intel RealSense camera. Strictly speaking, these are not really depth cameras, but rather ordinary monochrome cameras combined with infrared light projectors and filters that utilize various software algorithms to estimate the depth at each pixel.
2. Programmable Parallel Processing. Driven by the insatiable desire for realism in videogames, processors capable of executing thousands of floating point operations simultaneously, billions of times per second, have evolved both in terms of performance and programmability over the last two decades. This massive parallelism has ignited revolutions in many fields of science and engineering, including artificial intelligence and machine learning. It is this abundant compute capability that we leverage for holographic telepresence.
3. AR/VR Head Mounted Displays. Devices capable of tracking the orientation of a user’s head, and presenting geometrically correct synthetic stereo imagery are now widely available. These stereo images are fused in the brain to create spatial awareness, and a sense of presence.

Of course, a great deal of “secret sauce” is needed to combine these ideas into a working holographics telepresence system. Many challenges remain to bring this technology into the mainstream, and improving image quality will keep researchers busy for years to comes.

Why I joined 8i

Given the complex nature of this problem, joining forces with a company whose technical expertise and vision overlapped so closely with my own, just made sense. I feel as if I am continuing the journey I began four years ago. I am now part of world-class team whose only mission is to make this technology a reality, and forever change the way people communicate. At 8i, we are truly turning science fiction into science fact.