Episode 5: Want to Climb Everest With Your Phone? Matterport Unlocks a 3-D Experience from 2-D Photography
A conversation with Matterport CEO RJ Pittman.
In Episode 5, Lux Capital Managing Partner and Co-Founder Josh Wolfe interviews RJ Pittman, CEO of Matterport (a Lux portfolio company) at Lux’s offices in NY. Watch the entire episode above, and what follows is an edited version of the interview transcript.
Josh: Hey everyone, I’m Josh Wolfe, managing partner and co-founder of Lux Capital, a venture capital firm that invests in emerging science and technology ventures at the outermost edges of what’s possible. This is our new series, Futura, where we’re gonna introduce you to the rebels of science and invention who are turning sci-fi into sci-fact. Today we’re sitting down with RJ Pittman, CEO of Matterport, whose technology digitally captures and visualizes physical places and spaces in 3D on any screen, from a phone to a VR headset. RJ was most recently chief product officer at eBay and in the past was recruited to Google and Apple, having first met Steve Jobs on a Whole Foods checkout line in 1991. On today’s episode, we’ll show you how Matterport turns physical atoms to digital bits.
Josh: So, one of the really interesting things about the history of technology is you get something that was invented in one area, and then all of the sudden, that thing becomes super cheap, and then it gets re-used in another area that was heretofore never imagined. So, video gaming. You take these 3D cameras, and then you get to repurpose them into this entirely new industry. Tell me about Matterport, and how the technology works.
RJ: It starts with a 3D capture device. And to Matterport, historically, that really didn’t exist as a product as you said. So the company went out and created an amazing capture device that can scan a room in 16 seconds and produce a dimensionally accurate 3D representation of that space. And then you can take that space and augment it. Add tags to it, descriptors, identify what are all of the objects and furniture and other pieces in the room, and share it, publish it. That gives way to a wide variety of use cases, from real estate to construction to insurance and more.
Josh: So you’ve got the hardware and the lens, and then talk about the software piece of this.
RJ: Alright, so the software is what sort of brings it all together. Because, let’s say you’re capturing a house. There are a lot of different rooms that you’ve got to capture. And then somehow you’ve got to bring it all together into one cohesive home. How does it know when the bedroom connects to the hallway that connects to the bathroom that connects to the stairs that go down to the living room and the kitchen? And how does it form all of those pieces in a true model of the home with that same level of accuracy? And that’s where our AI engine comes in. That’s where “Cortex”, we call it, plays a very important role in bringing all of these pieces together in the exact orientation necessary. And then stitch together, seamlessly, to give you that totally realistic experience of being in the home.
Josh: So you start with hardware that nobody else had. And then, over time, as the technology gets better and better, we all have cell phones and people are increasingly having depth capture, now you move to Cortex which is the software platform that’s stitching things together. What’s the key advantage of that?
RJ: The real breakthrough, the part that I’m especially excited about is not just what Cortex has done to help us stitch together these great homes and buildings and structures that we have today, it has now seen over 1.5 million spaces, from all of our capture customers out there that have just been feeding it homes and buildings and stuff all over the world. And we built a deep learning engine with our computer vision scientists that are doing some amazing, amazing things to basically study and train on all of that 3D point cloud data. And the mesh data. So what does that allow Cortex to do? It allows it to recognize three dimensions from 2D. So, this is what we call “3D reconstruction.” And the power there is unlocking the ability to create a similar three-dimensional experience simply from 2D photography.
Josh: That’s never before been possible.
RJ: That has not been possible before. And certainly not with any level of accuracy that you could count on, or that you would believe if you were immersed in that space, digitally.
Josh: Now, we love that rebel spirit, and we also love rebel scientists and technologists and inventors. Is there something that you are breaking the laws or the rules when it comes to technology and physics in the way that the system works, that was never possible before?
RJ: If you look at a photo today, while you don’t see it in 3D, you can imagine what that space might be like in 3D. You have a sense of it in your mind, but it’s not something that comes to the foreground. And this is what artificial intelligence is all about. Can we actually systematize? Can we effectively program that thinking, that sort of imagination, that extrapolation, or in this case that 3D reconstruction, into the foreground and into our conscious mind? Absolutely. We’ve seen the power of AI go after the greatest chess champions and the game of “Go” and defeat them all, right? By getting inside the minds of humans and the way they think and how they strategize, and even how they emote in those games. It all plays a roll. So, when we talk about deep learning and training on these data sets like this to create something like a 3D world from a photograph, we have to take all of those aspects into consideration.
Josh: What is the actual mechanism by which this all works?
RJ: So, this greatly oversimplifies it, but one of the key techniques in deep learning is reinforcement learning. And what I mentioned earlier about that advantage we have of capturing over 1.5 million spaces around the world…
Josh: That’s the key data piece.
RJ: We put Cortex against that. And our data set that we’re training Cortex against is that reinforcement learning. So, it’s studying every space. It’s studying the depth, the corners, where the walls and ceilings and planes of that space come together, and the objects that sit inside those spaces. And so that it is much more able to recognize and anticipate in a 2D photo exactly where those dimensions might unfold.
Josh: When you think about the far future … Near future, we’ve got real estate, we’ve got insurance, we’ve got the ability for inspectors to come in and look at a base model of what a home has versus how it was damaged and make the comparisons. Real estate, the ability to do effectively remote viewing and buying and selling. What is the far-out future vision? Where we are capturing 2D dimensions, or 3D dimensions of the world, and recreating it in this simulacrum, basically in The Matrix? What is the far-out vision of what that enables?
RJ: I think it makes the world a smaller and more accessible place. One of the places that I would love to scan is Everest. You’ve seen movies, you’ve seen IMAX films of the ascent up into the death-zone, and it’s a fascinating, curious, mysterious and deadly place. But imagine being able to go one step further, and actually drop you right in the mountain, right at the Hillary Step in dimensionally accurate 3D. As if you were climbing that last thousand meters to the summit. And then, on the summit, and back down.
Josh: RJ, I love the fact that you are mixing the digital and the real and blurring the lines. Thanks so much for spending time today.
RJ: Great being here, thanks.
Josh: Well that’s it from us today. I want to thank the rebel scientists and inventors at Matterport for giving us a sneak peak of the future. I want to leave you with two recommendations, one sci-fi and one sci-fact. The sci-fi movie is Strange Days, a ’90s classic. The sci-fact book is The Beginning of Infinity by David Deutsch. If you want to get in touch with us, reach out at Futura@Lux.vc. We’d love to hear your crazy ideas and inspirations.
This episode’s sci-fi and sci-fact recommendations:
Strange Days (Sci-fi) and The Beginning of Infinity by David Deutsch (Sci-fact).
