An Interview with Mitch Gelman

Chief Technology Officer at the Newseum

What are some of the most interesting projects you’ve seen so far in the VR space? What ties them together?

What ties them together is the opportunity to use emerging technology to tell stories, to communicate in new ways. If you go back to newspapers, magazines, radio, television, the internet, each of those mediums allowed you do something, in terms of how a story is presented, in a way that was unique and specific to that medium.

Now with VR technologies, you’re beginning to see the emergence of the next medium, and the stories that are told through these features are as unique and exciting and different and special as TV was from radio, as radio was from magazine, as magazines with their big photos were from print, and it pulls a lot of different elements together, so that’s why I’m attracted to the potential of what people are referring to as VR. When they are referring to VR, what they are really referring to is a variety of different types of technologies that allow you to tell stories in different ways. As of now, there’s really three or four levels of VR the way it’s being used in the vernacular. At the Newseum, we’ve been exploring all of those different levels.

What are the levels of VR?

It starts with basic VR, entry-level which is 360-degree spherical video. You put a camera in a place and it captures the entire scene. It’s almost as though you as a person are standing there with eyes in the back of your head. An example here is The New York Times’ “Fight for Falluja.” The Newseum just shot something like this on Inauguration Day — both at the center of the protests, as well as at the center of the inaugural parade. The President’s inaugural address. It showed two perspectives of 180 seconds of America. This level of VR can be viewed in different ways: Facebook, with touch controls, touch screen, etc.

Or, this can be viewed in a headset that allows you to essentially look around and be more immersed in the experience. This forms the second level of VR.

The third level is what they refer to now as “room-scale”, or more immersive, walkaround VR. This is like the Pearl Harbor experience or Berlin wall beta that the Newseum has been working on. This is best done with Oculus rift, touch controls, motion sensors, or through an HTC Vive system. It allows you to put on a headset and go inside a scene, move around inside it, and you can get closer to or further from, or walk around the 3D individuals or objects in the scene. That requires an amount of game development or unity development or unreal engine development that makes it much more immersive than just looking around a spherical video. This enables one to move around a room in 1941 and touch objects, or going to a scene when bombs are falling.

The last level is augmented reality, which right now is a sort of sibling to VR. VR takes you and brings you into a virtual environment that removes you from the place you are in. AR brings the virtual experience into your world. It brings that into the place that you already exist in. So you can bring figures from history or basketball players, or anyone who has posed for or been modeled in a 360-degree hologram, you can bring them into your environment. The Newseum has been thinking about ways to bring Frederick Douglass or Rosa Parks or Woodward and Bernstein — heroes of the First Amendment — into mixed reality experiences that people can participate in.

The most successful attempt at this so far obviously was Pokemon Go — capturing little creatures in different places. But the same geo targeting and almost higher level geo locating of holograms can be used to tell stories through AR in ways that are only just beginning to be discovered.

One of the shortcomings of this medium has been the lack of haptic feedback, or not being able to really interact with the reality. Do you think that as the technology evolves, these levels will still exist, or are we generally moving towards a space that is more geared toward haptic feedback and interactivity?

Yes I think haptic feedback is going to be an important part of making these experiences more realistic. They currently work very well with a sense of sight and hearing, but touch is very likely going to become part of it. There are things currently in the market that are playing with the idea of different levels of resistance and touch. So I think that’s coming very soon.

What kind of stories do you think lend themselves best to this medium?

Two types: the first is stories that benefit from your ability to be in a place you otherwise could not be in. Like Falluja. You probably are not going to be embedded with troops that are fighting ISIS any time soon. So we can travel there and have as realistic an experience as we want, with bullets flying by, without the risk of being hit by a bullet, with VR.

The second component that works well in this medium is offering someone a point of view, or a perspective that they can’t necessarily experience themselves. It puts you into another person’s shoes. And allows you to move through an idea, an issue, a place with a perspective that you may not have been able to experience yourself.

So the two elements are: putting people in a place, and then providing a perspective are what work best. When the two can come together, then I think you have a very powerful use of the medium. But that’s just in terms of storytelling. The Newseum will be doing another event here in June in partnership with the American Film institute, a documentary series that will look at more non-fiction storytelling in this area. In March we will be doing an event which looks at other uses of VR and AR — focused specifically on the medical field.

There’s tremendous work being done that’s starting to come more into the mainstream that uses VR and AR for surgical theater and training and providing assistance to both students and doctors in the field that is being used to provide comfort to patients in ways that allow them to come out of themselves and come out of their illness and be in another place. PTSD, phobias, etc. have been areas where VR has helped doctors. And pain relief, too — burn victim centers are putting people in an environment that appears cold, that has actually reduced pain as much if not more than pain medications and traditional treatment.

So all of these are techniques that AR and VR are using and we’re going to try to bring together some of those thought leaders in March to talk about and demonstrate this.

I was going to ask what you think the use of this medium is for people who are not in the news or entertainment business. It was interesting to me to hear at the panel that there was so much collaboration with sponsors — the brands and organizations that want to get into this space. Health is one I hadn’t thought of.

Neurosurgeons are now taking CAT scans and other high end telemetric findings and using them in models that can allow a neurosurgeon, for example, to literally walk through the surgery that he or she is about to perform in a room scale/size rendition of the actual brain.

What other kinds of organizations do you think have a window of opportunity here with this medium?

Travel, job training, the military and defense industry who have been using it for a long time and are continuing to. Nonprofits and advocacy groups too — any space where creating an emotional connection with an issue or idea can be helpful. It is a powerful use of the medium. Education is another field in which people are making great strides into this.

The government has had some programs recently in which they have tried to fund the use of VR and AR to provide better experiences, more training and opportunity for people, particularly women and diverse populations to benefit from STEM areas. So they’ve been looking at science, technology, engineering, medicine and really diving in and using VR to bring people into those environments to help them get experience that they otherwise may not have had a chance to have. If you look at Google Earth has done for the Vive recently, through which you can travel places, you can see the new realities that are going to be available for people to experience. It’s a very fast-moving field.

Sports is another big potential area for immersion — bringing people places where they can’t necessarily go otherwise: courtside seats at a basketball game, or even into the huddle at a football game. A lot of these are continuing to evolve, and there’s a great deal of experimentation being done to see what works most effectively.

How do you think this will play out — will organizations partner with expert VR agencies? Or will they sponsor content in, say, the NYT VR app? Or do you think they will invest in their own VR creation?

I think now you need a certain level of expertise to get others involved. There always will be agencies or studios that focus on the high level professional production. In the same way that right now there are ad agencies that do TV commercials or films. But in terms of other areas I think you’ll be seeing a lot starting to be produced in-house or at home. As the cost of equipment and production overhead begins to come down. And it is beginning to come down rapidly in the last two years. We’ve gone from equipment in which every single piece of content needed to be meticulously stitched in post-production, to very high quality no stitch cameras that produce consumer-ready spherical video.

So you think monetarily it is a gamble that would pay off? To invest in something like this?

It’s not much of an investment right now — you’re looking at approximately $500 for a Nikon camera. And it’s getting cheaper.

And for a news org? I know that the New York Times 2020 report for instance has said that they would put a lot of their resources into new mediums, and explicitly called out VR. Do you think that’s the right direction to be going in?

Yes. Not everybody is going to be able to invest as much as The New York Times in the personnel, equipment or production and development of the stories. But certainly there are various layers which different organizations can do based on budget and demand. I think it’s very important for news organizations to embrace this because it allows them to give people a much more complete view of a story.

What about sharing? So much of news now is so much about how you share. Everything moves on social. It feels like a VR experience is so personal, and the only way right now to do this with level 1 VR on Facebook is to scroll around on a picture on screen, which still doesn’t feel as satisfying as it could be.

One of the things I know Facebook and Oculus are working on is bringing people into these virtual environments together. So if your father, for instance, were in India, but we were having this event in March, if this were next year or the year after, he could actually come and participate in a virtual environment. He could sit next to you in Washington, him in India, in the front row at the presentation and you can ask questions as though you are literally sitting next to one another.

What are some of the most successful applications of this that you’ve seen?

I like some of the work that’s been done in the Vive — the Google Earth is fun. Some of the simulations and game environments they’ve done. The pearl harbor piece takes you into a world — brings you into that time in America. Those are some of the better experiences. People are learning every day how to tell stories, learning about the potential applications of the technology: medical, environmental. There’s not an industry that this isn’t going to touch. And there’s not an industry where the use of the technology isn’t going to create or develop a technique that somebody in another sector won’t be able to use to help them in their own development of VR or AR products. Everybody is learning from each other.

Given that it’s going to become cheaper, do you think there’ll be easier ways to seamlessly integrate it? For example, without a headset?

I think so, but I’ll caveat that by saying anyone who tells you who knows something for certain should be taken with a grain of salt. But it’s moving very quickly and the growth and the divergent paths that people take with the medium are going to be exciting.

What’s next on the horizon in terms of technology-focused projects at the Newseum?

We’re looking at ways to build group experiences with VR. So instead of having to wear a headset, 50 people could walk into a headset together. We’re looking into building a spherical theater. And we’re looking at ways to use AR to bring people a more immediate connection and interactive experiences with some of the with the faces of freedom who have helped define the impact of the First Amendment in the US over the last 200+ years.

So we’re working on using spherical video, walkaround AR, unity development, new theater capacity, and augmented reality through volumetric rendered holograms and mixed reality experiences to try to build for the future and engage the almost half a million students who come here each year. We’re also working with our Newseum education program to try to bring the Newseum home and to school — into classrooms and dining rooms and share it with their friends and colleagues and families as well. So for those who can’t make it here, we want to allow them to do so.