Pulling Games Into The Third Dimension: Virtual Photogrammetry in the Looking Glass

Hi! I’m Evan, an engineer at Looking Glass Factory. This post is about how to take a holographic video game screenshot!

Capture scenic vistas from the comfort of your own home! (Assassin’s Creed: Odyssey)

But first, a little background.

Here at the Looking Glass lab in Brooklyn, we’re always hard at work thinking up new ways for developers to bring their content into the Looking Glass.

So far, all the official software built for the Looking Glass uses the HoloPlay Unity SDK, the HoloPlay.js library, or (coming soon!) the HoloPlay Unreal Plugin. We’ve been floored by the amazing projects developers have brought into the Looking Glass using these tools.

However, the Looking Glass isn’t limited to showing content that’s created with our SDKs; it’s a powerful tool for displaying generic multi-view 3D content. On a physical level, the Looking Glass works by taking a set of 2D images captured at different viewpoints and re-broadcasting them to the same perspectives. This means that someone looking at the device from a specific angle will perceive a 3D scene just as it appears from that angle in real life.

For more information on this process, please check out our How It Works documentation!

If you’ve been following our discussion channels, you may have kept up with the pipelines designed in-house and by our intrepid developer communities to pull iPhone portrait images, 360° videos, multi-camera 3D video, drone photos, and even composite images of the rotating moon into the holographic world. All of them use some variant of the following method:

• Start with 2D representations of an object (virtual or real) from different perspectives;
• Composite them into a single image, known as a ‘quilt’;
• Apply a fragment shader on your computer’s GPU which interleaves the pixels such that they become coherent when viewed through the optics of the Looking Glass.

Perceiving different views on the Looking Glass as the angle changes.

To unlock this ability for everyone we’ve built our own Lightfield Photo App, which is a simple tool that allows you to start with a set of images and composite them into a 3D hologram, no coding required.

But without photogrammetry hardware, how do we get the images?

Enter the virtual photography community.

Until now, it’s been tough to convey how software not built for the Looking Glass might appear inside it. Most games and 3D software built for a 2D monitor only provide one virtual camera, whose position is controlled by the game engine.

But it turns out there’s a ton of people that love taking beautiful screenshots of video games — and most 3D games have some sort of developer console you can access to change your character’s position. Why not write a script to move your character repeatedly along a linear path, taking a screenshot each time?

That’s how my initial approach worked: an AutoHotkey script that used the developer console in Portal and Half-Life to scoot back and forth, taking screenshots once per step.

I guess you could say it was a triumph? (Portal)

But then I found out that there’s a remarkable body of work behind forcing cameras in commercial games to change orientation, focal distance, and even output resolution. There’s an entire virtual photography community taking screenshots of video games with tools like Nvidia Ansel, which is designed to allow the player to move the camera independently of the game character, yielding gorgeous game screenshots impossible to take while playing the game itself.

Of course, the most intrepid screenshotters don’t feel the need to limit themselves to games supported by Ansel. So, digging a little deeper, I found Frans Bouma’s brilliant Injectable Generic Camera System plugin, which actively hacks into a running video game and overwrites the camera parameters in memory. Best of all, it’s open source, which facilitates easy ports to new games — so I modified it to bind a key that generates an entire set of lightfield photos with one button press.

Here it is in action. I freeze the game, hit a key to move the camera automatically across the scene, and save out a set of 45 screenshots that can be imported into the Lightfield Photo App.

Alexios has a rough day. (Assassin’s Creed Odyssey)

So attention, virtual photographers—as long as you have some way to manipulate your game camera, you can capture any 3D software and pull it into the Looking Glass — and if you’ve been using Frans’ camera tools, you can clone the latest version and test the Looking Glass functionality today!

Horses in the front! (Metal Gear Solid 5)

Are you a virtual photographer? If you’re curious about how the Looking Glass works, want guidance on setting this system up yourself, or have tips on how I can up my game, feel free to hit up our forum or Discord server!

Not a gamer? If you’re still interested in creating or viewing content like this, you can also check out our brand new platform for pre-rendered content (MadeWith Looking Glass) — where members of our community can share their holographic content with others. We’ve already seen some other developers uploading their game captures, and I’ve posted some personal favorites as well.

To the future!

evan@lookingglassfactory.com