Photogrammetry: Turning the Real World into a Virtual Playspace
When I was a kid, I used to watch a show called The Scariest Places On Earth, which is exactly what it sounds like. The crew would visit abandoned mansions, asylums, and prisons- fill the audience in on the backstory of each location- then lock some volunteer family inside for the night. As a ten-year old boy, I wanted nothing more than to visit and explore these haunts myself, but of course a lack of transportation (and a certain degree of cowardice) kept me very much at bay.
Cut to fifteen years later and a number of development teams are starting to leverage the power of Virtual Reality, and a technique known as photogrammetry, to make the dreams of my ten-year old self come true.
Photogrammetry is the method of acquiring, measuring, and combining a multitude of 2D photographs of real-world places and objects for the purposes of generating photorealistic, explorable 3D environments like the one detailed in the video below:
While it may be hard to believe, the experience shown above is actually a composite of many 2D photographs, taken from hundreds of angles, then measured, cropped, and stitched together by a software algorithm to create a fully 3D explorable environment. While this basic theory of photogrammetry is relatively easy to understand, in practice it is often an incredibly difficult and time-consuming affair, especially if your goal is to craft an explorable environment. But, if you’re looking to create a digital version of a real-world place, there is perhaps no other technique that will yield the same level of detail as photogrammetry.
One of the most important tricks to creating an immersive digital environment based on real-world locations is proper calculation of distances between objects and object features (measured on the X, Y, and Z axes). These calculations are done with software, such as Photoscan by Agisoft, which takes the images you feed it, breaks them down into “point clouds”, or clouds of interesting data points shared amongst the photos (color and depth for example), and then uses that data as a guide for combining the multitude of images into a cohesive 3D environment. Point clouds aren’t required for all photogrammetric techniques (for example, you probably don’t need much depth assessment if you’re only capturing the textures of a walled room and then slapping them onto a 3D model), but when developers are creating an entire virtual environment based solely on images, the point clouds help to asses unique object features and build more reliable 3D models.
This type of modeling can be seen in the short video for Chornobyl (Ukrainian spelling) 360, shown below.
In the ongoing race to create the most photorealistic looking digital experiences, photogrammetry is a powerful new tool in the arsenal of development teams who, for years now have been working to nail down and dismantle the effects of the uncanny valley. One of the primary causes of an uncanny valley sensation is the capability of the human brain to effectively sense patterns and note repetition. Photogrammetry’s greatest potential then, lies in its ability to limit patterns and repetition by effectively capturing the massive variability of real-world shapes and textures, thus lending objects and environments a much greater sense of realism.
With that said, this powerful new tool still has its limitations (at least in its current form). For one, moving objects cannot be captured at all by photogrammetry, which relies on the consistency of particular variables in each individual camera shot (if parts of the shot are moving, then there is not consistency). Also, the general look of glass (and anything with high levels of reflection) is so widely variable shot by shot, that the raw end result of photogrammetry is a muted, blank texture that looks wholly unnatural. Finally, there are always a handful of strange angles and shadows that need to be refined dramatically in post-processing in order for the end result to feel truly real.
As it stands, photogrammetry has to be combined with standard in-game assets and post-processing effects in order to create a believable, realistic environment. If you are creating an entire play-space based on real world locations, you can certainly use photogrammetry to capture the raw textures and scale of that particular place, but in order to make it feel real you will likely need to augment that raw information. Adding stylized filters for things like lighting, shadows, and reflections, as well as traditionally generated game assets, helps to give environments a sense of cohesion that makes them feel more “real” than what can be captured using photogrammetric techniques alone.
Besides raw graphical prowess, the 2nd greatest benefit to creators is time saved on modeling an object from scratch, (or, if you’re looking at the VR video above, time saved on modeling and texturing the massively variable amounts of trash and graffiti seen in that particular demo). If you know what you’re doing, photogrammetry can help to rapidly produce a bevy of content without exhausting your creative resources.
Photogrammetry has a number of valuable potential uses, but undeniably the most exciting one for consumers and marketers is the use of photogrammetry to help create virtual tours of famous locales from around the world. For me, someone who grew up wanting to explore “the scariest places on earth”, I couldn’t be more excited for the places this technology will be taking me.
** Written by Nathan Hoffmeier
