I saw my first mobile augmented reality demonstration in 2004. A small European startup had just sent our Developer Relations team at NVIDIA a video of their app — an interactive dinosaur pet that appeared on a mobile phone display when triggered by a paper-based QR code. For the most part, it was pretty amazing stuff in an age that pre-dated iPhones, Android, and high powered mobile graphics processing units.
What wasn’t amazing was the level of visual detail. The display was tiny and had very low resolution. The CPU was eking out just over 200 MHz of performance and had less than 100MB of HD space which meant you couldn’t render or store much locally. And with 3G networks still in rollout phase in many countries, there were limited options for reasonably streaming 3D to a mobile phone, even basic 3D like the model I was looking at.
Unfortunately, this was just one in a long line of cool immersive applications that was too far ahead of the infrastructure required to support it. The visual experience simply couldn’t match the level of expectations associated with a futuristic immersive AR interaction.
14 years later, while the mobile and graphics industries have come a long way, the fundamental challenge of delivering high-quality 3D graphics to augmented reality devices is still an unsolved problem. And like the mobile phone industry back then, today’s mixed reality market is fragmented, with dozens of devices on the market, all struggling to deliver the performance to match the expectation of a hi-def, fully immersive and interactive visual experience.
Companies like NVIDIA, Intel, Qualcomm Technologies, and others are driving the future of miniaturized chipsets as quickly as physics will allow them. However, even the cadence set by Moore’s Law has become irrelevant in a time when the width of transistor designs are now being measured by the width of a single atom.
It’s reasonable to surmise that mobile processors won’t catch-up to the computational capabilities of a high-end graphics rendering workstation anytime soon. This means that untethered mobile and mixed reality devices will continue to have significant client-side limitations for the foreseeable future when trying to render large, complex 3-dimensional graphics that were originally designed on high-powered PCs.
On the delivery side of the equation, ARCloud services, as explained in some great articles from Ori Inbar and Matt Miesnieks last year, will provide the infrastructure required to deliver AR content to users who can interact with this virtual content (and other users) on their mobile AR devices.
Those articles also highlight that while the server-side infrastructure is well-underway, with numerous corporate behemoths competing for a seat at the ARCloud table, there is still the unsolved challenge of how to deliver high-quality 3D content that mobile devices can actually render. In other words, what good is the theoretical ability to serve a 30 million polygon virtual model of a dinosaur walking the streets of San Francisco when my device simply can’t compute it?
Thankfully, there are a number of companies, albeit a pretty small number, who are developing solutions that help bridge this gap for a variety of consumer and enterprise industries. The company I work for is one of them, but at the risk of shameless promotion, I’ll focus on the problem that companies like mine are working on in a more generic sense, and why it is so difficult to solve.
Getting a massive 3D model to run on a resource-constrained mobile device is not only hard — in some cases, it’s actually impossible. Devices like the Microsoft Hololens, for instance, have suggested polygon thresholds of somewhere between 100,000 and 150,000 triangles per model. Sure, for some models where complexity isn’t that important this is okay. But for large-scale objects or structures with lots of curved surfaces, you’re going to run out of triangles quite quickly. An average architectural model of a typical commercial high rise is no less than 25 million triangles.
In addition to complexity, there is performance. 60 frames per second is an industry standard these days for 3D graphics — anything less and many users tend to get motion sickness after just a few minutes. Anything south of 30 FPS starts to do all kinds of weird things to the brain that give people headaches and eye strain — the last thing you want when a user is expecting an incredible immersive visualization they can really sink their senses into.
If the device isn’t capable of rendering at high quality and acceptable performance, the only choice left is to simplify the model and how it can be viewed. Solutions for doing this often include the arduous task of manually decimating your geometry (a somewhat aggressive-sounding term for simplifying), managing level of detail, automated occlusion culling, compressing textures and baking in lighting, and a host of other tricks. It’s a huge, tedious and usually very expensive task that often takes full teams weeks to complete.
The good news — there are solutions out there that can automate at least parts of this process. The company I work for has developed an automated optimization pipeline that literally works by clicking a single button. Export your model to the cloud, it is quickly optimized and then made available to be viewed on a mixed reality or mobile device. It’s a great solution for companies who don’t have the hardcore 3D graphics skill set to manually twiddle all the bits required for a good optimization and delivery of a complex 3D asset to a resource-constrained device.
Other solutions on the market drill down into single aspects of 3D complexity, offering more optimization parameters than you can shake a stick at. These are good options for more 3D savvy designers who want to simplify the optimization process somewhat, but are happy to wade knee-deep through lots of knobs and sliders. Some tools focus on single object reduction, while others focus more on optimizations of entire 3D scenes. And depending on your industry, there are tools designed to fit into the very specific workflows of game developers, architects, medical professionals, automotive designers, and others.
These design optimization tools and delivery platforms will become increasingly valuable, and eventually essential, once ARCloud services are ready to start delivering complex 3D in volume to mobile AR devices. To loosely borrow from an ancient proverb, 3D optimization will make it possible to finally push that camel through the eye of a needle, and help the industry make good on the promise of incredible, immersive visualizations on mixed reality devices.
Learn more about optimization will evolve the future of AR at www.umbra3d.com
