Foveated Rendering: Clearer VR with Less Fully-Rendered Pixels.
Two days ago, I was lucky enough to attend a local panel on Virtual Reality. The panel included creative, technical, & production directors from Red Storm Entertainment, Epic Games, Insomniac Games, & Grit Games.
While the panel was excellent on the whole, there was one thing that really stood out to me. During the Q&A section, a gentleman in the back asked:
“What are you guys most excited about seeing advance in the next 2 or 3 years?”
When it was his turn to speak, Shaun McCabe responded with zero hesitation:
“Super easy question. Foveated Rendering. That is the answer.”
What struck me about this response wasn’t just the answer itself, but the confidence, certainty, and downright bravado with which it was delivered. McCabe, a veteran of Insomniac Games and someone who has seen 3 new VR IP created under his watch in the past year and a half, seemed positive that foveated rendering was going to be the most important breakthrough of the next few years for VR. So, what is foveated rendering and why is it so important?
First things first, foveated rendering is a technique for cutting down on the power requirements of Virtual Reality by only fully rendering the parts of the display that your eye is actually looking at. The “fovea” itself is a little pit in the center of your eye that is the most densely packed with rods, meaning it it where your vision is clearest.
As you move further and further from the center of the eye, your vision gets blurrier rapidly. So when you’re entire eye is covered by a VR display lens, you’re really only able to see a small percentage of that lens at one time.
With all this in mind, the theory behind foveated rendering is surprisingly simple. Only fully render the part of the display that the eye is truly looking at. By doing that, you can cut down on VR processing requirements in a major way.
Above is an example of foveated rendering. The center (where you’re looking) is clear, and as you pull back the image gets blurrier, just like what happens with your own eye. Of course, the “center” is based on eye tracking and always lines up with the focus on your own eye.
By employing this technique, developers can get rid of what McCabe described as “a lot of wasted pixels.” Getting rid of those pixels allows for a dramatic decrease in the processing requirements of VR technology. I mean, imagine if your system only had t0 fully render 1/10 of the total pixels in your display at a time.
So why aren’t developers doing doing this now? According to Oculus’ Chief Scientist, Michael Abrash- who spoke about this subject at the recent Oculus Connect 3 conference- there are two obstacles standing in the way of foveated rendering.
- There is currently no rendering approach that is capable of taking full advantage of foveation. Traditional approaches to create variable pixel density in a single display require multiple passes (think multiple layers) stacked ontop of each other. Having to utilize multiple passes can mean actually having to draw more pixels than needed. According to Abrash, “It may be necessary to redesign the entire graphics pipeline in order to realize the full potential (of foveation)”.
- Foveated rendering requires virtually perfect eye-tracking technology, which is something we still don’t have (although there are companies, like Fove, working on it). Similarly to how imprecise or low-latency headtracking can cause nausea, imperfect eye-tracking combined with foveated rendering is likely to cause the dreaded VR sickness.
Still, while these are certainly obstacles that need to be overcome, Abrash stated at OC3 that “Foveated Rendering will be a core VR technology in 5 years.”
Foveated Rendering has the potential to a be mini-revolution inside of this already revolutionary technology. Besides simply helping lower processing requirements, it also has the ability to make VR more impressive to the average user. At that VR panel, McCabe summed it up pretty well.
“Why does this matter,” he asked. “Why does this tech buzzword matter? Well, for one, we don’t have parity between VR and console experiences. That is a huge problem. People expect things to look amazing all the time. If you’ve spent a lot of time playing Uncharted 4 or the Witcher 3, that’s what people want VR to look like. Immersion and presence can make up for this now, but in the long run, lack of parity is a really big problem.”
As someone who frequently puts people in VR demos, one of the things that I hear far, far too often from first time VR users is “is it supposed to be this blurry?” Unfortunately for VR developers, everyday people and everyday consumers do not care that VR is an emerging technology. They have been trained (just like we’ve all been trained) to expect a certain level of clarity. In order for VR to really thrive, it needs to reach those same levels of clarity. Foveated rendering is a massive step towards that, and it’s something I’m looking forward to hearing more about in the near future.
**Written By Nate Hoffmeier
