Head-Mounted AR/VR For Human Realistic 3-D Data Visualization

If you have done some data visualizations, chances are that you’ve heard the status quo about 3-D charts — they are bad and they should be avoided. There are numerous papers and books, arguing and demonstrating that 3-D charts are visually splendid but they can be mischievous about data, hence we should strive to present almost everything by 2-D plots.

To a large extent I agree with this argument. But with one very important clarification:

All these papers implicitly assume data visualization on 2-D displays, like monitors, projectors and TV screens.

On 2-D displays, the third dimension is artificially projected through a transformation matrix that distorts proportions. In other words, the sense of depth comes at the price of skewed and distorted images.

Up until the last 3–4 years, 2-D displays have been (with few exceptions) the only channel to present information. But that no longer is the case.

In this story I will share a different perspective on 3-D data visualization in Augmented and Virtual Reality through stereoscopic displays.

Before further reading, let’s watch a video that explains what we, at Progress Telerik, are after:

VR can improve readability and understanding of data significantly

AR/VR Device Types

Augmented reality devices are primarily divided into two groups — Mobile AR — that is AR on mobile phones and tablets — and head-mounted AR — like Microsoft’s HoloLens. Virtual reality devices are head-mounted only — be it through mobile phones and Google’s Cardboard, stand-alone VR headsets like Facebook’s Oculus GO, or high-end devices like Facebook’s Oculus Rift and HTC’s Vive.

We may also divide AR and VR devices into stereoscopic and non-stereoscopic. Stereoscopic devices are head-mounted only.

What is Stereoscopic?

According to the AR & VR Wiki XinReality:

Stereoscopic 3D (S3D) is a technique that produces an illusion of depth in a moving image by displaying two slightly different images to the right and left eye of the observer.

This technique produces human-realistic images that just feel natural to the brain because:

This ability is based on the characteristics of the human visual system. It interprets and processes the information gathered from the two images to produce stereoscopic depth.

Put simply, stereoscopic visualizations can safely take advantage of the third dimension without sacrificing data readability and producing distortion.

Even more, being human-realistic these experiences can take full advantage of things like shadows, lighting, occlusion and other 3D-specific effects to produce visualizations that are easier for the brain to understand and analyze.

Simplicity vs Richness

The KISS principle is very important for Data Viz, regardless of the display — be it 2-D or 3-D.

But let me put another perspective — 2-D displays force us NOT to display rich data at once. Because of the absence of the third dimension and the limited screen real estate. In multiple use case, however, we need to see the whole picture, not a small subset of it, to understand a pattern.

The human brain processes huge amount of visual information every millisecond and gets a sense of a visual scene for 100 ms. on average. Most of it is filtered but two major aspects are memorized — the boundaries of our last visual frame and the details of interest within it.

For example this beautiful picture of Dubai:

Source: AKSPIC

We see huge amount of visual information on this picture. What our brain will memorize for a small period of time for it, is:

  • This is Dubai
  • The sun is rising
  • The sea is on top
  • Lots of skyscrapers
  • Roads are generously lit
  • Probably some other stuff

Are we confused by that much of visual information (pixels on the picture)? No, it is the opposite — we are fascinated by the beauty of this city and the masterpiece photography.

My argument is that, hypothetically, there should be no limits for big and wide data to be presented in a similar way on 3-D displays, bringing sense of delight, immersion and better understanding of important insights through different levels of detail.

2-D vs 3-D

Let me compare two different visualizations. A typical sales data has multiple dimensions like Product, Date, Volume, Region, Sales Representative and much more. The most common approach to visualize, say the Product, Date and Volume dimensions, is by representing the Product dimension by multiple series — like Line Series for example:

X-axis is time, Y-axis is sales volume and products are plotted by multiple line series

For several products that’s just fine. But if the business that we monitor sells multiple products, then we will have multiple lines that overlap with each other, creating somewhat unreadable visualization. To overcome this, Data Viz tools usually provide means to show or hide products and monitor only those of interest.

I am not saying that this is not working. It is, given the 2-D displays that we use today. But I am arguing that there is a better way:

The Products dimension is now utilizing the X-Axis and the time is plotted on the Z-Axis

This screenshot is taken directly from the Unity3D player. I put it here for two reasons:

  • To illustrate that 3-D charts can be valuable for even simple use cases
  • To illustrate why 3-D charts are bad on 2-D displays — the farther the bars are from the Camera, the more they are distorted and skewed
Unfortunately, no one can be told what The Matrix is, you have to see it for yourself

There is no way for me to demonstrate the human-realistic third dimension on a 2-D display. No matter how many screenshots and/or videos I put here, you will have to put a stereoscopic HMD to truly experience it.

Virtually Unlimited Screen Real Estate

One of the true powers that both AR and VR unlock, is the unlimited screen real estate.

The biggest screen where you can plot data is the real (AR) or virtual (VR) world.

In VR I can spin-off as many data plots as I need to, arrange them in dashboard(s) and analyze them.

Let’s take a look at this example in Tableau:

Source: Tableau Public

The 2-D plots make a lot of sense and there is no reason for us to try to add additional dimension.

The same data, using the very same 2-D plots, but displayed in multiple dashboard, arranged in the VR space:

VR provides unlimited screen real estate
Please note that this visualization is displaying the same data in each dashboard. The demo was created solely to illustrate the unlimited screen space that we have in VR.

Besides the standard 2-D plots, we may add additional visualization that compares Sales vs Profit over time, using the same color mapping per Category:

Sales vs Profit over time, bubbles are color-mapped by Category

To achieve this in a 2-D plot, we will need two Y-axes per plot, and two line series — one for Profit and one for Sales. It will work, but in 3-D it is simply easier to understand.

Interactions

There are multiple ways to interact with data in VR. We can do it via touch controllers, gaze, voice or by simply walking around the data and exploring it from various angles, much like we would do in the real world.

For the BI dashboard prototype, we have implemented touch-controller support that can be used much like a desktop mouse:

Why VR

Probably you’ve already asked yourself “Why VR and not AR?”.

Today’s stereoscopic AR devices have one major limitation — significantly narrow field of view. Microsoft’s HoloLens have like 35 degrees FoV, Magic Leap’s device is a bit wider — but still marginally better — while most of the VR devices are 100 degrees and above. Screen real estate is fundamental piece in Data Viz. Limited FoV reduces screen real estate and the “Stereoscopic” argument is belittled by the miniature viewport. Surely, things will be fixed at some point in the near future, but till then we consider VR to be better suitable for Data Viz.

And with stand-alone, high quality, and at an affordable price VR headsets like Oculus Quest from Facebook, it will become much easier to experience VR at home:

Oculus Quest will democratize the VR user space

Still, being technologically backed by Unity3D, it is trivial to deploy the very same experiences to AR as well. If you feel that, besides the narrow FoV, AR is better suited for Data Viz, please let us know.

Experiment!

I do encourage all of you to experiment with AR & VR and realize the values of the technology in greater detail. Some may argue that hardware is not there yet — to some extent that’s true, but it will soon be!

As Michael Abrash, Facebook’s Chief Scientist on Oculus, nailed it:

AR & VR are going to replace personal computers and mobile as the primary way people interact with information and each other.

We are planning to upload the source code of two data visualization applications, that we created internally to validate the value AR & VR can add to business applications. This way you will have a quick path to experiment with AR/VR and data visualization in stereoscopic 3D. Stay tuned!

Thank you for reading and please do share your thoughts and comments with us.

You can follow me on twitter: