VR We Now?

Advances and challenges in cross-immersive analytics.

Image by S. Hermann / F. Richter from Pixabay.

Where are we now in mixed reality? Cross-virtuality analytics (XVA) is a novel field of research that combines different devices such as 2D screens, AR displays, and VR displays to enable immersive visual analytics by seamlessly integrating different devices and supporting multi-user collaboration. This article is based on a research survey published in EuroVis ’22 that analyzes existing work in cross-virtuality analytics.

Virtual screen setup.

Reality-Virtuality Continuum (Reference).

All devices can be classified using the reality-virtuality continuum (RVC) as defined by Milgram and Kishino in 1995. RVC ranges from conventional workstation-based visual analytics (VA) on 2D screens to 3D or immersive visualizations that employ augmented reality (AR), augmented virtuality (AV) and virtual reality (VR). Interactions and transitions between these different stages of RVC provides us with cross-virtual experience.

1. virtual reality framework for surgical applications 2,3. Visualization for smart manufacturing

1. Immersive Collaborative Analysis of Network Connectivity 2. Volumetric Data Visualization and Analysis by Combining Augmented Reality.

Applications in certain fields — such as network analysis, volumetric data, biology, or medicine — often have either inherently spatial or complex data. Thus, there is a need to combine the well-established 2D screen-based methods with combination of other stages in the RVC for optimized cognitive reception.

Levels of Cross-Virtuality (XV)

The survey identified four categories based on the interconnections between RVC stages.

Spatially agnostic XV: When two or more systems share common data but are NOT related by real world physical position and distance.

PC & HoloLens interaction for CERN data. Both are visualizing same data but are not related by their real world positions (reference)

Augmented displays: Systems that augment one stage of RVC with the help of another stage and are related by their real world positions. Common examples include combination of 2D surfaces with AR and/or VR systems.

Pulling a 3D model from a computer display (reference)

Networked XV: Systems that interconnect users in same or different stages of RVC.

Two users simultaneously interacting with a 3D object (reference).

The DataSpace perceived locally with AR (left) and from the remote user in VR (right) (reference).

Transient XV: Systems that allow smooth transitions between two or more stages of RVC. Implementing these systems is challenging.

User exploring an example scene in spatial augmented reality (left), see-through AR (middle) and VR (right) (reference)

(a) Simple Cut (b) Portal (c)Fade ( Exploration of Visual Transitions Between VR & AR)

Transition and Interaction Techniques in XV

Transitions guide the user during the shift from one stage to another so that users remain focused on their tasks. Common transition techniques are Portals, Fade, Offscreen transition, SimpleCut, Vortex and FastMovement. Each transition has its advantages and disadvantages which are explored in the following research (refer to table below).

Different transition techniques in comparison (reference).

For information visualization there are seven unique interaction techniques for XV Analytics:

1. Select — mark something of interest
2. Explore — show user something else
3. Reconfigure — show a different arrangement
4. Encode — show a different representation
5. Abstract/Elaborate — show less or more details
6. Filter — show results with conditions
7. Connect — show related items

Collaboration in XVA

The survey study found 64 papers research collaboration within the RVC. These research papers can be categorized based on their collaboration environment as:

• Same time, Same space
• Same time, Different space
• Different time, Same space
• Different time, Different space

Two users simultaneously interacting with a 3D object. Participants share the same physical space regardless of the interface. The techniques allow for manipulations beyond the arm’s reach. (reference)

Visualization in Virtual Analytics

37 different studies were classified into four categories based on their input data type and the the visualization technique they used

1. Trees, graphs and networks visualization

Highlighting and selecting nodes. (a) The orange node is being highlighted over by the user, ready to be selected. The edges connected to the highlighted node are also highlighted in orange while the user is pointing it with the controller. (b) The yellow nodes are the selected nodes by the user. © The light green nodes are pre-highlighted by the system to show task information to the user. This figure shows a path task. (reference)

2. Multivariate data visualization

Interactions with the shelf layout (top) and contained data visualisations (bottom): (a,b) adjusting layout aspect ratio by “grabbing” and moving a shelf post, (c,d) adjusting layout curvature by moving both posts, (e) rotating multiples via the ViewCube, (f) brushing a single data point, (g) brushing an axis using both controllers, (h) brushing a volume selection on all axes, (i) filtering on the y-axis with cutting planes. (reference)

3. Spatial data visualization

FiberClay is an immersive multidimensional visualization system. The user can navigate into trail sets to gain a better understanding of dense and complex datasets. (reference)

4. Geospatial data visualization

GeoGate visualization strategy: (a) 2D parts of maps that have difficult to interpret data set associations, (b) UI focuses on difficult areas, STC supports time and space in different dimensions to separate the data, and (d) user navigates the space using the UI (reference)

Challenges & Opportunities in Mixed Reality

In addition to the usability of systems, systems need to be evaluated for mixed reality specific constructs such as simulator sickness, cybersickness, presence and factors such as mental demand, cognitive load & plausibility of interacting. Almost all studies referenced in the survey used self reported feedback questionnaire. Alternatives or enchantments to self-reported data such as physiological measurements needs exploration.

Depending on transition technique different metaphors are conveyed to the user which can influence their perception of the system.
Future work could investigate desired effect of different transitions, for example depending on how subtle the transition is it can be used to maintain or break user’s presence in the virtual system

Space utilization across the RVC is a constraints for viewing large datasets or multiple views. There is scope for exploring different space constraints of devices across different visualizations.

Consistency of visualization and interaction techniques across devices. For the same data viewed on different stages in the RVC or in a collaborative setting, visualization and interaction techniques should match as closely as possible between the different scenarios or at least have consistent mental model regarding the outcome. Such translations of encodings and interaction techniques between devices are contrained by the device and above mentioned challenges and are an open challenge.

Conclusion

The literature survey revealed that not many systems exist so far which cover all these core aspects of XVA. There are research opportunities to fill the knowledge and give directions for future research.

References

• B. Fröhler et al. A Survey on Cross-Virtuality Analytics. Computer Graphics Forum, 2022. https://doi.org/10.1111/cgf.14447