Data in Situ
Bringing data into your world.
Humans are fundamentally embodied beings. Our bodies, the objects we surround ourselves with, and the physical environment we live in play a critical role in our thinking. People tend to remember things better when they act them out bodily; spatial concepts abound in language (“I’m feeling up today,” or “I’m on top of this”), and thinking about the future causes people to lean slightly forwards, whereas thinking about the past causes them to lean backwards. It follows that allowing people to use their surrounding environment to reason about data could help us work, learn, and explore. After all, people routinely talk about being “in the situation” or “on the ground” as benefits to seeing a problem and thinking about it more clearly. However, there is currently little understanding on the strengths and drawbacks of this kind of situated analytics.
In recent work, our team at the HCIL at University of Maryland surveyed the growing landscape of Virtual, Augmented, and Mixed Reality, where computer graphics is used to create immersive landscapes that are either entirely artificial (VR), overlaid on top of the real world (AR), or embedded into the world itself (MR).
Recent developments in technology for augmented and mixed reality has made it possible for even consumers to view information in the world around us, taking advantage of the embodied nature of the human condition. Microsoft HoloLens 2, the (now possibly defunct) Magic Leap, and even our phones can be used to do more than just play Pokémon GO — we can use it to visually represent data in the world around us. Old buildings, landmarks, and ruins can be brought back to life. Information about crime, traffic, and the environment can materialize before our very eyes. You can even turn your everyday life into a fantasy world filled with fantastical creatures.
Since our team deals with data, our survey focused specifically on how we can create data visualizations that are embedded into a physical context; so-called situated visualizations. To grapple with this issue, we first classified all of the existing systems and technologies — academic and commercial alike — using properties such as how the data is integrated into the surrounding world, the form factor of the device (hand-held, wrist-mounted, head-mounted, etc), how the user controls the system (touch, buttons, voice, etc), and how to navigate the data (by location, by gaze, by pointing, etc).
Analyzing this classification enabled us to identify typical patterns used in situated analytics. Head-mounted devices are mostly used for providing information about the site. Many situated visualizations take advantage of wayfinders or navigational aids to let users identify their locations while being in the place. Information is provided in user’s gaze either in the form of virtual objects or abstractions to increase the immersion of the scenery.
We took a step further by closely investigating common patterns found in situated visualizations. After categorizing visualizations into six application areas (e.g., engineering, shopping, tourism) we identified the main tasks that were commonly found in them. Then we carefully evaluated the advantages and disadvantages of situated visualizations contrasted with classical visualizations. We evaluated in three aspects: (1) how are situated visualizations affected by being in the field, (2) in what ways does current technology affect or limit the performance, and (3) how are situated visualization different from classic visualization systems in monitor displays?
While there is many well-documented research on situated visualizations, it is still true that many questions remain on the tradeoffs between efficacy of being “situated” in the locations compared with classical visualizations. For example, will situated visualization help long-term and short-term memory? Will situated visualization support the discover of data patterns in a serendipitous manner better than normal visualization systems?
To that end, as our future work we plan to provide empirical ground to many questions that are not fully confirmed. We developed a situated AR platform called Data Stalker. Data Stalker displays geo-tagged spatial data onto the real world, with an overview that summarizes the displayed dataset. While we have not yet evaluated our Data Stalker prototype, we have a list of predictions on the performance of three common visual analytics tasks: wayfinding, spatial layout, and data navigation. We will check back at a future symposium when the results are in!
