An Augmented Me, Myself, and I

Would you visualize your personal data in Augmented Reality?

This is Jordan’s personal visualization. The shoes, buildings, and stacks of cash are Jordan’s virtual addition to the museum’s infographic map.

Consider the story of a traveler named Jordan. Jordan is in a museum, looking at a map of Napoleon’s march to Moscow. Jordan has also been traveling across eastern Europe, and their mobile device has been accumulating data on their activities — their location, how far they’ve traveled, how much money they spent. For context, Jordan wants to compare their journey to Napoleon’s. So they open a tool on their mobile device and import their personal data. The tool uses the camera, and other hardware on Jordan’s device, to capture and interpret their physical environment. Now Jordan is able to visually situate their data over the museum’s map and make comparisons. The result is the personal data visualization above.

The tool that Jordan used to create this visualization is the Mobile Augmented Reality Visualization Tool, or MARVisT. In recent years, mobile devices have brought augmented reality into the mainstream. The creators of MARVisT propose to leverage mobile augmented reality in order to make personal data visualization accessible to the masses. MARVisT was developed to explore this idea, understand the current limitations and challenges, and determine the usability of such a tool.

Jordan is a character that represents MARVisT’s target user: someone with access to data, but minimal experience creating data visualizations. Existing data visualization authoring tools target professionals — not Jordan. Jordan does not have the expertise, time or resources to use those tools. MARVisT attempts to address these target user needs.

Like the tools that are used to create them, data visualizations are not necessarily user-friendly. Typical visualizations can be too abstract for non-experts to easily understand. So, MARVisT uses glyph-based visualizations instead. The shoes, buildings, and money in Jordan’s visualization are glyphs. They are symbols that give the user a more direct cue to their significance. Situating glyphs in an augmented reality environment gives the impression that the data has physical presence. This physicality has been shown to make the data more understandable and more memorable for users.

The glyphs: building, money, shoes

To create the visualization, the user selects the glyphs that will be associated with each data type. For instance, Jordan selected a building for hotel cost, stacks of cash for total expenses, and shoes for path of travel. Then the user chooses how the glyphs would depict the data. Jordan decided that the height of the buildings would represent the amount of money spent on the hotel. Meanwhile, the height of the money represents all the money spent in the same location. And the positions of the shoes indicates the path of Jordan’s journey. However, since non-expert users cannot be expected to know the best way to visualize their data, MARVisT was designed to support users throughout the process. It offers design automation when possible, and nudges users toward the best design choices and away from problematic design choices. To maximize flexibility, though, MARVisT does not enforce standards or prohibit design choices. Instead, it allows users to ignore nudges and learn the drawbacks of their choices experientially.

Designing MARVisT for mobile devices had numerous advantages. Notably, it allowed MARVisT’s creators to build on existing research and tools for mobile augmented reality. But it also supported the accessibility of MARVisT in two important ways. First, users are already familar with the operating systems and interface conventions of their personal devices . This reduces the learning curve for the user. Second, the inherent mobility of mobile devices supports spontaneous use scenarios. Jordan is able to pull out their mobile device in a museum and create a visualization on the spot.

A MARVisT visualization of key use frequency on a keyboard.

But Jordan is only a theoretical user, so MARVisT’s creators tested their tool on real people to assess its usability. They asked their participants to perform three tasks:

1. Recreate Jordan’s map.
2. Recreate the keyboard visualization in the image above.
3. And create a visualization of their own.

Based on the participants’ performance, they confirmed that current limitations in augmented reality and known issues with 3D visualization still presents some challenges. Augmented Reality has room to improve when it comes to accurately tracking the user’s environment. And, as the hidden P, U and W keys in the keyboard above demonstrate, 3D visualization is notoriously challenged by view obstruction. Ideally, the user could access this hidden information by viewing the keyboard from another angle. However, due to additional technical limitations, the virtual 3D objects tend to distort as user moves around. Even so, because personal visualization is a low stakes arena, potential perception inaccuracies are less consequential.

Although they encountered some challenges with the tool, participants gave generally positive feedback. They affirmed that augmented reality is an effective platform for personal visualization and that this preliminary design had high usability. The creators of MARVisT have good reason to be hopeful that further exploration and research into this topic will lead to improvements and solutions to the current limitations.

Full Citation

This blog post is based on the following academic article:

• Zhutian Chen, Yijia Su, Yifang Wang, Qianwen Wang, Huamin Qu, Yingcai Wu. MARVisT: Authoring Glyph-based Visualization in Mobile Augmented Reality. IEEE VIS w2019.