Recommendations on hardware and interaction for various types of AR and VR experiences

In this article we will look at various types of AR and VR experiences, both current and prospective, and offer some thoughts regarding the type of hardware functionality and the types of interaction behaviors we feel are requisite in order to fulfill the purpose of the experience.

360 videos

Videos are currently better served in VR headsets than AR headsets, due to continued AR issues of contrast and translucency, as well as field of view (FOV). We recommend precise rotational 6 degrees of freedom tracking with a response time of 12ms or less.

In 360 videos users will simply look around themselves. There is no translational movement (although folks are working on fixing that, to the chagrin of data networks and storage everywhere), so the interactions are passive beyond the rotational movement. Full 360 rotational tracking is, however, recommended.

Interactive 360 videos

Here we add basic interactivity to the videos. Generally having a look based selection method is good enough for these experiences, with a “Click” input method, like on Google’s cardboard or Samsung’s GearVR. We retain the need for 360 head rotational tracking.

360 videos offer the potential for user responsive behaviors. This could be something simple like letting the user select from branching options and the video continuing to a new clip appropriately, or it could be tracking what the user is looking at and branching that way. Giving the user ownership over the storytelling is very powerful, particularly if you combine the two such that the options the user is presented with are dependent on what he focused on during the previous segment.

  • Full rotational tracking
  • Momentary press button/”click” input
  • Reticle to direct input “click” to specific part of scene

Games

Games in VR and AR are likely to be very different in play mechanics, but the hardware requirements are going to be similar, due partially to the intended audience. Younger individuals will want a more intuitive and direct input method, and likely not jump through user interaction hoops.

If the game is fun, if they can behave in a natural, organic way and the game responds accordingly, they’ll be happy. This means precise tracking (no floaty tracking in AR HMDs) and one motion tracked controller. The current gold standard for this is the Vive due to the precision of the SteamVR lighthouse technology.

A VR game we developed for a GameJam

On the AR side the HoloLens is the current market leader for tracking, which is critical in fast moving games (as compared to slow-moving productivity applications. For most games room-scale tracking is recommended.

Recommended

  • 6 dof AR or VR HMD
  • Room-scale tracking
  • Two tracked motion controllers with grip input

Minimum

  • 6 dof AR or VR HMD
  • Hand-held clicker, preferably motion tracked

Games run the spectrum of interaction types, which is why we divide the hardware requirements into recommended and minimum. The biggest thing with games is that the user is able to interact with the world and the world is reactive to the user.

This is really more critical than whether the user is using their hands or eyes or head or mouse to interact. In games generally, but particularly in spatial games, it feels weird if the world is “dead.” This is because making it life size all around us makes our brains think things will behave like they do in real life.

  • World must react to actions of the user in a predictable manner
  • User must be able to influence progression/storytelling

Social Apps

Social VR relies on the sense of presence of other humans. This is conveyed via voice, avatar faces, and movements/gestures. Because we are so highly attuned to the nuances of human body language and facial expression it’s extremely important to nail both a good artistic style and the ability to emote via your chosen avatar. If we consider the normal behaviors of users currently engaging with social networks we see that they like to asynchronously share ideas and thoughts, images and videos, and experiences they’ve had or created.

There is also a high prevalence of trying to appear happier and more successful than an individual really is, or at least maximize the attractiveness, and thus attention, received. This is permitted by virtue of the asynchronous nature of social applications in their current state. If you are simply in front of people it is much more difficult to maintain the illusion of happiness, of polished elegance, and they can see that it’s not your Ferrari.

Facebook Spaces

Currently these things are limited or nonexistent in Social VR. We believe being able to visit another user’s “space” or page in VR, see the images and content they’ve shared, maybe they’ve recorded a “hologram” of themselves welcoming you to their room and giving you a tour. Maybe they’re in a game and recorded a hologram telling you to join them that you see when you meet in the place they said they’d be.

On the AR side one should be able to share a photo, geotagged to a location and anchored to a wall. A use case might be taking a photo with your phones while at a pizza place, throwing it on the wall, and then each time you went to that pizza place you would see your own decorations.

Hardware has to support the given modality of emoting that the developer chooses.

  • 6dof HMD with positional tracking and some type of input
  • Voice capabilities

The more interaction options the merrier in this category. Your experience will improve with finger tracking, eye tracking, better avatars, and even real time spatial projection of your own self, if so desired.

  • Ability to emote
  • Ability to show/hide a line-of-sight laser indicating what you’re looking at
  • Ability to interact with the other users and the world
  • Ability to share experiences synchronously or asynchronously

Work Training/Guidance

The current standard in this space is HoloLens, although ODG and Google Glass 2, among a number of others, are working in this space. Typically AR is the chosen modality here due to the ability to work on the actual machines and hardware rather than a simulation. There will be certain benefits to VR however, in that you can learn dangerous behaviors without being at risk (fire fighting and various jobs on oil rigs come to mind). We can differentiate as well instructor-led training and simulated pre-programmed training. Let’s look at them in pieces.

AR Synchronous utilizes a camera on the student’s headset to deliver a video feed to the instructor, which is able to draw remotely onto the student’s vision, pointing out the correct part, wire, tool, or sharing a video animation of an activity to copy.

AR Asynchronous utilizes recordings of instruction, perhaps given synchronously to a different student, and then played back ad infinitum. Because of its asynchronous nature one would want to give the student control over the playback timeline (with a live instructor one could simply ask a question before proceeding).

Microsoft HoloLens

Users of these training programs will generally not be AR/VR aficionados, so keeping it very clear and well explained is important.

AR: Wireless HMD with inside-out tracking and some form of input
VR: Room scale HMD with motion controllers or hand tracking

In the AR model the user is interacting with the real tool and machinery, learning processes with the guidance of an overlaid instructor’s notation and voice/image. Only basic input is really required, so even an on-device tap or swipe function will suffice.

For VR you’re simulating the actual devices themselves, so it’s more important to have accurate and realistic interactions for the user to engage in using motion controllers or hand tracking. Providing a sense of tactile feedback using haptics is important here.

Educational Applications

Education covers, by necessity, a very wide gamut of requirements.

We have kids who are focused on creativity and basic skills. For those we’re trying to elicit an interest in the world around them, acquire the simple tools they’ll need to compete in the higher grade levels, and inspire a love of learning. This could also afford children the ability to learn about and experience concepts and situations that are too adventurous to subject them to normally. A teacher-guide or student partner can monitor.

We have High Schoolers who are focused on rote memorization and analytical thinking. Here we start to need a higher fidelity of simulation, since they don’t require games for memorization at this point.

For college students we are in a situation very similar to a workplace’s needs, where students are focused on collaborative studying and projects, assimilating large amounts of new information, and applying their existing skills to these new challenges. So more sophisticated and realistic simulations are needed than at the High School level, along with tools that facilitate time management and collaboration.

Grade School: Basic AR/VR HMD with simple input button or swipes, ability for teacher to lead/load/guide experiences
High School: More advanced but still basic HMD with more complex input hardware
College: Full room scale with motion controllers. Social aspects very important given social nature of college work

For interactions we can likewise divide by level. For Grade School: Mostly look and click on things, move around and solve basic puzzles. In High School: Scientific and other educational simulations, puzzle solving games. In College: Collaborative work environments, team study group activities.

Collaborative Meetings

This category is essentially meetings, workshops, offices, and team building for large corporations or small distributed teams. There are a few basic needs here, starting with the basic Social VR needs of an avatar with VOIP. The needs for emotional conveyance are lower, however, particularly for the larger corporate entities. What is needed, however, is basic productivity tools, like a white board, kanban board, a way to present to a group and manage who is heard, a way to share files with each other and view them in the experience.

  • 6dof room scale VR and/or AR to support in-office and out of office folks
VW’s VR Collaboration Tool

The following are some interaction patterns you might consider:

  • Display gaze target
  • Show hands for gesturing
  • Write on whiteboard and other things
  • Share files
  • Present powerpoint-esque slides
  • Manage attendees and who’s allowed to speak

Personal Computing

This is one of the most challenging areas for AR and VR, and the one we have given the most attention to at Holographic Interfaces. For personal computing to be truly advanced by spatial interfaces we need to understand what spatial interfaces can and cannot afford the user in excess of a traditional interface.

Meta 2 AR Glasses

One thing that an AR system can do is offload all the menus and panels into the environment around your monitor, saving your high resolution/contrast monitor with its perfect color accuracy for the primary content. This is critical for more complex applications, and handy for simpler ones. Ideally a mouse cursor could flow right off the physical screen onto the surrounding content, projected outward from the user’s view origin.

Offloading Menus to save screen space. Design by Holographic Interfaces for internal project

Another area is for better manipulation of large multi-dimensional data sets. That may sound intimidating, but if you’re even had more than one related sheet in a Microsoft Excel document you’ve used such a data set. Check out our last article for a concept of how that might work. This can be expanded to other related areas, like finding the right file (file browsing), finding the right bit of information on the internet, seeing an overview of a company’s structure and work patterns, or following an email or internet forum conversation. All of these rely on the organization of disparate bits of data that are marked with various tags, each of which is essentially a dimension.

Spatial Manipulation of Complex Data Sets

An area where spatial interfaces don’t help is anything with lots of small text, like ebooks or text-heavy web pages. It also generally has difficulty where there’s a requirement for high levels of precision, due to both the fidgety nature of 3-axis inputs as well as the relatively low resolutions of the HMDs themselves.

Hardware

  • 3dof HMD preferably with eye tracking and hand tracking
  • User will likely use mouse+keyboard for most tasks

Interaction Behaviors

  • Everything you can do on a normal computer, plus spatial-specific
  • Organize information in a way that’s logical and visually clear

Collaborative Computing

This category is the spatial version of Google docs, where users can engage in synchronous and asynchronous productivity activities. The tools can be similar to Collaborative Meetings, except that they require the addition of the ability to edit more types of files and they do not require the presentation and meeting controls. In a sense these tools would also be similar as well to the Personal Computing tools, only network-based rather than locally run.

Hardware
6dof HMD with room scale and motion controllers to allow for seated and standing work and collaboration

Interaction Behaviors
This would include both the Collaborative Meeting interactions and the Personal Computing interactions, and is thus the most complex of all our scenarios.

This has been a cursory overview to offer some initial guidance of what you should be targeting for your chosen application. Doing a High School level educational experience? Make sure you don’t make it too simple (Google Cardboard), but don’t overwhelm them. Always remember that visual clarity and simplicity are imperative to good design, particularly in spatial experiences where visual complexity is by necessity multiplied. These are not hard and fast rules by any means. We’d love to hear your feedback about this or our other articles. If you would like us to discuss a certain topic in a future article, email us at contact@holographicinterfaces.com

)

Holographic Interfaces

Written by

Holographic Interfaces is a boutique research, design & development shop specializing in Augmented and Mixed Reality experiences.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade