Member preview

Mixed Reality Lending a Helping Hand

Go ahead and call me weird, but of all the Mixed Reality demos I did at this year’s Build, the one I found most fun had me repairing an electric panel.

To understand the deep sense of satisfaction I experienced when the pretend machinery setup at the Seattle Sheraton hummed to life, you first have to understand how utterly useless I am at any sort of DIY, so even though I’d been guided through the procedure step-by-step, I felt a real sense of accomplishment at the end, even if it was all make-believe.

I was trying out the Microsoft Remote Assist feature, a HoloLens app (although the functionality also works on PCs. Phones and tablets) designed to help Firstline workers collaborate remotely with hands-free video calling, image sharing, and a full complement of Mixed Reality annotations. That type of use case is not new in itself, as HoloLens works with Skype out of the box which of course enables you to video call someone have them “see through your eyes” and offer advice on what you should do. In fact, German manufacturing giant Thyssenkrupp has been using that functionality to improve the efficiency of their elevator repair crews for well over a year now.

What is different about this is that apart from being more polished and integrating more collaborative features in one place via Microsoft Teams it also plugs into the company’s strategic focus on AI. The idea is that as more enterprise customers use the service the system will be able to learn from the data and optimize the experience (for example, recommending the best expert to contact depending on the type of problem encountered and real-time availability, taking into account what time zones each user is in, etc.)

During the demo, the remote expert I had on video call not only told me what to do based on what I was seeing via the HoloLens, but was also able to look up and display a schematic map, pin it to the wall, and then draw holographic circles around the parts of the electric circuits that the drawing corresponded to. It really was a smooth experience, and I felt the confidence of all that unfamiliar expertise rushing to my head.

The other advantage for enterprise customers looking to use the service is that Microsoft promises to provide all this with advanced identity and security measures, which in settings where it might be a concern to share plans and reference materials relating to industrial processes.

A similar demo I tried in Seattle was from 3D app developer Taqtile, which generates step-by-step instructions supported by images, videos, text and holograms, all integrated into real-world physical actions. So here I had no input from a person, but (in spite of being that person who follows the LEGO instructions to the letter and ends up with a wonky masterpiece every time) but still managed to navigate the process so that at the end — lo and behold — the light switched on! It’s all about those simple pleasures.

Tech Trends Microsoft Build Mixed Reality HoloLens Remote Assist

Taqtile’s interface makes it easy for anybody to create such a list of instructions, which can then be used in a variety of scenarios from training to verification and compliance.

In the United States Air Force, for example, jet engine mechanics are not allowed to use operational airplanes for training. Some bases have training engines available, but these don’t necessarily match the aircrafts they would be called upon to repair, and many are restricted to learning from manuals and other supporting documentation, with their first hands-on experience being when they’re actually performing those tasks on planes that are in service.

Tech Trends Microsoft Build Mixed Reality HoloLens Remote Assist

This step-by-step training, however, allows those same students to practice alongside a physical training engine or a virtual 3D holographic one, which means they are physically performing and rehearsing those actions, building muscle memory and improving their confidence and knowledge retention.

Those materials can also be used by fully trained technicians to walk through a job and tick each step off the list. A mechanic in the field can select an appropriate job from the work order system, and will be provided with step-by-step instructions (overlaid on the physical equipment) on how to complete the job, which they can check off as they complete each task. The instructions include media such as video, audio, and images, and can link to external information as well as 3D models of the equipment’s “digital twin” and sub-parts.

Tech Trends Microsoft Build Mixed Reality HoloLens Remote Assist

For example, rather than having to rifle through a binder to find the appropriate Material Safety Data Sheet, the mechanic can access it directly from the relevant step in the tutorial. The inspector function allows a supervisor or compliance technician to review the mechanic’s work. The inspector can bring up the job completed by the technician, and can walk through step-by-step to ensure that everything was executed properly.

Mixed reality applications such as these can really help address a cross-industry challenge, a so-called plateau in productivity growth which results from a decline in number of skilled workers and increase in demand for maintenance and inspection expertise.

So while I enjoy a good VR rollercoaster or a spell of realistic zombie-blasting as much as the next gal, I do think these are the sort of practical, functional experiences that will sustain the growth of immersive technology while consumers wrap their heads around how it fits into their day-to-day lives. These applications give companies an immediate and often dramatic return on their investment, and — if my own experience is anything to go on — they can also be rather fun.

To find out how to leverage VR/AR/MR in your enterprise, Tech Trends offers bespoke Virtual Reality Consultancy support

Alice Bonasio is a VR Consultant and Tech Trends’ Editor in Chief. She also regularly writes for Fast Company, Ars Technica, Quartz, Wired and others. Connect with her on LinkedIn and follow @alicebonasio and @techtrends_tech on Twitter.

Originally published at Tech Trends.