From Hype to (Mixed) Reality: A look at our first industrial MR application

Ionut Bogdan Pop
Feb 7, 2018 · 4 min read

When our IoT team first got their hands on the HoloLens a while back, they were immediately at it. They wanted to see what the tech was capable of and how it could work in industrial scenarios. Having worked with one manufacturing client in the past, they had a pretty good idea of what users needed. The team soon came up with an idea and began drafting the underlying architecture.

The business case was simple: make machine servicing cheaper and more convenient. With the help of MR, field technicians could easily service faulty equipment or prevent it from malfunctioning as a result of predictive maintenance. They could even order spare parts via the HoloLens.

Image for post
Image for post

From Hype to (Mixed) Reality

While 2017 was expected to be the year that industrial AR/MR was going to go mainstream, according to , “real-life use cases were extremely limited in 2017 with much less activity than anticipated”. We experienced this first hand when talking to partners and clients. End-users were not yet ready to embrace it as a viable alternative to their existing setup.

Not deterred, the team were sure that manufacturers would soon come to realize that Mixed Reality would prove to be an excellent tool for their Product Support and Maintenance units. A few months later, our industrial MR application was up and running. It allows field engineers to train and be assisted by the headset when performing maintenance tasks or be remotely guided by the manufacturer’s support team through the process via Skype.

How does it work?

Below is a short collection of videos captured with the HoloLens. It shows the user as he replaces a malfunctioning battery on an industrial robotic arm, and follows a few primary uses cases: the virtual dashboard, support files, a training tutorial, safety alerts, and a spare parts order scenario.

The Virtual Dashboard

Navigation is done through hand gestures

The maintenance dashboard queries the backend system to displays malfunction tickets and media assets, such as documents, video, and holographic experiences. The user can navigate through the dashboard and filters the tickets to show only the ones assigned to him. The system displays the details of the ticket: general info, device model, service logs, and the option to order spare parts.

Support Files

Pdf, Excel, PowerPoint, Word, video files are supported

With virtual documentation there is no need for the user to take off the headset. He can filter and search for assets attached to the ticket, and open each document in a separate window.

Training

Using a virtual replica of the industrial robot, the application is a quick and inexpensive way for engineers to undergo training.

A Fanuc LR Mate 200iD robotic arm robot was used

After selecting a battery replacement holographic experience, the surrounding space is mapped. The user can “place” the hologram where he wants, rotate the robot using hand gestures, or increase and decrease the size of the hologram. The user is then aided by a step-by-step tutorial to complete the training scenario.

Safety

When getting too close to the robotic arm the user is warned about an imminent collision. The red box starts flashing, indicating that the user is in the danger zone.

Spare Parts Ordering

For added convenience, field engineers no longer need to access their computer to make an order for a replacement part. They can use the virtual dashboard connected to a purchasing system experience to make

The application can also be integrated with an online virtual shop

Users can easily request spare parts by accessing the dashboard and clicking on the Place Order button to make the purchase. Once completed, the order is confirmed, and the status is set to Ordered on the tickets details page.

Skype

The Skype integration is not shown as Microsoft provides it as an out-of-the-box feature for the HoloLens. There’s no actual customization that you can do, and we think the existing solution works well. You can watch to see how it works.

Chatbot & AI

Image for post
Image for post

We’re currently working on setting up a chatbot feature and an AI engine to enhance the application.


With MR inching its way forward into the industrial landscape, we believe more use cases and even case studies will be popping up throughout the year.

Until then, stay tuned and feel free to share your feedback and ideas.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store