From Hype to (Mixed) Reality: A look at out first industrial MR application

When our IoT team first got their hands on the HoloLens a while back, they were immediately at it. They wanted to see what the tech was capable of and how it could work in industrial scenarios. Having worked with one manufacturing client in the past, they had a pretty good idea of what users needed. The team soon came up with an idea and began drafting the underlying architecture.

The business case was simple: make machine servicing cheaper and more convenient. With the help of MR, field technicians could easily service faulty equipment or prevent it from malfunctioning as a result of predictive maintenance. They could even order spare parts via the HoloLens.

From Hype to (Mixed) Reality

While 2017 was expected to be the year that industrial AR/MR was going to go mainstream, according to iot-analytics.com, “real-life use cases were extremely limited in 2017 with much less activity than anticipated”. We experienced this first hand when talking to partners and clients. End-users were not yet ready to embrace it as a viable alternative to their existing setup.

Not deterred, the team were sure that manufacturers would soon come to realize that Mixed Reality would prove to be an excellent tool for their Product Support and Maintenance units. A few months later, our industrial MR application was up and running. It allows field engineers to train and be assisted by the headset when performing maintenance tasks or be remotely guided by the manufacturer’s support team through the process via Skype.

How does it work?

Below is a short collection of videos captured with the HoloLens. It shows the user as he replaces a malfunctioning battery on an industrial robotic arm, and follows a few primary uses cases: the virtual dashboard, support files, a training tutorial, safety alerts, and a spare parts order scenario.

The Virtual Dashboard

Navigation is done through hand gestures

The maintenance dashboard queries the backend system to displays malfunction tickets and media assets, such as documents, video, and holographic experiences. The user can navigate through the dashboard and filters the tickets to show only the ones assigned to him. The system displays the details of the ticket: general info, device model, service logs, and the option to order spare parts.

Support Files

Pdf, Excel, PowerPoint, Word, video files are supported

With virtual documentation there is no need for the user to take off the headset. He can filter and search for assets attached to the ticket, and open each document in a separate window.

Training

Using a virtual replica of the industrial robot, the application is a quick and inexpensive way for engineers to undergo training.

A Fanuc LR Mate 200iD robotic arm robot was used

After selecting a battery replacement holographic experience, the surrounding space is mapped. The user can “place” the hologram where he wants, rotate the robot using hand gestures, or increase and decrease the size of the hologram. The user is then aided by a step-by-step tutorial to complete the training scenario.

Safety

When getting too close to the robotic arm the user is warned about an imminent collision. The red box starts flashing, indicating that the user is in the danger zone.

Spare Parts Ordering

For added convenience, field engineers no longer need to access their computer to make an order for a replacement part. They can use the virtual dashboard connected to a purchasing system experience to make

The application can also be integrated with an online virtual shop

Users can easily request spare parts by accessing the dashboard and clicking on the Place Order button to make the purchase. Once completed, the order is confirmed, and the status is set to Ordered on the tickets details page.

Skype

The Skype integration is not shown as Microsoft provides it as an out-of-the-box feature for the HoloLens. There’s no actual customization that you can do, and we think the existing solution works well. You can read more about it on Microsoft’s website or, watch this video to see how it works.

Chatbot & AI

We’re currently working on setting up a chatbot feature and an AI engine to enhance the application.


With MR inching its way forward into the industrial landscape, we believe more use cases and even case studies will be popping up throughout the year.

Until then, stay tuned and feel free to share your feedback and ideas.


For the full application demo video, go here.