Augmented Reality maintenance repair guiding for non-technical people to smart office Internet of Things appliances

Preface

This article was written by me, Simon Irengård Gullstrand, and Jonas Wahlfrid during our masters in Computer science.

ABSTRACT

This paper examines if the use of Augmented Reality technology could improve the maintenance procedure of an off-theshelf office appliance; a coffee machine, compared to the traditional paper-based manual format. This scenario has been investigated by building a prototype to be run on the Microsoft HoloLens to guide a non-technical personnel through a simple repair task of the coffee machine. The evaluation of the prototype was done with an observation by comparing completion times and errors made while letting participants go through a repair task with the prototype vs with a traditional paper-based manual.

INTRODUCTION

Due to usage and aging many appliances and devices suffer from deteriorating usability [15]. This could lead to the appliance being less reliable and additional operation costs due to the malfunction. Malfunctions are time-consuming, expensive and requires a lot of effort. This is why keeping appliances in good condition by regularly maintaining them is important [15]. Today many of these appliances is connected to the Internet to benefit from the advantages it brings. This movement is called the Internet of Things (IoT) [2]. Augmented Reality is becoming more and more popular both as a presentation and interaction means [14]. An augmentation of the real world with virtual objects are called Augmented Reality (AR), contrary to Virtual Reality which is about creating an entire virtual world. In an AR system the virtual information is added on top of the real world view [13]. By mixing a virtual view with real life the user can get additional visual contextual information. The three-dimensional virtual objects are anchored in the surrounding real-world. The user can walk around the three-dimensional virtual objects, viewing them from every angle and interact with them [16]. E.g., an application domain is to display a instruction manual of the object that is to be assembled or disassembled together with the real world view. This ensures that the user does not need to switch between a paper manual and the assembly work, and can therefore focus on both at the same time [7]. Fundamentally the user is able to interact with the virtual world in some way, an example of this could be that the user is able to navigate by moving and looking around in the virtual world. Also to manipulate virtual objects, by moving them or changing their state in some way. For interaction in the AR space and for the users experience to feel as natural as possible one of the input modalities for AR devices today is to recognize hand gestures [17] [10]. The tracking device recognizes preprogrammed gestures and invokes different functionalities based on the gesture and where the user is pointing the device. AR is a great choice regarding guidance in maintenance activities since it has the potential to bring clear instructions to a novice repairer in a distraction free and pedagogical way [14].

Purpose & Motivation

Using AR as an aid for maintenance is not a new concept. However, to a large extent, the AR applications utilized on the market today are mostly used in industrial purposes and for expert users due to that it is still very expensive. For example when maintaining an engine there are a lot of components and it is hard to remember all of them. Also the paper based manual makes it time consuming to look up all the different parts, since it is forced into a 2D space and a form of indexing to list all the information. The AR alternative on the other hand displays the necessary information for every component as the expert repairer looks at that component [1]. This technology will in a couple of years be developed enough and tangible for the consumer market [11]. It is even predicted that the traditional monitor could potentially be replaced with convenient Augmented Reality headgear [11]. Therefore we think there is a real possibility to utilize it for the purpose of quickly maintaining appliances in-house for companies. For example in the past when there was a malfunctioning appliance the users was forced to report the error to the appliance service provider. The service provider had to manually manage and schedule the task to a repairer. The expert repairer then was sent out to the location in question to maintain the appliance. This is time-consuming, expensive and requires a lot of effort. When there is a malfunctioning appliance there should not be the need to call the service provider immediately, there should be a way to guide a nontechnical office personnel to repair the machine if there is a simple error. This is what we are aiming for in our solution.

Solution

The solution we are proposing could potentially save resources and reduce downtime for the company with the appliance by reducing the need for the service providers expert repairer attendance. Where instead the nontechnical office personnel at the company performs simple repair tasks on the appliance themselves. The AR technology approach for maintaining systems has been used for ages with great success, but mostly targeting expert users [14]. For example in a case where technicians worked with maintaining elevators, trials showed that the repair task could be accomplished up to four times faster than before by using the Microsoft HoloLens Augmented Reality (AR) head gear [12]. In our report, we have focused instead only on the novice repairer, in this case an office personnel. The AR technology has the potential to guide a nontechnical novice user in a technical way quicker and more pedagogic with fewer mistakes made, compared to the traditional paper manual format [14]. We aim to create an AR maintenance guiding application for the HoloLens for the purpose of repairing an already existing off-the-shelf coffee machine. By using the HoloLens we can utilize the pros of a hands-free access to augmented reality maintenance guiding and we are able to evaluate if the AR technology is better than the traditional paper-based manual.

Scenario
An expected scenario where our solution is applicable is when an office personnel approaches the coffee machine at the office and it appears to be broken. The office personnel will get notified through the coffee machine user interface that there is an error with the appliance. The office personnel puts on the HoloLens head gear and uses the AR maintenance guidance application to get instructions on how to repair the machines error. The office personnel is guided, step by step throughout the whole repair task which in this case is to replace a screw in one of the containers. After all the steps has been followed the error is repaired and the coffee machine is working again. The office personnel enjoys his coffee.

Research scope

For the purpose of narrowing down the scope of this paper the coffee machine was unplugged and only one simulated mechanical error is be implemented into the maintenance guidance application. Also the application is only deployed on one AR capable headgear device, the HoloLens.

Research question

Based on the purpose and motivation the following research question were raised:

1. How can we prove that repairing a coffee machine is faster and done with less mistakes by a non technical person with the aid of Augmented Reality rather than a traditional paper-based manual?

Expected results

The evaluation have proven that the maintenance of the coffee machine is faster and the office personnel made fewer mistakes with our AR maintenance guidance application compared to a paper-based manual. Also by allowing the repairer a hands-free experience we hoped to increase the chance of this happening. But since users generally are used to following the traditional paper-based manual format the result could also show that paper-based manuals actually are better than our AR maintenance guidance application.

RELATED WORK

This section aims to highlight the relevant state-of-the-art to this paper. Starting off with some differences in how the AR technology can be distributed to devices, then moving into several examples of maintenance aided by AR and how they are similar and different from our study.

Related to the field of AR with the goal of maintenance there has recently been many studies as maintenance is a very im- portant subject as discussed previously. The two mostly used ways of distributing AR applications today are to hand held and head mounted devices. A newly developed head mounted device is the Microsoft HoloLens, which is a battery powered wireless augmented reality Windows 10 computer, that can show the user three-dimensional virtual objects (more on the HoloLens later) [16]. Illustrated in figure 1 is a repairer wear- ing the HoloLens, and in front of him a representation of parts etc presented to him in form of AR objects. Also by pointing the HoloLens onto different real life objects, the software can recognize what it is and present useful information to the re- pairer for that particular object. All of this can also be done as mentioned before with a hand held device like for exam- ple almost any smart phone or tablet with the right software installed, but instead of just glancing you need to hold the device in front of you and pointing it onto the desired object to show AR based information of that object [1]. We think that the allowance of a hands free way of working with the machine while repairing it would benefit the repairer greatly. Instead of being forced to look away from what the repairer is doing and read from the manual and then back to the task, confuses him. To be able to instead mixing the instructions with the view of the machine would be wield much faster and more successful repair scenarios.

The application areas that has been mainly focused by wear- able and AR technologies in maintenance are to provide a in- formation, guidance and instructions e.g., manuals, pictures and being able to immersively collaborate with other people wirelessly (e.g., novice vs. expert) [9]. Studies suggests that AR technologies are feasible for providing maintenance in- structions because they are often faster to use, errors occur less frequently, and the participants endorse the use of the technology [1]. In one study the conclusion was that a novice user would benefit from the AR guidance systems when using them for simple instructions and guidance. But some main- tenance tasks are safety-critical, and it might not be feasible to let novices working by themselves with only the guidance of a technical systems to rely on [1]. In this paper we focus entirely on the novice user, and to see if we can get the same results with the focus of comparing paper-based manual to a AR based manual.

The company thyssenkrupp utilizes the technology of Mi- crosoft HoloLens to give 24 000 elevator service technicians access to augmented reality maintenance guiding and training [12]. The service technicians can visualize the elevator and its components, technical information, service logs, events and alarms ahead of a job, which makes it possible to train personnel, plan work and know which spare parts to bring in advance to going to the actual location. The paper [12] is rel- evant to our work because it shows how AR could be used to maintain a machine, and also how the GUI can look like in order to help the repairer carry out his task.

When working with the elevator the service technicians can get hands-free access to augmented reality maintenance guid- ing through the Hololens headgear. It presents to the service technicians a 3D representation of parts, task orders, safety alerts, service logs, events and alarms, visualized in figure 1. This new way of hands-free interaction can be compared to the traditional approach where the service technicians had to bring a laptop to gain access to the same information.

Figure 1. Augmented reality maintenance guide [12]

Trials in [12] have shown that the technicians work on site with the elevator can be done up to four times faster than be- fore by using the HoloLens technology. The HoloLens implementation gets events and alarms from elevators that are connected to thyssenkrupps MAX system, that is built with Microsoft’s IoT management system Azure. By utilizing Microsoft’s Azure IoT Suite [6], elevator data like for example motor temperature, shaft alignment, cab speed and door functioning, are obtained and reported back to the management system for analysis, monitoring and predictions for future maintenance before they occur by flagging the need to replace components and systems before the end of their lifecycle etc [12] [6]. Compared to our work thyssenkrupp has done basically the same thing as we are trying to do, but again with the focus on expert repairers. Their system also obviously involves a lot more components and subsystems, to empower the repairer in his task of repairing the machine. Most of the components connected in this example would in our scenario of aiding a novice repairer repair a coffee machine be unnecessary and probably just confusing for the novice. Our system at the same time as keeping in mind that it is a early proto- type has a very simple UI. But it could also be an advantage in this case since the novice repairer can avoid all the ”clutter”, information overload and focus on the task at hand.

In another recent study a experimental user study was per- formed on a office appliance to evaluate applications for AR [14]. It was done with a hand held device. In this study, the participants were given a task to replace a hard disk drive (HDD) in a laptop, and the completion times and errors be- tween using AR and paper-based manuals were compared. In the first cycle of tests the task of switching an HDD in a lap- top was carried out by 10 people, who were used to working with computers. The time it took to complete the task was about twice as long with the AR-based solution compared to a paper manual. However, there occurred only three mistakes with AR compared to ten with the paper manual. The same test was repeated with twelve participants who were not used to working with computers. In this group, it took an average of ten minutes with AR compared to eleven minutes using a paper manual. The standard deviation with AR was one and a half minute and two and a half minutes with a paper manual. By using AR, the number of mistakes decreased to one third compared to the paper manual. The study concluded that the lower the skill level, the greater the benefits of using AR solutions [14]. This conclusion was interesting to us. Since it basically conclude the same as our results showed, that the AR application is better compared to the paper-based manual. It is valuable to point out though that this study was different, where the participants used a hand held device instead of a head mounted one. Since the hand held device is harder to use than the head mounted device as previously mentioned, we do think that this will only empower the results to favor of the AR application

Yet another study, [4], which resulted the same way, that the AR application was the best as an instruction aid by measur- ing fastest assembly time and the amount of mistakes made by the participants. In this study the task was to assem- ble a motherboard using four types of instruction medias: computer-aided, paper-based manual, opaque augmented re- ality display, and see-through augmented reality display [4]. The difference in our study is that we focus on a IoT appli- ance and the fact that a non technical person should be able to follow the instructions and succeed with the maintenance task. Also in this study the AR applications was on a hand held device.

PROTOTYPE DEVELOPMENT

Prerequisites
In this section the prerequisites for realizing the prototype is listed.

Equipment
The main two devices used are the Microsoft HoloLens and the selected smart office appliance; the coffee machine illus- trated in table 1. The technical manual which was provided by Selecta was used to choose a suitable repair task. The task was carefully chosen based on the amount of steps necessary, the difficulty level and complexity of the process in the task. Then in turn the instructions for that task were used as a basis for developing the repair steps in the prototype.

Development tools

The chosen development tools are Unity3D to build the visual parts of the prototype, Visual studio to code the necessary functionality for the prototype and Vuforia to handle the an- choring of the 3D models to the real world in the augmented reality layer illustrated in table 2. Vuforia recognizes unique patterns in the marker so that it can know where to anchor the AR assets. The HoloLens emulator [8] was used to debug the application without uploading the project to the HoloLens on each iteration, this saved us a lot of time and resources during the development process.

Space
Since we are supplied with universal keys to the available coffee machines at Malmö University from Selecta. We need to lend the available coffee machines when they are not in use, which is located in the break rooms and we have been granted access temporarily to perform our tests.

Requirements of the Prototype
Use Case: Augmented reality maintenance guiding
Primary Actor: Non-technical repairer, he enjoys the coffee provided by the coffee machine. The non-technical repairer has limited experience in maintaining a coffee machine and therefore needs extra clear instructions. The instructions to him are given via an AR based application running on the HoloLens.

Goal in Context: The non-technical repairer’s goal is to re- pair the coffee machine using the Augmented Reality main- tenance guiding application so that he can get his coffee.

Preconditions: The non-technical repairer has been informed of an error in the coffee machine.

Postconditions: -

Basic flow:

1. The AR maintenance guiding application is started once the repairer puts the HoloLens on.
2. The AR maintenance guiding application identifies a Vu- foria marker which allows rendering of the maintenance steps in form of AR assets.
3. The AR maintenance guiding application guides the non- technical repairer to repair the coffee machine step by step.
4. The repairer succeeds with the repair and gets his coffee.

Alternate flow: -

The repair task
The chosen repair task aims to replace the screw in the pow- der container. The repair task is taken from the Selecta tech- nical manual and involves in total eight steps, four steps from starting off to the unscrewing the screw part and an additional four steps to put everything back in place. Like illustrated in figure 2 the steps are: 1. Pull down the white cover while also pulling down the green lever. 2. Pull out the container. 3. Turn and remove the lock ring. 4. Remove the screw by turning. 5. Insert the new screw by turning. 6. Turn and lock the ring. 7. Insert container. 8. Pull up the white cover.

Design
This section aims to describe the process for producing the AR maintenance guidance application. The development of the application was done in multiple iterations, where in the first iteration a low-fidelity visual prototype modeling the UI with plexiglass was developed and in the second iteration, the first full-scale AR maintenance guidance application version was developed by transferring the UI components in the low- fidelity prototype. Lastly in the third iteration the final ver- sion of the prototype were developed by adding the provided feedback from the previous iterations, to improve on the application.

The low-fidelity prototype
In the first iteration a prototype was developed utilizing plex- iglass for the purpose of modeling the user interface. This iteration aims to roughly sketch up the UI of the AR appli- cation at the same time as we get fast feedback about what ideas are good and bad before implementing them into the full-scale application. This was done by translating (porting) each step from the repair task to replace the screw in the pow- der container from the Selecta technical manual. And then by drawing the instruction steps on the plexiglass and document each step with photos, see an example of a step in illustration 2. As mentioned the maintenance instructions were ported from the paper-based technical manual provided by Selecta. In the paper-based manual there are images as well as text to guide the repairer, we combined both and reworked them to fit as AR assets. Since this task of porting from paper to AR was done by us, the authors without any guidelines, we identify threat to internal validity. We are biased and we do not have a lot of experience in maintenance related areas or in writing manuals. This could affect the end result of this paper, and therefore when developing this project further addressing this aspect is highly prioritized.

The AR maintenance guidance application
In the second and third iterations we developed the full-scale application and the AR assets, it was mainly done in Unity 3D. For code see [5] and for a video demo see [3]. The logical code components are written in Visual Studio Code in C#, since we are targeting a Microsoft product; the HoloLens is shown in table 2.

The second iteration aimed to build the full-scale applica- tion based on the low-fidelity prototype to see how it looked like in real AR. One thing that we learned instantly was that the thickness of the arrows really matters, in AR they faded into the background without proper filling. Other things we learned from letting a fellow researcher try the application and tell us what they thought about it. All the gathered feed- back were then transfered into the third iteration where we improved on all the flaws in the full-scale application.

How the application works and its components
The application starts the repair process of the coffee machine once the camera of the HoloLens recognizes a marker in the frame. It will then align the AR assets in front of the user in the predefined positions illustrated in figure 3. In order to traverse through the maintenance steps a voice command is provided to the user.

EVALUATION
The evaluation of the tests consisted of observations of in total fourteen participants carrying out the task of repairing a coffee machine. Seven participants using our AR maintenance guidance application and seven participants using the corresponding manual in paper-based format. Participants were chosen based on two conditions, firstly that their occupation is some type of office worker and secondly that they had no technical background in maintenance. The gender of the participants was completely random, listed in the result table 3. Also the age was completely random and varied a lot in be- tween 22–50 years old. The objective of the observations was firstly to measure the repair success rate by looking at the amount of errors made by the participants (wrong action performed). Secondly to measure the completion time of the repairing whole process. This way the results are aimed to answer the research question directly. We also conducted a short follow up questionnaire of the purpose of getting a deepened understanding of what the participants opinions were about their experience of the tests. After each test we also con- ducted a qualitative study in form of a questionnaire for the purpose of getting a deepened understanding of what the participants opinions were about their experience of the tests.

Figure 3. Step 1 of the AR maintenance guidance application

Study design
The tests as mentioned above was carried out by us the authors, observing participants while using the AR based manual versus the paper based manuals to repair a coffee machine. The repairing task was conducted during daytime on a stationary, in use, coffee machine at Malmö University. The task the participants performed as mentioned in section was to replace a screw in one of the coffee machines powder containers illustrated in the center to the left of figure 2. The participants were also given a brief introduction on how to use the HoloLens and our application then a short description of the task they were about to perform. The description in the case was: Put on the HoloLens, start the AR maintenance guidance application, follow the displayed steps in order to perform the repair task using AR guidance. Then lastly a short description of the task. We also conducted a short follow up questionnaire, the questions are attached as an appendix.

Results & discussion
The results from the observations are presented in table 3, with four columns and fourteen rows (one for each partici- pant). The columns are Method, Time, Errors and Error oc- currences. The Method column presents what method was used, the Time column displays in seconds the time it took for the participant to finish all the steps in the repair task. The error column shows the amount of errors the participants made and the last column, Error occurrence shows in what step the participants failed and did an error before succeeding with the step and moved to the next.

The results from the questionnaires summarized are that the AR experience with the HoloLens was impressive and they had never experienced anything like this before. The more valuable feedback involved suggestions on how to improve the UI and interaction in the AR application, for instance in- stead of hands showing where to interact with the machine, marking the object in question in a color would be easier. Also by using 3D models (as Vuforia markers) of the ob- ject that the user is going to interact with, we eliminate the need for picture markers and could potentially add additional steps for example if the user makes a wrong step he could be guided back. Above all, the majorities of the participants re- ported problems with the voice command to traverse through the steps, this could be solved by adding a AR asset button instead.

CONCLUSION
In this section, we aim to answer the research question with the help of the results from the evaluation process.

Research question 1: How can we prove that repairing a cof- fee machine is faster and done with less mistakes by a non technical person with the aid of Augmented Reality rather than a traditional paper-based manual?

Our prototype proved to be a application that non-technical office people found practical and useful to use to repair a mal- functioning coffee machine and also the participants made less error and completed the repair task faster with the AR maintenance guidance application, the results are listed in ta- ble 3. Therefor AR is better than the paper-based manual and the research question is answered. The results were how- ever quite close and that could be because people are used to following the traditional paper-based manual format as we expected. Also with taking the questionnaire results into con- sideration there is no doubt that we need to further develop the interaction and UI of the AR application for it to be use- ful in a real life scenario.

Future work
There is future work to be done in this project, as mentioned in section one aspect to address is to evaluate and possibly rework the translation of the paper-based manuals to AR as- sets to improve on the instructions in the AR application. As illustrated in table 3 the participants of the tests with the tra- ditional paper-based manual made the most error on step A and C.1 which might indicate that there could be some room for improvement in that manual. Also the design of the AR assets as well as user experience design of the application, was developed by us the authors with limited experienced in this area. We think and also based on the results from the questionnaires, that this is one of the most important parts to develop further.

A potential compliment to the Vuforia marker is the HoloLens sensor data from mapping the physical environ- ment, which we decided to turn off during our development. This decision was made because we want the positioning of the AR assets to be relative to the marker. and we were not sure if had access to the same room with a coffee machine for each test. In our maintenance case one of the steps include removing a part of the machine, and repairing it on a table in potentially in a new room. A new room is not mapped by the HoloLens, and the anchoring of the 3D objects would fail. Vuforia also has support for an actual 3D object of the coffee machine instead of using a simple (image of a unique pattern) marker. This approach was not feasible for us because we do not have access to any 3D models of the parts in the coffee machine.

EXTERNAL SUPERVISOR
Our external supervisor is Romina Spalazzese.

ACKNOWLEDGEMENTS
We would like to take the opportunity to thank our supervi- sors Shahram Jalaliniya and Thomas Pederson for all the ad- vices and guidance throughout the process of writing this pa- per. Also our external supervisor Romina Spalazzese for her advice and guidance as well. And Agneta Larsson at Selecta for providing us with a universal key to the coffee machines at Malmo ̈ University and maintenance technical manuals to conduct our research on.

REFERENCES

1. Aromaa, S., Aaltonen, I., Kaasinen, E., Elo, J., and Parkkinen, I. Use of wearable and augmented reality technologies in industrial maintenance work. In Proceedings of the 20th International Academic Mindtrek Conference, ACM (2016), 235–242.
2. Atzori, L., Iera, A., and Morabito, G. The internet of things: A survey. Computer networks 54, 15 (2010), 2787–2805.
3. Augmented Reality HoloLens maintenance guidance application. https://www.youtube.com/ watch?v=qQ59Wfuj6wI&t. Online; accessed 25 April 2017.
4. Baird, K. M., and Barfield, W. Evaluating the effectiveness of augmented reality displays for a manual assembly task. Virtual Reality 4, 4 (1999), 250–259.
5. Code Augmented Reality HoloLens maintenance guidance application.
   https://github.com/warriv93/Augmented-Reality-maintenance-guidance-HoloLens-application Online; accessed 25 April 2017.
6. Gubbi, J., Buyya, R., Marusic, S., and Palaniswami, M. Internet of things (iot): A vision, architectural elements, and future directions. Future generation computer systems 29, 7 (2013), 1645–1660.
7. Henrysson, A., and Ollila, M. Umar: Ubiquitous mobile augmented reality. In Proceedings of the 3rd international conference on Mobile and ubiquitous multimedia, ACM (2004), 41–45.
8. Hololens emulator website. https://developer.microsoft.com/ sv-se/windows/holographic/using_the_ hololens_emulator. Online; accessed 28 March 2017.
9. Lukowicz, P., Timm-Giel, A., Lawo, M., and Herzog, O. Wearit@ work: Toward real-world industrial wearable computing. IEEE Pervasive Computing 6, 4 (2007).
10. Malik, S., McDonald, C., and Roth, G. Hand tracking for interactive pattern-based augmented reality. In Proceedings of the 1st International Symposium on Mixed and Augmented Reality, IEEE Computer Society (2002), 117.
11. Oculus Connect 3 Opening Keynote: Michael Abrash. https://www.youtube.com/watch?v=AtyE5qOB4gw. Online; accessed 18 April 2017.
12. Microsoft HoloLens enables thyssenkrupp to transform the global elevator industry. http://tinyurl.com/ltlbvqk. Online; accessed 7 February 2017.
13. Milgram P and Kishino, F. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321–1329.
14. Sanna, A., Manuri, F., Lamberti, F., Paravati, G., and Pezzolla, P. Using handheld devices to sup port augmented reality-based maintenance and assembly tasks. In Consumer Electronics (ICCE), 2015 IEEE International Conference on, IEEE (2015), 178–179.
15. Segawa, Y., Ohnishi, M., and Ibaraki, T. Optimal minimal-repair and replacement problem with age dependent cost structure. Computers & Mathematics with Applications 24, 1–2 (1992), 91–101.
16. Taylor, A. G. Develop microsoft hololens apps now.
17. Zhou, F., Duh, H. B.-L., and Billinghurst, M. Trends in augmented reality tracking, interaction and display: A review of ten years of ismar. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, IEEE Computer Society (2008), 193–202.