Show Me How It’s Done: Using Augmented Reality (AR) To Tap the Expert’s Brain

Co-Authors: Victoria Claypoole, Kay Stanney

“We always know more than we can say, and we will always say more than we can write down” — Michael Polanyi

As a brand-new private pilot, I remember going up in a small plane and practicing instrument approaches (the ones you fly when the weather is bad) with my father — an airline pilot and former Naval Flight Officer. I was building flight time for my instrument training, and my father tagged along as a safety pilot to critique my flying. One day we were practicing Instrument Landing System (ILS) approaches…I was wearing foggles, which are instrument-training glasses that cover your field of view and force you to only look at your cockpit instruments. I did my best to fly the approaches “by the book,” and fortunately, I made it to the runway with only a few deviations from the flight path.

We took off again, and I handed the foggles and controls to my old man. He flew the same approach perfectly, with barely any deviations. Admittedly, I was a bit salty. From my perspective in the cockpit, he did the exact same thing I did. But since he has way more experience, he was implicitly picking up on different cues that guided his precise control inputs. During the debrief, I tried to get him to explain his exact mental process for flying the approach. He smirked and referenced an old joke “Son…. Do you know how you get to Carnegie Hall?… Practice!”

But it has to be more than just muscle memory from repetitive practice, right? What exactly was my father paying attention to or doing that allowed him to execute a perfect instrument approach? Why couldn’t my father simply tell me what was going through his mind as he landed?

Tacit Knowledge

We’ve all had experiences with experts that possess an enormous wealth of knowledge within their field yet struggle with discussing and transferring that knowledge to their colleagues or students. In this context, we are mostly referring to tacit knowledge. Tacit knowledge results from our learning experiences that we accumulate throughout our lives and involves qualities such as our wisdom, insight, intuition, and perspectives (Stephens, 2002). As Michael Polanyi, the originator of the tacit knowledge construct, stated in The Tacit Dimension (1966, p.4), “we can know more than we can tell.” Unlike explicit knowledge, which is fact-based and can be easily documented or verbalized, tacit knowledge is difficult to capture, organize, and distribute (some experts may not even know they have it) because it is acquired through direct experience and is applied subconsciously (McAdam et al., 2007).

So, effective transfer of tacit knowledge generally requires extensive personal contact with an expert who guides (often through intuition and not words) you through practice in a particular context (Schmidt & Hunter, 1993), like my father guiding me through my instrument approach via subtle body contortions meant to tweak my deviations.

Types of Knowledge

Tacit knowledge is comprised of all the taken-for-granted know-how that experts gain over years of experience (Fuchs, 2001). In every organization, no matter the size, you will have a pool of tacit knowledge possessed by the most senior experts. They are the living repositories of the long-earned “know-how” and “knowing in action” that is so highly valued within an organization, especially by younger workers. Such tacit knowledge can enrich the memory of an organization, organize the best practices, and capture the multiple perspectives of its experts. All of these effects can lead an organization to be more effective, efficient, and productive. Tacit knowledge is also highly valued in successful organizations because it is vital to the decision-making process in times of stress or uncertainty. But what happens when your experts retire or leave, and they take all of their tacit knowledge with them? As we discussed earlier, it is extremely difficult to document all of their “tricks of the trade,” and we can’t physically transfer an expert’s grey matter to a novice’s brain (although that would be both cool and ethically questionable)…but what if there were techniques or tools that you could use to capture, preserve, and transfer tacit knowledge to a younger, less-experienced workforce? This is a challenging problem that many organizations now face (Flood, Verdad, & Frederick, 2020).

So, How Do You in Fact Pick an Expert’s Brain…?

Some researchers have attempted to tackle this complex problem by proposing models that inform knowledge management practices in an organization. I think it’s valuable to briefly look at Nonaka and Takeuchi’s (1995; see also Farnese et al., 2019) description of how tacit knowledge is created and shared. Their Socialization, Externalization, Combination, Internalization (SECI) model, which considers knowledge creation as a dynamic process involving continuous and dynamic dialog between tacit and explicit knowledge generation, consists of the following modes of knowledge conversion:

1. Socialization (tacit to tacit)- Occurs through imitation, observation, and practice. Knowledge is transferred through face-to-face communication or shared worked experiences in a physical or virtual work-space.
2. Externalization (tacit to explicit)- Typically starts with a dialogue or collective reflection that is supported by an appropriate metaphor or analogy, which allows experts to articulate tacit knowledge in a manner from which explicit concepts can be gleaned.
3. Combination (explicit to explicit)- During combination, individuals exchange and combine explicit knowledge through different means such as telephone conversations, groupware, online databases, intranet, virtual communities, etc. In this process, explicit knowledge is combined or integrated and synthesized into more complex and systematic explicit knowledge.
4. Internalization (explicit to tacit)- The conversion between explicit and tacit knowledge is related to “learning by doing.” In internalization, explicit knowledge is absorbed and used to enrich one’s base of tacit knowledge by connecting formal knowledge to personal experiences.

While this model has been used extensively in academia and industry, it has several limitations worth considering. First, it assumes constant personal interaction between an expert and novice. Let’s be real; people come and go in organizations. This includes experts that eventually leave because of the inevitable baby-boomer silver tsunami (Flood et al., 2020), as well as millennials who have become known as job hoppers (Adams, 2016). In large organizations, hands-on personal interaction is also very challenging because there are likely people working from the opposite side of the world (especially in this post-COVID, remote work era). In addition, the generalizability of the model has been questioned with regard to limitations associated with cross-cultural transfer and replication, contextual factor constraints, and biases of local social practices (Farnese et al., 2019).

Further, socialization takes time and will only work to the extent to which there have been multiple people shadowing or picking an expert’s brain over the course of many months, if not years. But, since time is rarely a luxury for organizations, new technicians are oftentimes left to their own devices to learn “tricks of the trade” and develop their own heuristics (rules of thumb) to acquire proficiency or expertise via this particular approach.

Another methodology that has been employed for decades is the idea of Knowledge Elicitation (KE; Cooke, 1994). KE is the process of collecting human sources of knowledge, typically through interaction with an expert. In general, KE involves a psychological researcher shadowing an expert, observing their behaviors, asking a million questions, and then interpreting their findings for other people to use. This sounds great — like a one and done session, and now you’ve collected an expert’s tacit knowledge! However, KE actually tends to only extract explicit (knowing what) and implicit (knowing how), but not tacit (knowing in action) knowledge. Additionally, KE is highly manual, time-consuming, and tedious. Further, the results tend to vary from researcher to researcher and from expert to expert. Even worse, experts tend to be left out of the analyses and interpretation stage, which can lead to researcher bias and misunderstanding. So, while KE is good in theory, it’s not always practical or effective.

Knowledge Elicitation — But with a Modern Twist

The good news is, organizations can now leverage recent advancements in Augmented Reality (AR) technology to capture tacit knowledge from experts, which can then be used as job performance aid or job training aids. AR goes beyond current KE methods by situating experts within the real environment while eliciting their knowledge. Even better, AR puts the knowledge capture in the hands of the expert — by using this emerging technology to automate the KE sessions, we can reduce researcher bias and provide a seamless process to disseminate expertise once captured. We’ve developed such a modern approach to automating KE with AR, which we call EXTRACT™. EXTRACT™ was designed to specifically target the entire iceberg of knowledge: explicit, implicit, and tacit. First, EXTRACT™ uses a web-based automated structured interviewing technique to elicit explicit knowledge (knowing what). Then, progressive deepening questions are presented to support uncovering implicit knowledge (knowing how) in both web and AR formats. Finally, EXTRACT™ uncovers tacit knowledge (knowing in action) by using AR in context.

EXTRACT looks a little like this:

1. First, experts launch the EXTRACT™ web portal, fill out basic information, and respond to some initial KE probing questions (which were developed with Bloom’s taxonomy in mind to support training development later on).
2. Once the web form is completed, all of the collected information gets sent to an AR hardware device and EXTRACT™ software application.
3. From there, experts go to their operationally relevant environment and complete their tasks as if they were doing it in real life while seeing all the steps they entered in the web portal.
4. While completing tasks in situ, experts can capture first-person media of all the steps they take, and then they are asked to respond to additional probing questions (again, with Bloom’s taxonomy in mind).
5. Once completed, all of the data get sent back to the web portal where the expert can review, edit, and finalize their captured knowledge.
6. Finally, all of the elicited knowledge is used to automatically generate AR-based job performance aids and job training aids for novices.

EXTRACT(TM) Process for AR KE

So, some of you may be wondering, why AR? Why can’t the expert provide all of their information in the web form? Well, older methods of KE have used web-based technologies to capture expertise. However, in practice, they were typically limited to only capturing explicit and implicit knowledge (van Braak et al., 2018). Tacit knowledge was still largely missing from these methodologies. By using AR, we can situate experts in a contextually relevant environment. Coupled with the representation of all the information they entered in the web portal, completing tasks in the actual environment in which they would in the real world enhances an expert’s memory recall because tacit knowledge, while difficult to articulate, is best “communicated” via concrete situations. So, we prime memory in the web portal and enhance activation and recall of memory in AR. This is also supported by memory research; it’s very well established that encoding specificity — or retrieving information in the same manner in which it was encoded, enhances memory recall (Tulving & Thomson, 1973). So, the bottom line is that utilizing AR KE in operationally relevant environments and allowing experts to physically complete actual tasks results in a greater breadth of knowledge capture, including difficult to articulate tacit knowledge.

We’ve all heard the saying “an image is worth a thousand words,” right? Well, AR KE is more than that; how many times have we seen pictures or diagrams of equipment on textbooks and still felt confused about how they work (like those pesky IKEA instructions)? With the AR KE approach, the expert provides the novice with fully contextualized visuals within relevant environments, which taps into a wealth of knowledge associated with all those experiences packed into their brain. With the AR KE approach, you can store that media in a virtual library, and your organization now has its own “Khan Academy” for the newbies. No more second-guessing and praying to every deity that yes, this actually is Component 248, which (hopefully) fits into Object C1.15.

Actual IKEA instructions…that are not exactly helpful….

I can only imagine how much more quickly I would have closed the gap between my father’s seamless approach and my less-adept flying if we had used this AR KE approach. My father could have used the web portal to describe the instrument approach step-by-step to gather his knowledge. Then, once situated in the cockpit, embodying elements of perception and interaction with displays and controls, he could have shown me, step-by-step with first-person media, the steps he takes and the cues he is orienting to (maybe trimming control pressure off the yoke or making small power adjustments to stay on glide path) to ensure that no deviations are taken during the approach.

This modern AR KE approach has been implemented recently in the Navy’s newest aircraft carrier, USS Gerald R. Ford (CVN 78) (see Claypoole et al., 2020). The approach empowers the Navy’s most experienced Engineering Duty Officers (EDOs) or Master Chiefs (MCPO) to share their long-earned venerated knowledge with their junior Sailors…. rendering that information accessible long after they retire! So, the AR KE approach provides a means to rapidly and effectively transform experts’ knowledge into organizational knowledge that can persist long after retirements or job transitions. This is a critical capability for the Navy and other organizations across the globe. At the end of the day, sharing is caring, and it becomes even more critical when it benefits your organization.

References

Adams, A. (2016, May 12). Millennials: The job-hopping generation. Gallup. https://www.gallup.com/workplace/236474/millennials-job-hopping-generation.aspx

Claypoole, V. L., Stanney, K. M., Padron, C. K., & Perez, R. (2020). Enhancing naval enterprise readiness through augmented reality knowledge extraction. In Proceedings of the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) Annual Meeting, Orlando, FL.

Cooke, N. J. (1994) Varieties of knowledge elicitation techniques. International Journal of Human- Computer Studies, 41(6), 801–849

Farnese, M. L., Barbieri, B., Chirumbolo, A., & Patriotta, G. (2019). Managing knowledge in organizations: A Nonaka’s SECI model operationalization. Frontiers in Psychology, 10:2730. doi: 10.3389/fpsyg.2019.02730

Flood, F., Verdad, H., & Frederick, L. (2020). Silver Tsunami of retirement: Implications for consideration. In. A. Farazmand (ed.), Global Encyclopedia of Public Administration, Public Policy, and Governance. Springer, Cham. https://doi.org/10.1007/978-3-319-31816-5_3995-1

Fuchs, T. (2001). The tacit dimension. Philosophy, Psychiatry & Psychology, 8(4), 323–326.

McAdam, R., Mason, B., & McCrory, J. (2007). Exploring the dichotomies within the tacit knowledge literature: towards a process of tacit knowing in organizations. Journal of Knowledge Management 11, 43–59.

Nonaka, I., and Takeuchi, K. (1995). The knowledge creating company. New York, NY: Oxford University Press.

Polanyi, M. (1966). The tacit dimension. Chicago: University of Chicago Press.

Schmidt, F. L., & Hunter, J. E. (1993). Tacit knowledge, practical intelligence, general mental ability, and job knowledge. Current Directions in Psychological Science, 2 (1), 8–9.

Stephens, R. A. (2002). Knowledge modelling and representation. http://www.csm.uwe.ac.uk/~rstephen/courses/UQC833hm/week6/lecture.html

Tulving, W., & Thomson, D. (1973). Encoding specificity and retrieval processes in episodic memory. Psychological Review, 80 (5), 352–373.

van Braak, M., de Groot, E., Veen, M., Welink, L., & Giroldi, E. (2018). Eliciting tacit knowledge: The potential of a reflective approach to video-stimulated interviewing. Perspectives on Medical Education, 7, 386–393 (2018).