Report: How XR Works With Your Brain
Why extended Reality (xR) is an Effective Tool in Manufacturing: A Brain Science Analysis.
This report focuses on a learning science evaluation of extended reality (xR) technology’s potential as a learning and training tool in the manufacturing sector. xR technologies come in two major forms: augmented reality (AR) — in which the learner is in a real environment where digital information is overlaid onto the learner’s field of view — and virtual reality (VR) — in which the learner is immersed in a completely new virtual environment separate from their physical surroundings.
This report shows that xR technologies hold great promise in manufacturing because they reduce the cognitive load on the learner, provide the opportunity for limitless practice, and speed learning and retention by broadly engaging multiple learning systems in the brain in synchrony.
The Importance of Learning and Training in Manufacturing
High quality training tools are necessary to enhance accuracy and efficiency in the manufacturing process, where one mistake or small inefficiency can have significant ripple effects that are costly and time-consuming. For example, individuals tasked with supply chain management must quickly and accurately evaluate inventory levels, place necessary orders, and facilitate the movement of supplies and goods. All personnel in manufacturing must be well versed on the environmental health and safety rules, regulations, and emergency action plans. This is critical for compliance purposes, but also to reduce the risk of accidents that can endanger workers and slow or stop the manufacturing process.
Product development is critical and involves an iterative learning process in which prototypes are developed, evaluated, and ultimately lead to a viable product. Product manufacturing requires highly trained technicians to operate and maintain the manufacturing equipment. When the equipment breaks down operators must troubleshoot and address the problem as quickly, accurately and safely as possible. Field service is also important to keep customers happy and their equipment running smoothly.
Learning Science of Training in Manufacturing
“Learning is an experience. Everything else is just information” Albert Einstein
This is an insightful quote from Albert Einstein that is supported by learning science — the marriage of psychology and brain science. As elaborated below, experiential learning is effective because it facilitates the engagement of multiple learning systems in the brain in synchrony. Experiential learning also provides the foundation for the effectiveness of AR and VR in manufacturing and other sectors.
The human brain is comprised of at least three distinct learning systems; a schematic of theseis provided in the figure below. The cognitive skills learning system in the brain has evolved to obtain and process knowledge and facts. Whether comparing inventory with a checklist, studying safety rules and regulations, reading equipment training manuals, or vetting product descriptions, the cognitive skills learning system is being recruited.
Cognitive skill learning tends to involve processing text and schematics and is limited by the learner’s working memory and attention span. It requires focus and mental repetition for long-term memory storage. The cognitive skills learning system encompasses the prefrontal cortex, hippocampus and associated medial temporal lobe structures in the brain. The ultimate goal of this system is to transfer knowledge from short term memory in the prefrontal cortex to long term memory in the hippocampus and medial temporal lobes. Processing in this system is adversely affected by stress, pressure, and anxiety. This system is slow to develop, not reaching maturity until individuals are in their 20’s, and begins to decline in middle age.
The behavioral skills learning system in the brain has evolved to learn behaviors. It is one thing to know what to do, but it is completely different to know how to do it. Knowing the safety rules and regulations is completely different from knowing how to initiate those behaviors in an emergency. Memorizing the operations or repair manual for a piece of machinery is different from knowing how to operate or repair the machine. Behavioral skills are learned by doing.
Processing in this system is optimized when behavior is interactive and is followed in real-time (literally within milliseconds) by corrective feedback. Behaviors that are rewarded will be more likely to occur in the future, and behaviors that are punished will be less likely to occur in the future. Interestingly, this system does not rely on working memory and attention, and “overthinking it” hinders behavioral skills learning. Behavioral skill learning is mediated by the basal ganglia and gradual, incremental dopamine-mediated changes in behavior. The ultimate goal of this system is to train direct neural connections between sensory regions and motor regions in the brain that drive behavior.
As Einstein so eloquently stated, experience is at the heart of all learning. It is also the key ingredient in xR training. The experiential learning system has evolved to represent the sensory aspects of an experience, whether visual, auditory, tactile or olfactory. Every experience is unique, and adds rich context to cognitive and behavioral skills learning. The critical brain regions associated with experiential learning differ as a function of the sensory input. Visual representations are formed in the occipital lobes and auditory representations are formed in the temporal lobes. Tactile representations are formed in the parietal lobes and olfactory information is represented in the piriform cortex and olfactory bulb.
xR Applications in Manufacturing
The key ingredient of xR technology in manufacturing is that experiential learning systems are being recruited in synchrony with cognitive and/or behavioral learning systems. Here I outlined 5 use cases for xR in manufacturing.
Supply Chain: Suppose you are tasked with inventory control. In a typical scenario, you might approach each set of inventory, count it, note this on a clipboard or tablet, and tag inventory that is low and in need of replenishment. In this case, your cognitive system is highly engaged. You must focus on the inventory and count it, then shift attention to the clipboard or tablet, find the appropriate location, write the number down, then repeating the process. The working memory and attention load are high, attention switching is necessary, and thus, the potential for error and reduced efficiency is also very high.
Now consider the same situation but where you don a pair of augmented reality glasses. You scan the inventory room and visual cues on the display direct you to the first set of items to be counted. You scan the items and a built-in algorithm counts the number of items present and the number needed. When the inventory is low a warning is presented and you are asked to tag this item for replenishment. The glasses then direct you to next inventory item to be counted. Once the inventory process is complete the system prompts you to send the inventory needs to the relevant purchasing departments. In this case, your cognitive load is reduced, attention switching is minimized, the experiential and cognitive systems are engaged, and you are guided through the process in the most behaviorally efficient method possible. The process is faster and more accurate and the likelihood of a supply chain error is reduced.
Environmental Health and Safety: Suppose you work in a manufacturing plant and you need to learn the steps to take to keep the workplace safe. You could read a workplace safety manual that describes the rules and regulations with text and figures. This places a heavy load on working memory and attention to translate the abstract information into a visual representation of the steps you would need to take during a safety incident. Alternatively, you could read the manual and also watch a video that shows actual or simulated safety conditions. This is better because the experiential systems in your brain are being engaged to some degree. However, in both cases, your goal is to store this information in memory for use later during a real-world situation in which your and your co-worker’s safety is in jeopardy.
Alternatively, you could be directed to predefined locations in the workplace while wearing augmented reality glasses. At each location your view of the workplace is augmented with static text overlays or dynamic video that provides specific information on the safety guidelines or usage of safety tools. For specific safety equipment, you could receive step-by-step instructions or visual labels describing exactly how the equipment works. You might then be asked to demonstrate your skill with a specific safety tool, with or without text-based prompting and receive real time feedback. Analogously, you might don a VR headset and be transported into the middle of a workplace emergency. You might scan your virtual environment watching personnel follow or fail to follow the guidelines within this chaotic situation while receiving auditory feedback on the appropriate actions.
In the AR and VR examples, there is minimal need to translate abstract text or figures into a visual representation because this information is being supplemented with experiential learning that is proximal and salient. Because multiple brain systems are active, the memory traces in each system are strong and interconnected. Behavioral learning will also occur when you demonstrate your skill with safety tools and receive feedback either with the AR tool, or in VR with haptic feedback incorporated.
Product Development and Prototyping: Suppose you are part of the product development team. This is an iterative process in which schematics and computer aided design (CAD) tools are utilized and ultimately lead to the development of a prototype. As with reading text and studying static figures, this places a heavy load on the cognitive skills system in the brain to construct a 3D dynamic image from a series of 2D static images. It is difficult to know if this translation is effective for each member of the team and whether all team members “visualize” the same product. This is suboptimal and inefficient.
Instead, imagine donning an AR or VR tool in which a virtual 3D dynamic prototype is presented simultaneously to all members of the product development team early in the development process. The team is either physically present in a single room or are virtually collaborating via VR. Either way, all team members can view the single prototype, can “hold” it in their hands, and can manipulate the prototype — all while discussing specific design features that they like or dislike. In this case, the cognitive load is minimized because a 3D dynamic prototype is present from the start. In addition, because the prototype can be shared virtually, team members in any location can collaborate.
Equipment Operation and Repair Training: Suppose you are training on the operation and repair of a large, expensive, and rare piece of equipment. Because there are only a few training centers around the world you begin by reading and memorizing the operations and repair manuals. Then you fly to a training center for an intensive 6-week course where you attend classroom training sessions and obtain hands-on equipment training. Once the training is complete you return to work as a certified operator and repair technician. Although certified, there is still much on-the-job learning to obtain to become an expert. In this scenario, you begin the training process with a heavy load on the cognitive system as you attempt to construct 3D dynamic representations of the equipment from 2D static training manuals. With that information in hand you begin the intensive hands-on training that will facilitate behavioral skills learning. Although the practice is intense, it is impossible to obtain enough experience in a 6-week course to attain expertise.
Now suppose that you begin the training process by exploring the equipment in virtual reality. You can walk around and view the equipment while receiving verbal or written instructions about the specific operations of each part. If you have haptic hardware, you can touch the equipment and manipulate it, again being driven by verbal or written instructions. You might even have a virtual trainer in the same room with you pointing out specific features and answering questions.
Instead of flying elsewhere to a training facility, you might don your VR headset each day for virtual lessons, and your homework might involve troubleshooting an array of equipment malfunctions. You might even be placed in a virtual emergency situation in which you receive real-time feedback or as part of a test. Once certified, you might continue with more advanced VR training on more complex and rare emergency situations, but ones that can have a major impact on the manufacturing process. In this case, you are receiving cognitive and behavioral training simultaneously with experiential learning. There is no need for physical travel, and the rare and expensive piece of equipment is available, at your fingertips, 24/7. You can train on rare and dangerous situations that would be difficult, if not impossible, to simulate during real-world training. Your confidence and level of expertise will be higher and you will achieve these more quickly.
Troubleshooting and Field Service: Suppose you are a consumer of a manufactured product (e.g., computer, appliance, DIY furniture, etc.), and you attempt to troubleshoot some problem either with online manual, YouTube videos, or on the phone with a technician. You find yourself trying to convert the abstract information into actionable behaviors to fix the problem. You are constantly switching your attention back and forth from the manual or video to your product, or trying to find the right words to explain the problem on the phone wishing that the technician could “see” what you see. This is frustrating and ineffective.
Now image that you could focus your tablets camera on the device and step by step instructions were provided for how to troubleshoot the problem, or better yet provided a bird’s eye view for a remote technician. A built-in algorithm or the technician instructs you on what to do and instructions are supplemented by arrows, highlighting or some other tool to guide you. In this case you being guided on how to fix the problem with information that augments your view of the product. The problem would be fixed quickly, accurately and you will be up and running again.
Big Data: The Hidden Bonus of xR Training
The focus of this report has been on the psychological and brain-based advantages of xR in manufacturing, but there is another advantage of xR technologies that is equally important and that is the promise of actionable data. With these technologies one can quickly obtain subjective ratings of confidence, satisfaction and engagement, or develop objective tests to determine whether learning has actually occurred and to challenge the learner. These can be combined with eye gaze and heat map patterns that provide a direct window onto attentional processes and engagement, or haptic data to provide real-time behavioral feedback. These data can be used to speed iteration toward optimal solutions and to quantify the ROI associated with xR technologies.
Quality Content and Optimized User Experiences
AR and VR technologies offer promise as effective tools for manufacturing by facilitating broad engagement of cognitive, behavioral and experiential systems in the brain that are critical in manufacturing. A note of caution though. Although these tools have great potential in manufacturing, the tool in and of itself will not meet this challenge. These tools provide augmented and virtual experiences and information to human users.
Thus, the augmented and virtual content and temporal presentation of the content must be optimized for human consumption. Augmented information can reduce the cognitive load, but can also overload the user. Virtual information must be realistic and lead to a veridical sense of “presence”. Optimization follows from good experimental testing and modification. Fortunately, the rich and broad set of real-time data that can be extracted from AR and VR technologies makes this a viable and realistic prospect.
For companies looking to get into Immersive technologies our VR Consultancy service offers comprehensive support in strategic deployment of Virtual, Augmented and Mixed Reality
Todd Maddox is Science, Sports and Training Correspondent at Tech Trends, and the CEO of Cognitive Design and Statistical Consulting. Follow him on Twitter @wtoddmaddox
