Immersive Futures in Education and Design

By David Mayman, Kate Mashek and Blake Hudelson, Method

This is part II of Method’s ‘Designing for Education in the Age of AI’ series. Part I can be found here.

Here’s a challenge: think back to your early school days. Let’s say 6th grade. What can you remember? Do you remember homework assignments or your teacher’s lectures? Can you recall anything specific that you learned that year?

You may find that the only things you can remember are your science fair project, that awesome field trip to the beach, or that trip to the zoo with your family. Perhaps your memories of school are focused on a teacher that mentored you and helped you see the world in a new light?

Think about what your specific memories of that time in your life have in common.

Chances are those memories are of immersive and personalized experiences that were not part of your everyday routine.

At Method, we’re constantly asking questions about how the intersection of design and technology can solve really hard problems. Our team set out to study some of the problems we face in education and learning, and to test a hypothesis about the potential role of emerging tech.

To examine our results, it’s important to understand the state education is in and what’s not working.

The current state of learning

We are coming to the end of an educational era that was designed for the industrial revolution — where efficiency and rote memorization were prioritized over personalization and critical thinking. This factory model educational approach is bending under the pressure of 21st-century problems and more than ever we need new ways to educate our future leaders.

“Any fool can know. The point is to understand.” — Albert Einstein

In addition to a new skill set, learners have to overcome a rigid system that, in many cases, does not appreciate their individual needs for success. No two students are the same. A student’s environment, learning needs, and current emotional state all impact their ability to engage and succeed.

The other side is equally challenging. Teachers are asked to meet each student where they stand while still reaching set goals for the class collectively. Often, instruction becomes “scripted” in delivery and teachers are forced to teach to the average of the class, deprioritizing those ahead or behind. Add to that the realities of underfunded schools, and a teacher’s ability to effectively teach is even further compromised.

https://commons.wikimedia.org/w/index.php?curid=49726769

How can 21st-century tools help?

In the professional sector, we’ve seen an explosion of new technologies that are changing the way we tackle today’s problems. Data science, artificial intelligence, virtual reality, and augmented reality are all promising technologies making headway in many industries and we were excited to explore it to the education sector. So what could these tools be used for in education? It turns out quite a bit.

We see an exciting opportunity with the combination of AI and VR* to enable immersive, personalized, context-aware experiences for education. Let’s break down both of these technologies.

*We should note that we’re using VR as a contemporary stand-in for any number of present and future immersive technologies, i.e. Augmented or Mixed Reality.

Personalized experiences

Artificially intelligent systems are incredible at gathering massive amounts of data, finding correlations, and predicting and acting on future outcomes. We can apply this technology through a lifelong learning system. Throughout a student’s life, individualized data can be collected and synthesized into a personalized “fingerprint”, built from their learning history, success patterns, and their present emotional state.

Imagine an AI assistant that knows when a student is most productive, vulnerable, confident, or insecure. What if it knew what subjects the student excels at or struggles with, or what learning modes a student responds best to, and provided materials unique to the student’s abilities, interests, and state?

Google Assistant provides users with relevant daily suggestions based on their behaviors

Immersive experiences

Virtual environments, as you may know, are fabricated experiences that utilize sight, sound, touch, and our own spatial perception to put us in new places. They are fantastic storytelling vehicles that can quite effectively capture our emotions. While we’re fascinated by how immersive virtual environments are, we’re more so interested in how easily reconfigurable they can be. Compared to the physical world, virtual environments can be modulated in ways that aren’t possible in reality — in this case, to cater to the attention of a learner. They are a great adversary to this moving target.

Immersive virtual environments are used successfully in gaming, retail, and advertising

The real opportunity: AI + VR

We see great potential in the combination of these two counterpart technologies. An AI assistant paired with a set of virtual experiences capturing real-time emotional and situational feedback can modulate to adjust to a student’s individual needs, down to the second. Specific teaching decisions could be made with massive variety and flexibility, constantly updating as more performance data is collected and shared.

Together these technologies pose an opportunity to democratize active learning through participation and engagement — offerings that only a tiny percentage of students are able to receive today with one-on-one instruction. The combination of AI and VR could not only drive engagement, but aid in recall and mastery of concepts through multi-modal activities and rich stories.

This imagined AI assistant could be of great value to the teacher as well. Teachers might ask it for help generating personalized content for all of their students, using the insights it has gathered over years of assisting each student. This system might be leveraged in a number of ways, high- or low-tech, such as creating personalized worksheets for homework.

Designing an adaptive learning framework

To turn these considerations into a model we could prototype, we used a framework to define a smart learning experience. It can be described as the combination of three core parts: inputs, decision making, and outputs.

1. Inputs

Inputs include the student’s current emotional state, educational performance history, and level of aptitude and grit. All of these inputs are collected to create a “fingerprint” unique to that student, which is used in conjunction with the intended knowledge or concepts to be learned to generate personalized content.

2. Decision-making

The AI assistant filters these data points, assessing not only the individual student’s data, but leveraging data from other students and best practices from pedagogical resources.

3. Outputs

Once the system has enough information to understand the student, it can adjust the tone, context, and teaching mode to provide a personalized and flexible lesson. It also does the job of informing the instructor of the student’s progress along the way. The AI can empathize with the student, and adjust the mood or environment to better suit their needs and emotions. The AI can adjust the context (characters, storyline, environment) to better engage the student and their areas interests. The AI understands what the student can already grasp, and may employ metaphors or similar concepts to provide scaffolding and connect with past learnings.

Prototyping a smart learning experience

To provide a snapshot of what this experience could be, we created Scout — a personalized learning platform to help students learn throughout their lives. As a test output, we built an interactive VR prototype that brings a student through an immersive lesson. This prototype is viewed through the eyes of Taylor, a 6th grade student learning about photosynthesis and plant health. It demonstrates how an output (a VR experience) might be modulated based on imagined inputs (Taylor’s interests, learning modes, current emotional state).

A walkthrough of Scout, a prototype for VR learning

We find it particularly exciting to think about an agile experience that can change its approach during a single lesson based on the student’s interpretation of the subject matter and emotional reaction. This prototype moves from instruction, inquisition, to free exploration in a matter of minutes based on Taylor’s attention span and interest level.

The Scout exhibition at Method’s headquarters in San Francisco

Finding the right conditions and scenarios for optimal learning is an intricate and delicate balance. We imagined Scout as an ecosystem of touchpoints and learning experiences powered by low and high tech. From in-classroom supplemental information, to worksheets, to full immersive environments. The medium is as important of a decision as the lesson itself.

Designing with change in mind

One of the most exciting aspects of this proposed system is how new emerging technologies may slot into the framework. VR is the most mature immersive technology to date, but is definitely not the only high-tech means of supplementing education. Our team is perhaps even more excited about mixed and augmented reality adding the capabilities of digital on top the real world. As technologies continue to be refined, the possibility of a powerful learning system will become clearer.

There are currently massive efforts to perfect immersive experiences across other industries that we work with. Advertising, gaming, and entertainment tend to benefit from these advances first. As technological advancement continues to accelerate, we find increasing importance in thinking wider in how we apply innovation. We see it as our responsibility as designers to find other ways to use technology and advance human capabilities.

This is an ongoing exploration by Method. As always, if you have perspectives to share, are excited by our work, or curious about Method, please comment below.