What if: Vocabulary Training Became an Immersive Showdown in your Living Room?

Our experiment, Language Defense, serves as a showcase to explore the enhancement of language learning in Mixed Reality. (Source: AI-generated/Leonardo.ai)

Two weeks ago, we explored the pivotal role of communication in business. Introducing our new Taikonauten research project, we embarked on a captivating experiment: merging vocabulary training with a Tower Defense game in Mixed Reality as a secondary language learning tool. Now, it’s time to unveil the results of our 14-day-experiment. What’s the current status of the prototype, what challenges have we faced and how do we feel about the result? Join the journey and explore our discoveries and beyond.

💡Who are the innovative minds behind the magic? 💡

The Research & Development lab at Taikonauten is composed of eight individuals, bringing together expertise in engineering, UX/UI design, strategy, research, creative technology and communications.

The story so far

You missed our previous article about language learning opportunities with Mixed Reality? Let’s quickly summarize the main facts in bullets:

• Communication is crucial for company success
• There is evidence that Mixed Reality may have huge potential to elevate language learning results
• Currently, no existing Mixed Reality applications cater specifically for language learning
• We envisioned the future of language learning applications in Mixed Reality with three technical use cases
• We opted for a quick reality check and thus decided to develop a prototype for the Language Defense experience, merging vocabulary training with tower defense gaming principles, all within a 14-day timeframe

Light up your wildest visions

Actions speak louder than words. So let’s face the implementation. But what does it need to develop the prototype?

# 1 Linguistic content:

We use a Large Language Model (LLM), powered by artificial intelligence, to generate language content for effective vocabulary training. LLMs are trained on vast datasets of textual information to process, comprehend and produce natural language.

To ensure contextually relevant and domain-specific responses, we utilize the OpenAI Assistant API. By specifying initial scenarios within prompts, we direct the bot to act as an English vocabulary trainer, providing words related to specified topics.

The effectiveness of this approach hinges on the selected model and its version, as outcomes can vary. Each version of the model serves as a checkpoint, representing its state at a specific point of time. Therefore, prompting should be tailored to meet the specifications of each individual model.

💡By the way: The Assistants API allows you to build AI assistants within your own applications. Assistants have instructions and can leverage models, tools and knowledge to respond to user queries.

Prompt: Create a multiple choice quiz for business english to learn vocabulary. It should have one question and three possible answers. One of the answers should be correct. The other two wrong. Give me an example output for this as a json object that I can integrate into an app.

{
  "question": "What does the term 'ROI' stand for in business?",
  "options": [
    {
      "text": "Return on Investment",
      "is_correct": true
    },
    {
      "text": "Return on Income",
      "is_correct": false
    },
    {
      "text": "Revenue on Investment",
      "is_correct": false
    }
  ]
}

#2 Technical foundation:

Time to bring Language Defense to life in a Mixed Reality environment. We won’t delve into the complex details of Mixed Reality development in Unity at this point. To learn more about the technical development process check out our comprehensive tutorial about Mixed Reality creation with Unity.

Considering the simplicity of navigation, integrating controllers at this stage seems unnecessary from our perspective. Instead, we opt for gesture recognition and motion tracking, ensuring seamless interaction with vocabulary, even in sunny conditions where controllers may struggle.

However, we assume that some users may prefer controller interaction. Therefore, including controllers in future iterations of this prototype is at least conceivable.

💡By the way: To ensure the application’s full accessibility, additional navigation options such as head or eye tracking are necessary in a ready-to-use version, which we left out in this demo of Language Defense for now.

# 3 UI elements:

For an intuitive experience, we create a matching user interface in our desired look — a timeless design with minimalistic elements that interact with the real world. We aim for a cyberpunk-inspired theme with futuristic elements, industrial colors and animated robots.

For this approach, we design the most important elements for the application’s skeleton in Figma:

• one start button
• one question button
• three answer buttons
• one game board

Furthermore, we obtain the 3D robot assets, including animations, from the Unity Asset Store.

💡By the way: Are you missing any game elements that would round off your experience, such as a progress bar or lives counter? Initially, the prototype focuses on merging language learning with tower defense to assess its core concept’s effectiveness and gather feedback from testers. But rest assured, we’ll continue refining the learning experience as our research progresses.

3, 2, 1…time to reveal the prototype

Curious about what Language Defense looks like? Let’s go. Sit back, relax and enjoy the teaser:

Language Defense Teaser. (Source: Taikonauten)

What is your initial impression of the prototype? As time flies by in just 16 seconds, let’s revisit our illustration of the concept from our previous article:

Our chosen gameplay variant for Language Defense. (Source: Taikonauten)

We are definitely proud of the result. But as expected the application comes along with a large number of open questions we need to answer.

Gladly, the tight time frame of 14 days for the development has the decisive advantage that validating and collecting external feedback is possible at an early stage. Let’s find out how the prototype is perceived by our colleagues: It’s time to carry out the first reality check with some Taikonauten team members outside our Research and Development lab.

Don’t stop believing, but start validating

Generally, the gamified setting of vocabulary training in a Mixed Reality environment is perceived as an innovative secondary learning tool, which helps to validate previously acquired language knowledge. But we quickly realized that using an immersive technology does not automatically lead to an immersive learning experience. It is primarily about the seamless fusion of the digital and real worlds.

“I would rather imagine having to actually walk through the characters or interact with them in some way.”

In addition, our colleagues not only missed a greater variety of features to suit their preferred learning style, such as listening and comprehension tasks, but also basic functions of the game that we have not yet implemented.

“I would question whether multiple choice for vocabulary is the only effective way to learn a language. Also a game progression with different levels of difficulty and a ranking system would be great.”

And what about its applicability in everyday working life? Well, expandable. Although the approach is perceived as innovative and refreshing compared to classic language learning applications, there are still a few concerns.

“In the office context, I’m currently wondering where to spawn the game board. Additionally, I would expect the vocabulary to be associated more with the objects in the office rather than with virtual opponents.”

In summary, although we have achieved our goal of developing an Mixed Reality application that facilitates language training, there is still plenty of room for further enhancement to deliver a fully immersive experience tailored to the demands of office life. Our most important learning:

“At the end of the day, it’s not about the technology, it’s about the overall experience”

Prototype, validate, learn, iterate

Let’s take a moment to reconsider the opportunities and limitations and identify the most important questions that need to be addressed as we continue with our project.

How might we…

1. identify the defining characteristics of immersive learning?
2. determine how people learn best and how our prototype aligns with these principles?
3. evaluate whether the combination of vocabulary training and a tower defense approach truly enhances language learning, or if we are only scratching the surface?
4. enhance the learning experience further with artificial intelligence?
5. explore additional possibilities for language learning in a Mixed Reality environment?
6. uncover hidden obstacles in Mixed Reality that may hinder full immersion in language learning experiences?
7. determine the essential requirements of an immersive language learning application?

Let’s tie up loose ends

To summarize, Language Defense serves as an excellent starting point for in-depth research based on our exemplary use case. Our internal tests already indicate that the concept of Language Defense acts as a unique and refreshing secondary learning tool for vocabulary training in a Mixed Reality environment.

Let’s explore if there is even more to it. Immersive language learning is a complex field and we expect that vocabulary training is just one facet of the equation. In the further phase of our experiment we will delve deeper into the fundamentals of immersive language learning by theoretical research, enhanced ideation sessions and enriching discussions with experts. Stay tuned and follow our Taikonauten Magazine to explore further insights.

Editorial: Shirley Schmolke