IBM Watson Helps University Students Learn Mandarin
The Rensselaer Polytechnic Institute (RPI) in Troy, NY is using IBM Watson technology to provide an immersive, real-life experience using avatars to teach students Mandarin.
Universities have come a long way in their language studies, moving beyond just traditional classes in a lecture hall and prerecorded lessons without interaction. Now students have access to technologies such as artificial intelligence (AI) and natural language processing (NLP) in an immersive learning environment. That is what’s happening at the Cognitive and Immersive Systems Lab (CISL) on the Troy, NY campus of the Rensselaer Polytechnic Institute (RPI). It’s here that the institute has teamed up with IBM Research to create the tech for a six-week summer course in which the students interact with AI agents in environments that include street markets and restaurants.
The course, entitled “ AI-Assisted Immersive Chinese,” features a 360-degree panoramic display system of computer-generated scenes in the lab’s Cognitive Immersive Room (CIR), which is also called the “Situations Room.” The virtual scenes take place in China. Students converse with avatars that are powered by Watson Assistant and get corrected on their speech and pronunciation. In the CIR, students use conversational AI, narrative generation, spatial context awareness, and gesture and facial recognition tech.
The CIR is an example of the diverse range of potential for IBM Watson in several industries in addition to education. These industries include agriculture, human resources (HR), and fleet management.
At the RPI, in addition to Watson Assistant, students are using Watson Speech to Text, Watson Text to Speech, and Language Translator in their studies.
An Active Learning Experience
For Julian Wong, a junior at the RPI, the immersive AI experience was an appealing way to learn Mandarin because it gives him a more active way to learn. Because Mandarin is a tonal language in which a change in pitch can alter the meaning of words, the avatars in the AI experience provide feedback on tone to make sure Wong and the other students are getting the pronunciations correct.
Working with AI in the CIR provides a nice balance with the regular classroom time, according to Wong. “Meeting to talk to the computer…definitely helps with a lot of aspects of learning the language, especially verbal skills,” Wong said.
Wong’s Mandarin class meets four times a week. On Mondays and Fridays, he attends a class in a traditional classroom with Helen Zhou, Associate Professor at the RPI. There he learns new vocabulary and gets an introduction to phrases and grammatical structures. On Tuesdays and Thursdays, the class meets in the CIR, where students conduct conversations with virtual agents. In a restaurant environment, Wong said students can go through the entire process of sitting down in the restaurant, looking at a menu, ordering food, speaking with a waiter on how the food is prepared, and paying the bill. Other environments include a garden, a market, and a school campus.
The Mandarin class in the immersive classroom differs from a traditional environment because it provides a multimodal experience of language and video, according to Zhou. She said a traditional class would be less immersive and interactive. “In the simulation in the immersive classroom, we have two agents to fight with each other, to try to persuade the students to buy the product,” Zhou said. “So students would understand by personal experience rather than watching videos or listening to lectures only. I believe they would pick up the culture more easily than just by learning from textbooks or listening to tapes.”
In the CIR, students take quizzes related to the scenes with which they’ve interacted and raise their hands to give their answers. Computer vision tech from the cameras in the room capture their gestures. The AI assistant gives a quiz related to the scene and students must answer with a phrase in Chinese.
“If you want to select one of the pop-ups that’s on the screen, then you can hold out your hand, palm open, and then you close your hand,” Wong explained. “You select whatever the cursor is currently on. For the little quizzes, if you know what the right answer is, then you can move your hand over to the characters that you want to select, and then close your hand to select them. It will let you know if you got it right or not.”
The virtual agent understands the students most of the time, except for when they may not have their microphone close to their mouth. The system has different voice recognition settings that pick up students if they’re off with their pronunciation. It also provides immediate feedback in a conversation if you don’t have a professor present. On their own time, the students in the Mandarin class go to a website to access voice clips to practice.
A cool feature of the immersive experience at the RPI is spatial context awareness, in which multiple virtual agents on the screen can tell which one you’re making eye contact with. “If you say something while looking at one agent, then that agent will respond and not the other one,” Wong said.
How the Immersive Experience Developed
The immersive experience at the RPI came was created when language professors and game design professors met to discuss a better way to teach language. A plan to develop a role-playing game soon veered into a discussion of how to teach Mandarin through a simulation of real-life scenarios, according to Hui Su, who is both the Director of Cognitive User Experience at IBM Research and Director of the CISL. IBM and the RPI began developing the course at the end of 2015, and they established the joint lab at that time. The CISL then took shape soon after.
“The purpose of the lab is to build the cognitive and immersive environment to improve group activities, to augment group intelligence in the context of learning and decision making,” Su said. “We also focus on building cognitive immersive boardrooms…to look at the data of a critical situation and try to make sense of the information and create decisions or recommendation for the decision-makers.”
As part of the AI language learning program, IBM and the RPI are experimenting with pitch contour analysis. Su described how this tech works. “When you speak a syllable, when you pronounce a syllable, the underlying tech will capture the voice, and it generates a visual contour of how you pronounce that syllable,” Su said. “Then it uses that visual contour and compares that visual contour with a native speaker’s visual contour.”
Students can then make adjustments in their pronunciation to reach the correct tone. They’ll learn that they need to change pronunciations of certain syllables. With the knowledge students gain by learning Mandarin in the immersive classroom using Watson tech, they’ll have more confidence conversing in real life, according to Su.
“The whole idea is to provide enough cultural context through our immersive environment and AI tech to enable the students to practice an exercise,” Su said. “So they will not have fears like talking to people in real life.”
A Bridge Between the Tech Industry and Academia
IBM and the RPI are able to build a bridge between IBM researchers and faculty members, with IBM embedding a researcher on campus, namely Su. Meanwhile, Zhou, the professor of the class at the RPI, also provides feedback to the designers, users, and teachers on how to improve the immersive experience in the classroom.
“I need to give them immediate feedback to debug or to improve the design so that we can offer a more natural classroom to the students,” Zhou said. “So there’s lots of work involved but it’s worth it.”
The Future of Language Learning
This type of immersive environment is valuable because of the ability to provide a seemingly real-life environment, according to Zhou. “The AI immersive environment can not only enhance students’ speaking and listening, but also the pragmatics of using that language in real-life situations,” Zhou said. “That’s why AI is more ideal for me as an instructor so that students can get a sense of using the language early on, even for beginner learners, without traveling to the country.”
Multimodal learning environments like those offered at the RPI will expand the role of AI in the classroom. With the experience that AI provides through virtual agents, students will be prepared to tackle various scenarios in the real world with the necessary language skills.
Have any questions about artificial intelligence and how it’s used in education? Join the PCMag@Work discussion group on LinkedIn, and you can ask vendors, other professionals like yourself, and PCMag’s editors.
Originally published at https://www.pcmag.com on August 8, 2019.
