Autism XR Program Helps Students Navigate Social Minefields
Autism XR pairs students with autism with virtual humans who coach them through social situations. We talk to Marc Petz, the teacher behind this award-winning web-based AR program.
By S.C. Stuart
When Marc Petz was a first-year computer graphics/animation instructor at Michigan’s Kent Career Technical Center, a colleague was struggling to meet the needs of a student with autism. Petz’s class was asked to develop visually appealing stories the student could use to practice targeted social behaviors, and Autism XR was born.
The web-based augmented reality program has students participate in scenarios with virtual humans who—through facial recognition and natural language processing—coach them through specific social situations. The first iteration of Autism XR, an app-based virtual reality prototype, went on to become a top 10 national finalist in Samsung’s Solve for Tomorrow Tech for Good competition. It’s now being retooled as an AR program for general use.
Petz has also been working on a children’s book about inclusive learning environments with coworker Mary Musto and writing interactive stories with his kids via a startup called Kit Tales. We spoke with him recently to find out more. Here are edited and condensed excerpts from our conversation.
PCMag: Marc, explain how Autism XR has evolved.
Marc Petz: We decided to shift the direction of the idea to a fully web-based interactive AR environment, which currently is an experimental platform. With some of the current limitations associated with Google ARCore, WebXR, and phone technologies, we were forced to improvise.
One example is the eye-detection component of the application. We are not able to split the allocation of the front and back camera while there is an AR session active in the browser. Because of that, we had to use relative coordinates of where the camera is facing as opposed to real-time eye tracking.
How many scenarios are included in the current iteration of the application? And which have proved most useful?
There are currently nine scenarios that have been built and one how-to/tutorial scenario. One of our two research scenarios was used in a small research study in multiple classrooms with high school-aged students. “Initiate a conversation with someone you know” proved to be extremely useful and produced amazing results.
Did you employ voice actors?
No, most of the character voice acting was from my wife, children, and myself. A few students also contributed to the acting. I recorded all the voice acting with a small little portable studio I built for the purpose of voice acting.
It’s important to note that this program works without special equipment-no VR headsets or AR-modified glasses needed-as long as a user’s device can run WebXR.
We definitely saw an opportunity to build a tool that has the potential to benefit many people and eliminating any barriers was a huge priority. The high-end VR and AR equipment offer many great benefits, but we decided early that we wanted our solution to be very accessible, easy to use, and affordable. WebXR is the platform that allows us to accomplish those goals.
Talk us through setup.
Autism XR uses the WebXR functionality, which is still an experimental technology that has some steps that users have to take if they have a supported phone. The major players in this technology have limited the supported devices to mostly the newer, most recent mobile devices. An easy way to determine if an Android device is supported is visit the Google Play Store and attempt to install the Google ARCore app. If the Play Store has it available, then it looks good for your device to fully support WebXR.
With ARCore installed we need to get the Chrome Canary experimental browser installed on the mobile device. The Canary browser has all of the XR technology enabled (unlike the mainstream Chrome browser), and we found version 70 is the most stable for our current build.
Once the Google ARCore app and browser are installed, what do you do next?
After installation of the Google Chrome Canary v70 browser is complete, type Chrome://flags in your Canary v70 browser. Search for WebXR while on the Chrome://flags page and enable all of the results returned from the search. This will allow your device to use the WebXR content through the browser. If you were able to successfully complete the first three steps, then you are ready to open up your Chrome Canary v70 browser and navigate to marcpetz.com and begin using the web application.
What machine learning elements have you embedded in the application, and how do they enhance the experience?
We are using TensorFlow for the machine learning element. We are using it to build a facial/emotional recognition component in the application. This part of the application is not fully implemented yet due to the challenge of getting the AR session to split share the forward and rear camera on the mobile device. The current build of the emotional detection is working with good confidence, so I believe that once it is fully integrated into the web application it will enhance the learning experience greatly by coaching the users to practice appropriate facial expressions and emotional responses to various situations.
You collaborated with Mary Musto, teacher consultant and interventionist for Kent ISD’s transition center (special ed/gifted programs). Did you draw on specific academic research together?
Her experience and background was an integral piece in developing content that works. There was limited research on the topic of VR/AR and autism, but we did discover that many of the studies indicated promising results regarding the potential that technology has for improving communication and social behaviors for those with Autism spectrum disorder.
How did you test the program?
We were able to get approval from our school district to conduct a very small scale research study to validate the program and how effective it can be for improving targeting behaviors. Although our research might be considered insignificant due to having few participants, the results we had was remarkable, and it certainly justifies an expanded research trial. The students improved the targeted behavior from the training and also generalized the learned behavior from the training to other behaviors.
Can you share some of the feedback with us?
We conducted surveys after the research study was concluded and all the students who participated in the study indicated that they enjoyed using the application and it was fun and easy to use. They gave themselves positive ratings regarding improving the targeted behavior and recommended others to use the application. They also indicated that they would like to use it for other social skills and behaviors.
What’s next for Autism XR?
We are currently seeking funding to continue developing the application. Up to this point it has been fully developed by a small team of high school and college students, and it is our goal to keep students involved with the development. Our goal is to continue developing Autism XR with a vision that it will be an application that will ultimately help people and improve their lives.
Originally published at https://www.pcmag.com on December 31, 2019.
