Tutorial: Expressive AI-Driven Conversational Characters in AR: Spirit Character Engine in Unity with ARCore + SALSA LipSync
Spirit Character Engine is an Authoring Tool and SDK for creating expressive, AI-driven characters within a narrative context. They can exhibit personality, emotional expressivity, and conversational richness. The way the technology works is this: you pass in user input (whether that's typed or spoken natural language and even gestures) to the API, and Character Engine will select what the most salient response might be — from the Character’s knowledge model and plot information. It then sends this selection in a callback to the game engine — giving you information on the character’s chosen response (in text), their emotional state, and more.
Given this, it’s easy to integrate Character Engine with a number of options for audiovisual representation using third party tools. A quick, easy option if you want to make an AR character with some animation and lip sync ability tied to the character’s speech output is to use Unity, with the following tech stack:
Spirit Character Engine + SALSA + RTVoice + ARCore + Unity
Getting Started
- To get started with Unity & ARCore, you need the latest version of Unity (2017.2+) and the Android 7+ SDK (API level 24 or higher). For more information on this, you can read this official Google quickstart for Unity.
- Also, download the ARCore SDK for Unity.
- Create a new Unity 3D project and import the ARCore SDK for Unity .unitypackage file by clicking the menu Assets > Import package > Custom Package and import everything in the Unity package.
- Once you know the location of your project on disk, export your project from your Character Engine Authoring Tool, by pressing the ‘Export’ button in the menu. First, make sure your Project Settings are set to export to a StreamingAssets folder in your Unity project.
- Set up your project for Character Engine according to the instructions in your Spirit documentation — this is currently limited access to existing beta clients, but you can also sign up for Beta access now.
5. Next, the rest of the ARCore setup can be found here: https://developers.google.com/ar/reference/unity/
6. Finally, the SALSA integration is quick and easy — you can download the SALSA with RandomEyes with LipSync from the Unity Asset Store.
7. Install SALSA with RandomEyes into your project by importing from the asset store straight into your project.
8. Install RT-Voice. You can download it from the Unity Asset Store here: https://www.assetstore.unity3d.com/en/#!/content/41068
9. Import the SALSA with RandomEyes RT-Voice support package (Salsa_RTVoice). Select [Assets] -> [Import Package] -> [Custom Package…] and Browse to the [Salsa-RTVoice_1.0.0.unitypackage] file and [Open].
10. Set up a SALSA 2D or SALSA 3D enabled visual character. The SALSA integration is quick and simple — the SALSA 2D or SALSA 3D AudioSource just needs to be set to the RTVoice (TTS) plugin GameObject/AudioSource. SALSA takes care of the rest.
That’s it! Below is a demonstration of this tech stack at work with a very quickly put-together Character Engine project.
This example demonstrates setting up a character that is rendered in AR, and provides robust yet lower-fidelity animations and lip sync as its a 2D, cartoon-ish character.
However, we also support many types of other audiovisual representation for Characters too — such as high-resolution lip-sync and facial animation. In fact, our upcoming demos to be debuted at Game Developers Conference 2018 address these and provides great examples. If you want to get in touch to discuss these, let us know: hello@spiritai.com.
Spirit AI builds tools to make the future of digital interactions better: both with virtual humans, and real humans. We make Character Engine, for authoring dynamic improvisational AI characters, and Ally, a tool for detecting and intervening in the social landscape of online communities — to curtail online harassment, or to promote positive behaviour.
