How I Made AI NPCs that can control their actions along with dialogue
AI NPCs that can control their actions along with dialogue. For instance, if I ask an NPC to tell me its favorite magic spell, it not only tells me the spell but also performs it!
Video Demo ✨ ☃️
Tools 🚀 🔧
- 🤖 Cohere’s Language Models and Embedding Models
- 🗣️ ElevenLabs
- 📝 AssemblyAI
- 🗃️ ChromaDB
- 🔗 Langchain
- 🎮 Unreal Engine
- 🌲 Quixel Bridge
- 🧔 Metahuman SDK
- 🙆 Mixamo
- 🏃 Runtime Audio Importer
How it Works 🧠💡
🤖 Cohere’s Command model generates the character’s response using a template that includes talking style, action list, bio, etc and then parses the action and dialogue.
🎮 Unreal Engine can’t run Python directly, so it uses text files for communication. When you hit Ctrl+T, Unreal tells Python to start by writing to a file. Python checks it every second, and when it’s done, it writes to another file, which Unreal checks too.
📝 AssemblyAI transcribes player dialogue, while ElevenLabs generates voice dialogue. Unreal imports the audio into the game using a plugin called 🏃Runtime Audio Importer, which is then used by the MetaHuman SDK to create facial animations for the characters.
🤘 Remember the action value we parsed? We will save that to a text file, Unreal Engine accesses this file, and depending on the action it finds, it selects the right branch, plays animations, and makes environment changes accordingly.
✨ The action can range from a basic backflip animation to something more intricate, such as spawning items like flowers or magic rings, or even creating AI-generated music using MusicGen and playing it in the level.
⏰ High latency can be improved with audio and facial animation streaming. In the demo characters can perform up to 2 actions but you can add more as long as you have the animations, which I got from Mixamo. Each action leads to a specific branch based on the choice made by the LLM.
🗃️ The GitHub Repository will have the Python files, Unreal Engine blueprints, C++ code, and links to 3D assets and tools used in the game. If you find it helpful, consider giving it a star!
❤️ Thanks
If you found this interesting check out Alystria AI for more fun projects
https://www.linkedin.com/company/alystria-ai- 🌐 Github: https://github.com/AkshitIreddy
- 🐤 Twitter: https://twitter.com/Akshit2089
- 💡 LinkedIn: https://www.linkedin.com/in/akshit-ireddy
- ✍️ Medium: https://medium.com/@akshit.r.ireddy
