Talk to GPT-4 Powered NPCs in your Favorite Games!
Interactive LLM Powered NPCs, is an open-source project that completely transforms your interaction with non-player characters (NPCs) in any game! With this project, you can engage in conversations with NPCs in any game using your microphone to speak.
The project uses sadtalker to synchronize character lip movements, facial recognition to identify different characters, vector stores to provide limitless memory capacity for NPCs, and pre-conversation files to shape the dialogue style of each character. By analyzing the specific NPC you’re engaging with, including their personality, knowledge, and communication style, the system adapts accordingly. Moreover, the NPCs are even capable of perceiving your facial expressions through your webcam, adding an additional layer of depth to the interactions.
This project targets previously released games like Cyberpunk 2077, Assassin’s Creed, GTA 5, and other popular open-world titles. These games, with their plentiful NPCs and beautiful environments, have long been yearning for a missing feature: the ability for players to engage in conversations with any NPC they want to talk to. With this project, we aim to fill that void and bring immersive dialogue adventures to these games, allowing players to unlock the untapped potential within these virtual worlds. The goal is to enhance the gaming experience of already-released titles that are unlikely to receive this feature from their original developers.
One of the remarkable aspects of this project is its versatility. You don’t need to modify the game’s source code or engage in complex modding procedures. Instead, it works its magic by replacing the facial pixels generated by the game, seamlessly integrating the facial animation into your gaming environment.
Whether you’re exploring ancient dungeons in Assassin’s Creed Valhalla or walking the neon lit streets of Cyberpunk 2077, Interactive LLM Powered NPCs takes your gaming experience to new heights. Prepare yourself for engaging, realistic, and meaningful interactions with NPCs that bring your virtual world to life.
Demo 🚀✨
GitHub Repo😄📜
How It Works 🤔💭
The project’s functionality is as fascinating as the conversations it enables. Here’s a breakdown of how Interactive LLM Powered NPCs works its magic:
🎙️ Microphone Input: Speak into your microphone, and your voice is converted to text.
👥 Facial Recognition: The system employs facial recognition technology to identify which NPC you’re interacting with, whether it’s a background character or a side character.
🔍 Character Identification: Based on the dialogue and character recognition, a unique personality and name are generated for background NPCs. For side characters, their specific character traits and knowledge are accessed.
🧠 LLM Integration: The transcribed text and character information are passed to a Large Language Model (LLM), which generates a response based on the given context. The LLM also utilizes vector stores containing character-specific and world information.
📁 Pre-Conversation Files: To ensure the NPCs speak authentically, the project utilizes a pre-conversation.json file containing the character’s iconic lines and lines that reflect their talking style. Random lines from this file are passed to the LLM while generating response, enhancing the generated dialogue.
🗣️ Facial Animation and Speech Generation: The LLM’s response is converted to speech using text-to-speech. Then a facial animation video is generated, using the extracted face image of the NPC with the audio.
🕹️ Integration with the Game: The facial animation video and audio are seamlessly integrated into the game by replacing the displayed face pixels with the generated facial animation, making it appear as if the NPC is speaking naturally.
😃 Emotion Recognition: Using your webcam, the system captures your facial emotions, allowing the NPCs to adjust their responses accordingly, creating a more personalized and immersive experience.
🐎 Non-Visual Interactions: The project goes beyond face-to-face conversations by enabling interactions even when the NPC’s face is not visible, such as during horseback riding or intense combat sequences. To engage with the NPC, simply say their name first, followed by your dialogue. The system will then follow the same process of transcribing, generating responses, and converting them into speech, providing a seamless and uninterrupted conversation experience, even in action-packed moments. So, whether you’re galloping through the countryside or battling formidable foes, you can still communicate with NPCs and enjoy the full range of interactive dialogue features.
🌐 Game Compatibility: Works seamlessly with any game, without the need for game modifications or altering the game’s source code.
Tools Used 🚀🔧
- 🍪 Cohere’s Language Models and Langchain for LLM Agent
- 🍩 SadTalker For Facial Animation
- 🍰 Edge-TTS for default voices
- 🧁 DeepFace for facial recognition, detection, gender, age, emotion detection
- 🍭 Chromadb for local vectorstores
- 🍬 SpeechRecognition package for speech recognition
❤️ Thanks
Thanks for reading, you can find me here:
🌐 Github: https://github.com/AkshitIreddy
💡 LinkedIn: https://www.linkedin.com/in/akshit-ireddy
✍️ Medium: https://medium.com/@akshit.r.ireddy
📚If you’re a beginner like me check out my silly blogs on Machine Learning, Deep Learning and Data Analysis
- A Silly, Fun-Filled Machine Learning Guide for Beginners
- A Silly, Fun-Filled Deep Learning Guide for Beginners
- A Silly, Fun-Filled Guide to Statistical Methods in Data Analysis for Beginners
