Introducing Toffee: A New Era in Conversational AI

VectorChat Team
4 min readJun 16, 2024


In the rapidly evolving realm of digital interactions, conversational AI stands out as a pivotal technology, enhancing user entertainment and productivity across various sectors. From personal assistants to fictional characters, the demand for truly natural, personalized, and immersive interactions is growing exponentially.

What is Toffee?

Toffee is a character-based conversational AI platform striving to create interactions that blur the line between AI and human, providing users with highly personalized and immersive experiences. At its core, the platform is free and user-centric, with a strong commitment to decentralizing all inference to maintain independence, minimize censorship, and maximize user autonomy.

The Magic of Candies

At the heart of the Toffee platform are “Candies” — user-created bites of knowledge that enrich characters with detailed information and immersive instructions on mannerisms and speech. These Candies lay the foundation for creating realistic characters, extensive knowledge bases, and effectively infinite, meaningful conversations.

Imagine seamlessly adding extensive JavaScript knowledge to your coding assistant, or enriching your Daenerys Targaryen character with the complete Game of Thrones lore. Candies support various media, including text, image, audio, video, and links to platforms like YouTube and Spotify. Users can easily create and share Candies, making the character creation process accessible, engaging, and community-oriented.

An example scene where a user personalizes a character with candy.

Customization is key in Toffee. Users can tailor characters with Candies from a vast library of community-created content or their own knowledge packs, even applying them to characters created by others. There is no limit to the amount of candies one can add to a character, allowing for incredible, unique experiences!

Endless, Immersive, Personalized Conversation

One of the most frustrating aspects of current conversational AI platforms is the limitation on context. Characters often forget messages after just a few interactions and lack detailed knowledge about the character they’re supposed to be. With minimal memory and knowledge, meaningful conversation and lifelike interaction are impossible.

Toffee addresses this with Retrieval-Augmented Generation (RAG), ensuring characters can intelligently access relevant past conversations and knowledge. Instead of forgetting messages, characters evolve alongside the user, facilitating meaningful, personalized interaction.

Boosting Productivity

A conversation where a user is preparing their message with files.

Toffee is not just for entertainment; it’s also a powerful tool for productive use. The platform supports multi-modality, allowing users to incorporate significant amounts of data into conversations. Users can upload text files, PDFs, and images during interactions, enhancing the context beyond what is provided through applied Candies.

Additionally, Toffee allows users to preserve and export valuable context accrued over the course of a conversation, allowing for the seamless transfer of rich context to other characters or additional instances of the same character.

For example, take writing, where the AI assistant gradually learns the user’s style, word choice, and some niche information about the topic. When a new instance of the AI is needed, users can add this rich context as a candy, avoiding the need to rebuild it from scratch.

Voice: The Next Frontier of Conversational Intelligence

A voice call between a user and a character; the character is currently responding.

Voice interaction is the next domain in conversational AI, and Toffee aims to be at the forefront of this evolution. The platform supports real-time, realistic audio interactions, allowing users to call characters just as they would a friend. High-fidelity voice interactions are crucial for creating truly immersive, natural experiences, therefore Toffee specializes in generating high quality voices even with limited data.

With just a five-second audio clip, users can create realistic voices, while a one-minute sample allows for even higher fidelity. These voices can be kept private or shared with the wider community, similar to characters and Candies.

Looking ahead, Toffee envisions setting new standards in conversational AI, with a commitment to continuous innovation and enhancing user experience. Whether for productivity or entertainment, Toffee aims to deliver an unparalleled user experience.