Feedback Collection Mechanisms for RAG and Prompt-Engineered Systems in Production

When running RAG and prompt-engineered (PE) systems in production, gathering feedback is key to keeping these solutions accurate and relevant. Feedback generally falls into two categories: explicit and implicit, both crucial for optimizing the system’s performance.

Explicit Feedback is straightforward — users directly tell you what they think. This can be through ratings, comments, or suggestions about the system’s outputs. Users might provide this feedback through in-app prompts like thumbs-up/down buttons or post-interaction surveys. But it doesn’t stop there. Explicit feedback can also come through other channels like email, Slack, Zendesk, or even phone calls to product owners. If users aren’t happy with a response, they’ll find a way to let you know or they will churn.

Implicit Feedback is more subtle, focusing on user behavior. This includes how long someone spends on a response, whether they scroll past it quickly, or how often they rephrase their query. While harder to interpret, implicit feedback can provide deep insights into the system’s effectiveness and how well it meets user needs.

Why Feedback Matters

Collecting feedback is crucial for several reasons:

1. Optimizing Accuracy: User feedback helps fine-tune prompts and retrieval methods as they have the most context on the task, ensuring the system continues to deliver relevant results.
2. Enhancing User Experience: Understanding user interactions allows for adjustments that make the tool more intuitive and effective.
3. Managing Risks: Feedback helps catch issues early, like when the system generates irrelevant or biased content.

How to Collect Feedback

Here are some effective ways to gather explicit feedback:

1. In-App Prompts: Simple thumbs-up/thumbs-down buttons or quick surveys embedded in the app make it easy for users to give feedback on the spot.
2. UI Widgets: Built-in widgets like star ratings or comment boxes enable detailed feedback when needed.
3. Email, Slack, Zendesk, and More: Users can share feedback through a variety of channels, from emailing support teams to messaging on Slack or filing a ticket in Zendesk. Even a direct call to the product owner can be a valuable source of explicit feedback. Of course, the latter is not the ideal …

Implicit feedback, though less direct, is equally important and can be the hardest but most rewarding to gather. Tracking user behaviors — like how they navigate the app or which responses they spend the most time on — can reveal what’s working well and what needs improvement, even without explicit complaints. There are some techniques that engineers can implement to gather implicit feedback:

1. Preprocess Chat History: Start by preprocessing your users’ chat history into an easy-to-work-with format, such as a CSV file or, for more professional and automated analysis, a database. Then, for each user request, select the next five consecutive requests, skip the following five, and repeat this pattern.
2. Determine a Similarity Threshold: Establish a threshold for similar requests. You can do this by manually finding examples where users ask the same question but phrase it differently or using GPT to generate such variations. Calculate a similarity score for each example and use these scores to determine an average threshold.
3. Analyze for Implicit Feedback: Apply the threshold to the selected requests from step one. Filter out similar requests, and if you find at least two to three similar requests, it likely indicates that the user was not satisfied with the initial response and is rephrasing their question to get a better answer.

Feedback Intelligence: A Holistic Approach

Feedback Intelligence brings everything together in one place. It doesn’t matter if the feedback comes from an in-app, an email, or even a phone call — it consolidates and organizes all explicit feedback so you can easily see what’s working, what’s not, and why the users are not happy. This is particularly useful in applications like conversational AI, chatbots, and agents where quick and clear feedback is essential for making improvements.

But here’s the thing: Most users — around 90% — won’t give you direct feedback. That’s where Feedback Intelligence really shines. It automatically tracks how users interact with your system, picking up on subtle behaviors like navigating, rephrasing queries, or responding to outputs. This implicit feedback is crucial because it gives you insights you might miss if you only rely on what users explicitly say. By combining both types of feedback, Feedback Intelligence enables Ai teams to continuously optimize the RAG and PE systems to align the output with users’ needs.

For more technical information, check out our product documentation.

co-authors: Erik Harutyunyan & Mels Hakobyan