Part 1 (Overview): What is Feedback Intelligence?
In the past few years, we have built ML-powered products, optimizing well-known metrics on the test set. This is known as heuristic-based evaluation.
With the rise of LLM-powered products (conversational Ai, agents, etc), heuristic-based evaluation simply does not get the job done due to one missing variable… the user.
When it comes to evaluating the reliability of live LLM products — user interactions, experiences, and feedback are often overlooked and underutilized resources. These resources are referred to as “feedback.”
There are two types of feedback:
- Explicit Feedback — thumbs up/down, star rankings, raw text messages (via form, email, slack, zendesk, etc), etc.
- Implicit Feedback — user interactions (prompts), experiences, time metrics, pausing the generation before completion, copying, pasting, etc.
Explicit Feedback is scarce and unstructured… Implicit Feedback is abundant, unstructured, and noisy.
So, how do Ai teams (PMs and devs) currently harness the power of this feedback to make their LLM products better?
By working their a$$es off to set up manual pipelines and manual review processes!
We spoke to more than a hundred Ai teams… shockingly ⭐:
- No one can do this work efficiently
- No one enjoys this work
Because of these manual or non-existent pipelines, they fail to truly leverage this feedback in an automated way. Leading to unhappy users, customer churn, and wasting precious money on inefficient Ai team time.
Until now…
Introducing *drum roll* 🥁…
The all-in-one Feedback Intelligence solution for Ai teams — Build reliable LLM products faster. Automatically turn LLM user feedback into actionable insights to analyze root causes, prioritize and resolve issues.
- Part 1 (Overview): What is Feedback Intelligence
- Next up! Part 2 (Connectors): How to Cut Through the Feedback Noise
- More to come after!
Any and all comments or questions are welcome! DM us if you are interested in using the beta or contributing to our open-core.
Co-author: movchinar
