Part 3 (Insights): How to Harness Feedback Effectively

Welcome intelligentsias… we meet again! 🥳

In Part 2, we touched on the connectors required to gather implicit and explicit feedback, as well as additional parameters that help guide our Insights.

Today, let’s tie everything together and learn how to harness feedback effectively via our Insights!

We believe the most effective way for Ai teams to harness feedback is… well, like this:

We handle all of the hard work in order to provide digestible information that removes the need for any mental effort or manual work.

Here’s how…

In order of operation, for each feedback gathered (implicit or explicit), we conduct:

1. Issue Clustering — cluster the feedback into a respective issue type, ie:
   Insufficient knowledge, misunderstanding user intent, ambiguous response, and more!
2. Sentiment Analysis — identify relevance, conciseness, completeness, emotion, etc
3. Issue Prioritization — generate an impact score, a mechanism to measure issue severity. Calculated from the number of impacted users, (1) Issue Clustering, (2) Sentiment Analysis, and more!
4. 🔑Root Cause Analysis (RCA) — understand the issue to its core:
   What went wrong, where it went wrong, why it went wrong, and lastly, how to resolve it
5. Custom Evaluation Benchmarks — personalized evaluation of the product and underlying model (RAG, fine-tuned, and/or prompt engineered), as it relates to user expectations and experiences.

Next up, *drum roll* 🥁…

The series finale! In Part 4, we will understand how to utilize feedback and insights to make LLM products more reliable via our Resolutions.

– –

🔭 Stay tuned there’s more coming! On the way are demos, case studies, and our Open-Core so any builder can leverage this magic for free!

Co-author: movchinar