Part 4 (Resolutions): How to Utilize Feedback and Insights to make LLM Products More Reliable

Intelligentsias… the finale is here! 🥳

In Part 3, we explored how to harness feedback effectively through issue clustering, prioritization, and 🔑 Root Cause Analysis via our Insights.

Now… let’s dive deeper into how to utilize feedback and insights to make LLM products more reliable via our Resolutions.

We set out to solve a single problem Ai teams face with their products: to understand if user expectations were met, and if not, what to do about it.

Our journey to solve this problem is similar to the movie The Goonies. 🏴‍☠️ 🔱

How? At the end we found the treasure…

Otherwise known as, a solution that all Ai teams find value from. *That was corny… I know*

So, after the feedback data has been wrangled together and the insights have been shown… we finally get to the question of “what actions can we take?”

The answer: our Resolutions

We offer different actions that fit infrastructures of all shapes and sizes. In no specific order, here are the current Resolutions:

• Generate a synthetic golden set: no more expensive and time-consuming labeling… synthetically generate a high-quality dataset from real human usage that ensures appropriate benchmarking, quality control, cost-effective evaluation, and more reliable accuracy and precision.
• Recommend prompts: spend less time with trial-and-error… identify the pitfalls of current prompts along with recommended prompts optimized to meet user expectations.
• Automate hyper-parameter testing: avoid time-intensive trial-and-error testing… find and resolve the underlying issue within the retrieval process through 🔑 RCA and automated hyper-parameter tweaking.
• Fill knowledge holes: eliminate the manual process of finding missing context… identify what is missing through 🔑 RCA along with recommendations on what to add to the knowledge base.

Although this series has come to an end, there is much much more to come! Everything from analytics to custom rules to new user-facing interfaces.

The possibilities are endless! 🚀

Reach out if you’re interested in learning more or collaborating!

On the way are demos, case studies, and our Open-Core so any builder can leverage this magic for free!

Co-author: movchinar