Product Analytics — Traditional vs AI Products
Ahh the good ‘ol days, when click-through rates, usage metrics, and conversion funnels were all a product team needed to thrive.
Traditional product analytics are the crutch all product teams lean on for optimization. Tracking everything from user actions (page views, clicks, heat maps, etc), conversion rates, retention, and churn to A/B testing, cohort analysis, and much much more.
Then… *dramatic pause*… everything changed
With the rise of LLM-powered products (conversational AI, agents, etc), we stepped into uncharted waters… Why? There are no longer pre-defined user journeys. Each journey generates unique responses and dynamic conversations.
This shift to product engagement makes it extremely difficult to use traditional product analytics tools to understand a user’s experience and expectations.
So, what can be done? There are two options:
- Define your own evaluation metrics and do continuous manual analysis. This is an excellent option for early-stage products, less so for scaling and mature products.
- Sit back, relax, and embrace Feedback Intelligence.
Despite Feedback Intelligence (FI) having arguably the longest name ever, it makes sense as an out-of-the-box concept. FI acts as the missing link between traditional analytics and the unpredictable world of LLM-powered products.
So, to best describe what Feedback Intelligence is:
It’s a solution designed to understand and analyze the unique interactions between users and LLM systems. By first evaluating high entropy LLM-powered products, it can then optimize those products.
As an example of that: FI allows Ai product teams to understand your user’s experiences and expectations (evaluate), and then personalize and improve those experiences to better align with expectations (optimize).
How does Feedback Intelligence work?
It interprets context, sentiment, and nuances in human-Ai interactions through:
- Sentiment analysis — identifies relevance, conciseness, completeness, and emotion in each user interaction.
- 🔑 Root Cause Analysis (RCA) — understands issues at their core: what went wrong, where it went wrong, why it went wrong, and how to resolve it.
- Customer evaluation benchmarks — creates personalized evaluation metrics for your product and underlying LLM-infrastructure, based on user expectations and experiences.
- Synthetic golden set generation — automatically generates high-quality datasets from your user engagement, ensuring personalization without the expense.
- Prompt optimization — automatically identifies pitfalls in current prompts and recommends personalized alternatives to better meet user expectations.
- Automated hyper-parameter testing — leverages 🔑 RCA to find and resolve underlying issues in the retrieval process, avoiding time intensive manual testing.
- Knowledge gap identification — leverages 🔑 RCA to find missing context and recommend information to add.
We are extremely bullish that in order to optimize solutions, you must first evaluate and understand your audience. Collecting high-quality personalized data is the difference between successful and unsuccessful Ai solutions.
😤 Enough said.
