AI Chat Logs — The Hidden Goldmine Your Company Hasn’t Discovered Yet.
Most of the focus of Generative AI like ChatGPT has been around putting prompts in ChatGPT and getting responses.
Most knowledge workers in companies are pretty thrilled that they can go to ChatGPT, put in some prompts, and get responses that save them time.
This is clearly an easy, low hanging fruit for a lot of knowledge workers.
But that is just the start ..
Improving personal productivity is always a good thing for individual growth and for making knowledge workers more efficient.
The more advanced companies and early adopters have taken this a lot further by creating custom GPTs with their own data and knowledge.
As more people get accustomed to using ChatGPT, companies will wake up to the fact that they can quickly and easily create domain-specific versions of ChatGPT with their own data (In technical speak, this is called RAG — or Retrieval Augmented Generation)
These days, you can build a custom GPT with your own data for just a few hundred dollars, in a few hours. With easy no-code systems, you don’t even need programmers or IT people to do this.
The smarter companies have already started deploying company-specific chatbots for their customer support. These chatbots, usually powered by ChatGPT, are orders of magnitude better than traditional chatbots and are trained on every aspect of the company’s knowledge and information. You can see an example here.
These chatbots, when deployed for customer support, are used extensively by employees and prospective customers when deployed on the website or on the help desk.
Case studies are emerging that show tremendous value and return-on-investment. The big factors are:
- Reduction in ticket volume (called “ticket deflection”)
- Quicker ticket resolution times
- Overall improvement in customer satisfaction.
These case studies are already showing that customer service is the lowest hanging fruit when it comes to deploying generative AI in businesses. The deployment of GenAI for customer support is well underway!
But that is not the focus of this blog post. The focus of this blog post is around the goldmine of information that you get from the chat logs.
Once deployed, your employees and prospective customers are telling you exactly what they want, what their roadblocks are, and other customer insights. This information is invaluable to literally every stakeholder in your company.
What Are Chat Logs?
Chat logs are the log entries of the prompts the user is asking the AI AND the responses from the AI.
When you deploy a chatbot to your employees or customers, these conversation logs contain key information that is of interest to every department in your company, from customer support, to product development, to R&D, to executive management, and even to legal & compliance.
So what changed in 2023?
Now you might be saying to yourself : Chat logs have been around for a long time — so what changed in 2023, dude?
There were two main changes in consumer behavior:
Longer Conversations
Previously, chatting with chabots used to be very painful. The average American hated chatbots and usually conversed in tiny 4–5 word sentences with the chatbot.
With the release of ChatGPT, the consumer behavior is changing and they are typing in larger sentences (even essays) and having longer conversations with ChatGPT-style chatbots.
Judgment Free Zone
This one is new — people are saying things to the AI that they would never say to a human.
AI is a judgment-free zone
And due to this, the chat logs are showing raw, unbiased and unfiltered interactions of employees and customers with the AI.
These raw and unfiltered interactions usually bring out the truth — a perfect scenario for true unbiased feedback. Customer support managers and product managers love this!
How Does GenAI Change Chat Log Analysis?
Previously, analyzing these chat logs used to be a real pain because the chatbots were very rudimentary.
In addition, the technology to analyze the chat logs and glean valuable information from it was very limited. You needed data scientists or statisticians to enter the fray and come up with sophisticated insights reports.
However, now with the emergence of LLMs and the power of GenAI, it has become much, much easier to do advanced analysis on the chat conversations.
As a simple example, doing sentiment analysis would take days or weeks in 2022. However, now with ChatGPT, doing sentiment analysis is just an easy English prompt away.
Who Can Benefit From This Goldmine?
Let’s take a look at what each department in your company can gain from an advanced analysis of the conversation logs.
1. Customer Support.
This is the lowest hanging fruit. If the chatbot is deployed for customer support and is used by employees and customers, the analysis can uncover a host of insights:
- Primary issues that users are having
- Feedback that the customers are giving
- Top content that is missing from the help desk
- Key points of frustration
These reports can be easily calculated dynamically using GenAI and LLM analysis on the conversation logs.
2. Product Development.
By looking at the conversation log, product managers can easily see what is missing from the product feature set.
Your customers will tell you what they want. (Again remember: It’s a judgment-free zone!)
And the conversation logs are the first point in getting massive amounts of information regarding missing features and issues with the product.
As an example (shown above), we’ve now had a company-specific chatbot deployed on our website for months now. The analysis of the conversation log guides the product development on the missing features that are important to our customers.
When we see trends emerging around missing features, those features get prioritized in our product development roadmap.
In addition, if we launched a new product or feature and customers are having problems with it, those signals quickly emerge in the chat logs.
3. Sales
So this is the cool part. Initially, we thought that chatbots are mostly for customer support.
But then we realized that chatbots — and the conversations logs in them — are also an excellent indicator of sales information and a driver of sales leads.
For example, the conversation logs will give you key insights into pricing and any obstacles/blockers that customers are facing in the buying process.
It will indicate to salespeople what it is that customers care about in their own words and how they should react to them. (Hint: this is great for sales training!)
4. Marketing
The conversation logs are a goldmine for marketing and marketing use cases.
The immediate impact: the logs will tell you what content is missing from your product marketing knowledge base.
It will tell you things that you should be talking about. A true “content gap analysis using chat logs”
By analyzing our conversation logs, we were able to come up with a content calendar of blog posts and FAQ questions that were missing from our website. This gives the marketing department an immediate action item around plugging this content gap.
These blog posts are now a driver of SEO traffic. And the additions to the FAQ, preempt tickets coming into the help desk.
5. Legal & Compliance
Right now there is a lot of FUD (fear, uncertainty and doubt) around AI. And some AI projects are being stonewalled on safety concerns.
The conversation logs AND the metrics around AI safety scores give AI champions the ammunition they need to allay the fears (usually raised by the lawyers and security officers!)
By analyzing the chat logs, you can calculate important safety scores like anti-hallucination score, racial bias score, diversity scores, and company-specific metrics that you would care about to mitigate any risks.
Using these scores and metrics, the FUDs of the chief security officer can be mitigated.
Please note: I said mitigated — NOT eliminated. That will never happen.
By having these scores and metric reports readily available, businesses can shoot for near-zero-risk deployments.
This is the dream scenario for most businesses. High ROI, with near-zero risk.
Conclusion.
While all the media attention that has been around AI prompting and AI responses, it is the humble conversation log that is the actual goldmine of customer intelligence and insights.
The conversation log provides a turbo boost to Generative AI deployments.
Simply replacing your old chatbot with a GenAI-based chatbot will itself provide fantastic ROI, no doubt.
But then when combined with this turbo boost and added benefit of conversation analytics and intelligence, the true power of generative AI can be put to work for your business.
