New Approach to Sentimentality with Gen AI: Review Scoring
Introduction
Online reviews have become a cornerstone of consumer decision-making, with a significant portion of internet users contributing to this ecosystem each month (GWI). While star ratings offer a glimpse into customer sentiment, the nuances of human language often elude traditional sentiment analysis methods, leaving businesses with a binary understanding of satisfaction — positive or negative, black or white, 1 or 0. Recognizing this limitation, we at Karaca sought to harness the capabilities of Generative AI (Gen AI) models to introduce a more nuanced approach to sentiment analysis in online reviews.
Furthermore, an estimation by Bain & Company suggests that approximately 80% of companies utilize sentiment analysis to measure customer satisfaction, underscoring the significance of this approach in contemporary business strategies.
Method
Our journey began with exploratory experiments, prompting a Generative AI model, Gemini Pro, to assign themes to unstructured review comments. Instead of a singular theme, Gemini Pro returned multiple themes, reflecting the complexity inherent in human expression. Through iterative refinement, we identified eight key categories crucial for our Sales, Product Development, Customer Experience, and Product Marketing teams: quality, durability, performance, ease of use, safety, shipping, customer service, and price. These categories were further classified into three tiers: “good,” “fair,” and “bad,” facilitating a more granular assessment of sentiment.
To handle the sheer volume of reviews, we automated the process using the model API, employing loops and exception handling to ensure smooth execution. Leveraging Google Cloud Platform (GCP), we established a pipeline to continuously map new comments, updating our scoring system dynamically. Categories were assigned numerical values — 0 for “poor,” 0.5 for “fair,” and 1 for “good” — enabling the calculation of average scores for each evaluation category.
Results
Visualizing the outcomes enabled our business teams to delve deeper into the nuances of customer feedback. Unlike traditional sentiment analysis, our method allowed for a more nuanced understanding of product strengths and weaknesses. By correlating review scores with digital performance metrics, we observed a compelling association, indicating that negative product feedback can impact overall conversion rates. Armed with this insight, proactive business teams can influence long-term conversion rates by addressing underlying issues highlighted in reviews.
Conclusion
Our approach to sentiment analysis represents a departure from binary assessments, offering businesses a more detailed understanding of customer sentiment. By harnessing the capabilities of Generative AI models, we were able to decode the subtleties of human expression, enabling more precise insights into product performance and customer satisfaction. Moreover, the dynamic nature of our architecture ensures adaptability in an ever-evolving landscape, where Generative AI models continue to refine their accuracy and precision over time with minimal intervention.
