Natural language processing of customer reviews
Making sense of the unstructured review data with machine learning
Recently, I’ve stumbled upon an article published on the Harvard Business Review website about the problems with 5-stars rating systems. The article pointed out a few pressing issues but most importantly made me curious about the correlation between given rating and the content of the review itself.
I’ve decided to perform an experiment and see how a review text sentiment maps to points given and if there are any deviations from what we usually expect in these cases.
To be able to gauge the predominant sentiment of a review text, I’ve decided to employ Natural Language Processing. Essentially, Natural Language Processing (NLP) is the technology used to help computers understand the human’s natural language.
There is an enormous potential sitting in our unstructured data. Customer emails, support tickets, product reviews, social media … All of that can provide insights into customer sentiment that can be used for the benefit of our businesses. As it turns out, Machine learning is particularly good at accurately identifying specific items of interest inside vast swathes of text.
There are many NPL frameworks at our disposal. Probably most prominent is Google’s Tensorflow. The most known application of TensorFlow’s NLP is Google Translate.
To make things simple and fast, I’ve opted in for a managed service that AWS provides: AWS Comprehend.
Amazon Comprehend is a natural language processing (NLP) service that uses machine learning to find insights and relationships in text. No machine learning experience required.
To perform the experiment we need access to the reviews data. Luckily, I’ve found that TrustRuby Reviews has free API access available.
With a few simple GET request, I was able to grab hundreds of reviews of different business alongside their scores (for brevity I’m not going to show how to do it, but should be pretty straight forward).
I’ve transformed review JSON data into a simple CSV file.
Next, I’ve uploaded CSV file to previously created S3 bucket.
AWS Comprehend set up through the console is pretty straight forward. We have to pick the name, type of analysis and language text is in.
Next part is just defining the data source and output location.
Results are in
Sentiment analysis job was performed relatively quickly and results were stored in an S3 bucket inside the output directory.
As you can see from the results excerpt, AWS Comprehend service labeled every line with a predominant sentiment.
Before merging analysis results with our original review content to compare sentiment and points given, we need to establish expectations. Let’s say that one and two points should map to negative, three to neutral or mixed and four and five to positive sentiment.
So without further ado, I present you the results!
I was slightly surprised with some of the edge cases like 3.23% of people who gave 5 stars review left a review that was perceived by AWS service as negative.
What I also found interesting is that three points reviews are mostly with text recognized as negative.
It is also worth noting that for this analysis we used predominant sentiment classifications even though some of them were on the border. AWS Comprehend service gives confidence score for every sentiment.
"Text": "Cool online shop for super food 'shrooms. Really like the design and how easy it is to navigate. Just ordered lion's mane mushrooms elixir",
Regardless of edge cases, it is resoundingly clear that negative sentiment reviews are followed by one or two starts ratings.
Thanks for sticking around. Feel free the share you opinion about NLP and customer reviews in general in the comments. Cheers!