Build a Recommendation System Using Amazon Review Data

Find your desired product via NLP and Sentiment Analysis

Aaron Zhu
Aaron Zhu
Jun 11 · 7 min read
How to choose the best child car seat?

Check out the Web App and the code on my GitHub and feel free to let me know if you have any questions!

Project Background

Do you the same experience as I do when you are shopping online? You search a product name in an E-Commerce website. Hundreds of products might show up. You sort products by rating or price, go through these similar products one by one, scan other customers reviews and try to find if other customers had given reviews on the product features you are interested in. This process is very time-consuming and could quickly get overwhelming. It seems like shopping online doesn’t make our lives easier.

As an average shopper like most people, I would like to create an app to quickly find the desired product when choosing from numerous products from an E-Commerce Website.

A Recommendation System is a good start to tackle this challenge. But there are some issues developers might face when developing a Recommendation System for E-Commerce Websites.

  • Cold Start problem in Collaborative Filtering method due to the lack of shoppers’ historical sales data

In this project, I want to create a recommendation system in a different approach. It takes customers product reviews as inputs, which are available nowadays in many E-Commerce Websites, such as, amazon.com and implement Sentimental Analysis on features that customers desire.

This approach is different from average customer rating we find on E-Commerce websites because this approach is based on feature-level ratings. All customers need to do when searching a product is providing product features they are interested in. Then the recommendation system would automatically returns products with highest feature ratings based on other customers reviews. This approach is very similar to what an average shopper would do manually, except this recommendation system can return your desired products in a second.

Scrape Amazon Review Data with Selenium and Beautiful Soup

Scraping Amazon product reviews are straightforward using Selenium and Beautiful Soup.

Selenium is a framework that controls the browser interactions programmatically such as clicks, form submissions and mouse movement. In addition, selenium makes it easier to handle a webpage, in which the content is generated by JavaScript.

BeautifulSoup is a Python library for parsing HTML, XML and other markup languages. It creates a parse tree that can be used to extract data from HTML. If a webpage displays data relevant to your project, but it doesn’t provide a direct way of downloading the data. BeautifulSoup can help you parse specific content from a webpage, remove the HTML markup and scrape the information.

Take Amazon webpage for example, in a given review page, we can use “find” function in BeautifulSoup to extract data (such as, Review Title, Review Date and Review Content) using HTML tag elements.

# rating 
rating = i.find('i', {'data-hook': 'review-star-rating'}).get_text()
# title
title = i.find('a', {'data-hook': 'review-title'}).get_text()
# date
r_date = i.find('span', {'data-hook': 'review-date'}).get_text()
# review
review = i.find('span', {'data-hook': 'review-body'}).get_text()

We can navigate in a webpage and go to other webpages using Selenium functions, “find_element_by_xpath” and “click”.

driver.find_element_by_xpath('//a[@data-hook="see-all-reviews-link-foot"]').click()
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//*[contains(text(), 'Next page')]"))).click()

Pre-Process Review Data Using NLP

After scraping the review data from amazon.com, we need to pre-process text data by tokenizing, removing stop words, stemming, converting numbers to text and removing punctuation. Python library, “NLTK” provides many functionalities for Text Preprocessing tasks.

Filter Relevant Reviews via TF-IDF and Cosine Similarity

In the app, I created several pre-defined features that are important to products. When parents are shopping for a baby cart seat, there are some key features they are looking for, such as, Lightweight, comfortable, safety, easy-to-install, easy-to-clean, newborn-ready, etc.

Not all reviews would mention these product features. Some reviews have nothing to do with features we’re interested in. To filter reviews that are relevant to pre-defined features, I used techniques, such as, TF-IDF and Cosine Similarity to compute relevant scores.

TF-IDF stands for “Term Frequency — Inverse Document Frequency”. It is a popular method to vectorize the text data in the form of numerical value so that the computer can understand the text. Unlike Bag-Words, which simply creates vectors containing the count of word occurrences in the documents, TF-IDF increases the weight of a word proportionally to the number of times it appears in a document, but offsets the weight by the number of documents that contain the word. For example, the words that are common in every document, but don’t provide relevant information (such as, this, what, how and if) would rank lower.

Compute Cosine Similarity

Once the review data are converted into number values, I can compute the similarity score between each product review and pre-defined product features using following mathematical formula. Cosine Similarity score would range from 0 to 1. A cosine value of 0 means two vectors are at 90 degrees to each other, meaning these two vectors are not related to each other. The closer the cosine value to 1, the more relevant a product review to the pre-defined features.

I used a threshold (0.1) for cosine similarity to filter reviews that are relevant to the pre-defined features. This reduced the review data we will implement sentiment analysis on from 100K to 10K, so that the run time is reduced dramatically.

Obtain Feature Rating Through Sentiment Analysis

Even though overall sentiment score for a product review is useful, I went a step further to compute the sentiment score for specific features of a product.

For example, in follow customer reviews, if a customer are interested in whether a car seat is easy to install and adjust. The word, “easy ” in the following review would capture the sentiment for “install” and “adjust”, which is the essence of this recommendation system.

Review 1: Easy to install, easy to adjust incline, 4 year old- almost 5, is comfortable.
Review 2: Installation is difficult. Not easy to adjust to the seat.

To achieve this, I made an assumption that a customer will express their sentiment about a feature within a window of words around the feature word. So I created a windows of 3 words to both the left and right of the feature word and feed this window of words into a sentiment analyzer.

So for each review, I will detect if it contains feature keywords, if it does, a sentiment score will be produced associated with s specific feature.

In the process of computing sentiment score, I used a Python package, “Swifter” to speed up Pandas processing. Swifter can help to execute many different codes repeatedly through parallel processing.

In the end, at product level, I computed average sentiment score for each feature. The final data output is very lightweight as there is only one record per product. So the front-end app will be very fast to run because complex end-back computations can be implemented beforehand.

When a customer choose the features they are interested in, the recommendation system would return top products with highest sentiment scores based on sum of sentiment scores across all selected features.

Store Data in Google BigQuery and Create a Dash App using GitHub and Heroku

Once data are cleaned and pre-processed, data will be stored in Google BigQuery so that my application can access to the back-end data.

A web application is created using Dash and deployed using GitHub and Heroku. If you’re interested how to set it up. See more details in my previous article.

Final Note

  • Instead of computing simple average sentiment score for each feature at product level, I can try to come up with a metric to capture the credibility of the reviewer and use this metric as weight to compute aggregate sentiment score.

Thanks for reading!!!

MLearning.ai

Data Scientists must think like an artist when finding a solution

MLearning.ai

Data Scientists must think like an artist when finding a solution, when creating a piece of code.Artists enjoy working on interesting problems, even if there is no obvious answer.

Aaron Zhu

Written by

Aaron Zhu

Data Science | Machine Learning | Economics Consulting https://www.linkedin.com/in/aaron-zhu-53105765/

MLearning.ai

Data Scientists must think like an artist when finding a solution, when creating a piece of code.Artists enjoy working on interesting problems, even if there is no obvious answer.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store