Using Natural Language Processing to Classify and Analyze User Feedback

Jayant Madugula
Nov 7, 2018 · 9 min read
Illustrations by Mengdi Zhang

Tools

For both parts, I used a series of natural language processing (NLP) techniques and machine learning models. I used Python 3 along with a number of libraries focused on NLP and machine learning to implement the project. I mainly depended on spaCy, textacy, sci-kit learn, pyLDAvis, and pandas.

Data

The project began with gathering data. Being an iOS intern, the Jet iOS app’s reviews were a natural first source. However, we didn’t have access to enough reviews, forcing me to look for additional data.

Preprocessing

Feature Extraction

  1. noun chunks
  2. subject-verb-object (SVO) triples
  3. key terms, via the SGRank algorithm

Filtering

The idea for filters came from an observation that many of the topics discussed in our reviews are expressed using a relatively small set of words.

  1. Count the score as a feature, then send all reviews to a machine learning model for classification.
Results from filtering for shipping reviews

Classification — Training and Testing Models

The classification step for my project was relatively straightforward. spaCy provides a TextCategorizer, which implements a Convolutional Neural Network (CNN) behind the scenes. The TextCategorizer was quite simple to implement and immediately gave solid results. Training on a sample of ~5k reviews, I got the following preliminary results (each model trained for 10 iterations):

  1. “pricing” label: precision value of 0.958 and F-score of 0.969.
  2. “shipping” label: precision value of 0.905 and F-score of 0.897.

Next Steps

Unfortunately, my summer ended before I could experiment with more combinations of filtering, features, and models, but the early results were promising. The results posted above are based on preliminary training runs on small subsets of the total available data, suggesting performance will only increase when the full datasets are used. From the tests I ran, I found it interesting that model and filter performance changed noticeably for different labels, suggesting a one size fits all approach is not ideal in this case. Instead, a pipeline composed of unique binary classifiers for each label could be the best option.

Jet Tech

Sharing our engineering org’s learnings & stories as we build the world’s best experience to shop curated brands and city essentials in one place.

152

152 claps
Jayant Madugula

Written by

Computer Science student at Columbia University

Jet Tech

Jet Tech

Sharing our engineering org’s learnings & stories as we build the world’s best experience to shop curated brands and city essentials in one place.