Building Hater News
15 min readOct 27, 2014
Hater News
Haterz gonna hate. But now you can know who the haterz are. I wanted to use data science / machine learning to identify and rank trolls by their “hater” level throughout the internet. I started with Hacker News ( check out Hater News ) and I wanted to explain the how and what I’ve done so far.
This post is long and detailed but I’ve tried to explain and post my code for how I’ve built out the Machine Learning aspects of this project using Python and Scikit Learn.
My Goal:
Look at any user from hacker news and be able to tell if they are a “hater” or not based on a score I give them. Then turn the whole thing into a web app so anyone can use it.
Feel Free To Look Under The Hood.
The Classifier Data I’m Using.
*Side Note: Eventually I would love to turn this into a Chrome App that will just real-time analyze any user on a page when you visit hacker news and put a score right next to them so the world can see if they hate or love. I’m also going to build versions of this for Twitter, reddit, Instagram, Facebook, and maybe even dating apps. If you want to help, reach out!
Lets Get Started!
We will begin by importing all of the required items we need from things like scikit learn, pandas, numpy, and even play around with things like nlk and re.
The basic idea will be to split out each comment, create features based on things like if a word shows up in a comment or not, if insulting words appear, and other types of features. Once I have that then I will add in a classifier and if it scores well we are on our way to finding haters!
In [1]:
# Importing my standard stuff
import pandas as pd
import numpy as np # Importing different Vectorizers to see what makes my soul sing the loudest.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer # Importing various Classifiers to see what wins out. There can only be one!
from sklearn.linear_model import Perceptron
from sklearn.svm import LinearSVC
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression # Snagging the validation stuff to see how well things do.
from sklearn.pipeline import Pipeline
from sklearn.cross_validation import cross_val_score # Getting Stopwords to potentially improve accuracy.
from nltk.corpus import stopwords
# Messing with tokenization via NLTK
from nltk import word_tokenize, pos_tag # Messing with Stemming
from nltk.stem import PorterStemmer
# Messing with Lemmatizing
from nltk.stem.wordnet import WordNetLemmatizer
# Messing with some other string splitting techniques. You can use the following thing like this: # re.split(‘; |, ‘,str)
import re
Exploring Different Ways To Achieve My Goal.
Here I am importing my data using pandas, pulling out the right features and assigning them to my X & y to put into my models. You’ll notice that I’m actually using both my test & training data in a single variable because of how cross_val_score works later on.
In [2]:
# Setting up my english stopwords, yo.
stopwords = stopwords.words(‘english’) # Here I combined my training & test data. I will used this as main total “training” data in production & for cross_val stuff.
corpus = pd.read_csv(‘data/total_data.csv’) # Setting Up my X & y
X_raw = corpus.Comment
y_raw = corpus.Insult # Setting Up My Different Pipelines. I tried out several different combos to see where to go next.
pipeline = Pipeline([ (‘vect’, TfidfVectorizer(binary=True, use_idf=True, stop_words=’english’)), (‘clf’, LogisticRegression()) ])
other_pipeline = Pipeline([ (‘vect’, CountVectorizer(binary=True)), (‘clf’, LogisticRegression()) ])
LSVC_pipeline = Pipeline([ (‘vect’, TfidfVectorizer(binary=True, use_idf=True, stop_words=’english’)), (‘clf’, LinearSVC()) ])
Perceptron_pipeline = Pipeline([ (‘vect’, TfidfVectorizer(binary=True, use_idf=True, stop_words=’english’)), (‘clf’, Perceptron()) ])
EVALUATION TIME!
Here I am looking at Accuracy, Precision, & Recall (learn more) and using the cross_val_score method which splits up my data, takes the pipelines I made, and then evaluates it over and over, in different combinations against itself to get a reliable score.
I want a good overall score but I would also like to improve my Recall as much as possible. Recall gives me a measurement of false negatives, meaning a score based on if I miss-classify something by saying a comment is NOT insulting when in fact it actually is. I am interested in a high score using this measurement to ensure no hater is left behind.
In [3]:
# THE INITIAL ‘STATNDARD’ MODEL I STARTED WITH
accuracy_scores = cross_val_score(pipeline, X_raw, y_raw, cv=5) precision_scores = cross_val_score(pipeline, X_raw, y_raw, cv=5, scoring=’precision’)
recall_scores = cross_val_score(pipeline, X_raw, y_raw, cv=5, scoring=’recall’) print “***************OUR RESULTS***************”
print “TfidfVect + Logistic Regression”
print “———————————————-”
print “Accuracy:”, accuracy_scores, “Mean:”, np.mean(accuracy_scores)
print “Precision:”, precision_scores, “Mean:”, np.mean(precision_scores)
print “Recall:”, recall_scores, “Mean:”, np.mean(recall_scores) # VS NORMAL COUNT VECTORIZER **NOTE** Out of all of the models without extra features, this performed the best for my goals.
other_accuracy_scores = cross_val_score(other_pipeline, X_raw, y_raw, cv=5)
other_precision_scores = cross_val_score(other_pipeline, X_raw, y_raw, cv=5, scoring=’precision’)
other_recall_scores = cross_val_score(other_pipeline, X_raw, y_raw, cv=5, scoring=’recall’)
print “CountVect + Logistic Regression”
print “———————————————-”
print “Accuracy:”, other_accuracy_scores, “Mean:”, np.mean(other_accuracy_scores)
print “Precision:”, other_precision_scores, “Mean:”, np.mean(other_precision_scores)
print “Recall:”, other_recall_scores, “Mean:”, np.mean(other_recall_scores) # VS NORMAL COUNT VECTORIZER
LSVC_accuracy_scores = cross_val_score(LSVC_pipeline, X_raw, y_raw, cv=5)
LSVC_precision_scores = cross_val_score(LSVC_pipeline, X_raw, y_raw, cv=5, scoring=’precision’)
LSVC_recall_scores = cross_val_score(LSVC_pipeline, X_raw, y_raw, cv=5, scoring=’recall’)
print “TfidfVect + LinearSVC”
print “———————————————-”
print “Accuracy:”, LSVC_accuracy_scores, “Mean:”, np.mean(LSVC_accuracy_scores)
print “Precision:”, LSVC_precision_scores, “Mean:”, np.mean(LSVC_precision_scores)
print “Recall:”, LSVC_recall_scores, “Mean:”, np.mean(LSVC_recall_scores) # VS NORMAL COUNT VECTORIZER
Perceptron_accuracy_scores = cross_val_score(Perceptron_pipeline, X_raw, y_raw, cv=5)
Perceptron_precision_scores = cross_val_score(Perceptron_pipeline, X_raw, y_raw, cv=5, scoring=’precision’)
Perceptron_recall_scores = cross_val_score(Perceptron_pipeline, X_raw, y_raw, cv=5, scoring=’recall’)
print “TfidfVect + Percptron”
print “———————————————-”
print “Accuracy:”, Perceptron_accuracy_scores, “Mean:”, np.mean(Perceptron_accuracy_scores)
print “Precision:”, Perceptron_precision_scores, “Mean:”, np.mean(Perceptron_precision_scores)
print “Recall:”, Perceptron_recall_scores, “Mean:”, np.mean(Perceptron_recall_scores)
return:
***************OUR RESULTS*************** TfidfVect + Logistic Regression ———————————————-
Accuracy: [ 0.81969697 0.81439394 0.81790592 0.81942337 0.81866464] Mean: 0.818016967858
Precision: [ 0.85806452 0.825 0.84615385 0.8313253 0.84713376] Mean: 0.84153548429
Recall: [ 0.38108883 0.3782235 0.37931034 0.39655172 0.38218391] Mean: 0.383471659586 CountVect + Logistic Regression ———————————————-
Accuracy: [ 0.83939394 0.84848485 0.84218513 0.84825493 0.84066768] Mean: 0.843797305375
Precision: [ 0.75464684 0.76895307 0.74137931 0.75694444 0.73793103] Mean: 0.751970939603
Recall: [ 0.58166189 0.61031519 0.61781609 0.62643678 0.61494253] Mean: 0.610234495933 TfidfVect + LinearSVC ———————————————-
Accuracy: [ 0.83181818 0.82878788 0.8323217 0.84066768 0.82776935] Mean: 0.832272957189
Precision: [ 0.74329502 0.743083 0.73605948 0.75367647 0.7122807 ] Mean: 0.737678935001
Recall: [ 0.55587393 0.53868195 0.56896552 0.58908046 0.58333333] Mean: 0.567187036854 TfidfVect + Percptron ———————————————-
Accuracy: [ 0.8 0.80606061 0.79742033 0.79817906 0.79362671]
Mean: 0.799057341242
Precision: [ 0.64214047 0.63716814 0.6231003 0.61452514 0.60795455] Mean: 0.624977719778
Recall: [ 0.55014327 0.61891117 0.58908046 0.63218391 0.61494253] Mean: 0.601052267562
Now That I Have A Baseline, Time to Pick A Model And Build It Out Further.
I ended it up picking CountVectorizer + Logistic Regression which you will see further down. In order to improve the score I started to play with extra features adding them manually and “hand making” them. These are almost meta data about each document that I was hoping would help give extra information to my classifier. They are things like How many “bad words” are used in a specific comment or the ratio of bad words used vs how many words total were in a comment. I even tried to look at things like if someone spoke in all CAPS or not to see if that could help predict if the comment is insulting or not.
In [4]:
# Setting up our actually train and test data sets
train_corpus = pd.read_csv(‘data/train.csv’)
test_corpus = pd.read_csv(‘data/test_with_solutions.csv’) # Getting our list of “badwords”
badwords = set(pd.read_csv(‘data/my_badwords.csv’).words)
# If you would like to see what some of these words look like, play with this:
print “Some Badwords We Will Check:”, list(badwords)[0:9]
print # Setting Up my X & y
# NOTE: If you want to use all the data (including the test data), uncomment out the other X & y trains below.
X_train = train_corpus.Comment
y_train = train_corpus.Insult
X_train = corpus.Comment
y_train = corpus.Insult
X_test = test_corpus.Comment
y_test = test_corpus.Insult # Just giving feedback so we know if we are using all of the data to train the model or not.
print “X_train’s number of instances:”,X_train.shape[0] print
if X_train.shape[0] > train_corpus.Comment.shape[0]:
print“************************************************************ *************************************************”
print “***JUST AN FYI, YOU’RE USING BOTH THE TRAINING & TESTING DATA FOR TRAINING. SHOULD BE FOR PRODUCTION ONLY.***”
print “*************************************************************************************************************”
else:
print “———————————————————————————————————”
print “You are using just the training data for training. Makes sense, right?”
print “———————————————————————————————————”
return:
Some Badwords We Will Check: [‘bugger’, ‘jizz’, ‘l3i+ch’, ‘f u c k e r’, ‘s hit’, ‘ma5terbate’, ‘bum’, ‘bimbo’, ‘pimpis’]
X_train’s number of instances: 6594 *************************************************************************************************************
***JUST AN FYI, YOU’RE USING BOTH THE TRAINING & TESTING DATA FOR TRAINING. SHOULD BE FOR PRODUCTION ONLY.*** *************************************************************************************************************Here I add in various extra features to help my classifier out. These are features that I had to manually create and add into my model so I didn’t use cross_val_score and had to evaluate them manually.
Features I added to improve the model are the following:
- badwords_count – A count of bad words used in each comment.
- n_words – A count of words used in each comment.
- allcaps – A count of capital letters in each comment.
- allcaps_ratio – A count of capital letters in each comment / the total words used in each comment.
- bad_ratio – A count of bad words used in each comment / the total words used in each comment.
- exclamation – A count of “!” used in each comment.
- addressing – A count of “@” symbols used in each comment.
- spaces – A count of spaces used in each comment.
In [5]:
# Since I was unhappy with aspects of my score, I added addtional features my classifier can use
# This is just a count of how many bad word
train_badwords_count = []
test_badwords_count = [] for el in X_train:
tokens = el.split(‘ ‘)
train_badwords_count.append(len([i for i in tokens if i.lower() in badwords])) for el in X_test:
tokens = el.split(‘ ‘)
test_badwords_count.append(len([i for i in tokens if i.lower() in badwords])) # **SHOUT OUT**
# I was messing with stuff from Andreas Mueller for these next features. Thanks man! ☺
# His Blog Post on this: http://blog.kaggle.com/2012/09/26/impermium-andreas-blog/
# **SHOUT OUT**
train_n_words = [len(c.split()) for c in X_train]
test_n_words = [len(c.split()) for c in X_test]
train_allcaps = [np.sum([w.isupper() for w in comment.split()]) for comment in X_train]
test_allcaps = [np.sum([w.isupper() for w in comment.split()]) for comment in X_test]
train_allcaps_ratio = np.array(train_allcaps) / np.array(train_n_words, dtype=np.float)
test_allcaps_ratio = np.array(test_allcaps) / np.array(test_n_words, dtype=np.float)
train_bad_ratio = np.array(train_badwords_count) / np.array(train_n_words, dtype=np.float)
test_bad_ratio = np.array(test_badwords_count) / np.array(test_n_words, dtype=np.float)
train_exclamation = [c.count(“!”) for c in X_train] test_exclamation = [c.count(“!”) for c in X_test]
train_addressing = [c.count(“@”) for c in X_train]
test_addressing = [c.count(“@”) for c in X_test]
train_spaces = [c.count(“ “) for c in X_train]
test_spaces = [c.count(“ “) for c in X_test] print “train_badwords count:”, len(train_badwords_count), “test_badwords count:”, len(test_badwords_count),
print “train_allcaps count:”, len(train_allcaps), “test_allcaps count:”, len(test_allcaps)
return:
train_badwords count: 6594 test_badwords count: 2647 train_allcaps count: 6594 test_allcaps count: 2647Time To Pick Our Model & Add In Our New Features.
Now that we have our new features we need to pick a vectorizer, smash the new features inside of our vectorized old features, and evaluated everything with our classfier.
In [6]:
# SIDE NOTE: TfidfVectorizer sucks for this type of problem. Well, you could use it and not use idf and it does get better but not too much…
# vect = TfidfVectorizer(binary=True, use_idf=True, stop_words=’english’)
vect = CountVectorizer(binary=True)
clf = LogisticRegression()I’m basically fiting and transforming our features via our vectorizer and then useing .reshape( ) & np.hstack( ) to manually add in our extra features.
In [7]:
X_train_transform = vect.fit_transform(X_train)
X_test_transform = vect.transform(X_test)
train_reshaped_badwords = np.array(train_badwords_count).reshape((len(train_badwords_count),1))
test_reshaped_badwords = np.array(test_badwords_count).reshape((len(test_badwords_count),1)) train_reshaped_n_words = np.array(train_n_words).reshape((len(train_badwords_count),1)) test_reshaped_n_words = np.array(test_n_words).reshape((len(test_badwords_count),1)) train_reshaped_allcaps = np.array(train_allcaps).reshape((len(train_badwords_count),1)) test_reshaped_allcaps = np.array(test_allcaps).reshape((len(test_badwords_count),1)) train_reshaped_allcaps_ratio = np.array(train_allcaps_ratio).reshape((len(train_badwords_count),1)) test_reshaped_allcaps_ratio = np.array(test_allcaps_ratio).reshape((len(test_badwords_count),1)) train_reshaped_bad_ratio = np.array(train_bad_ratio).reshape((len(train_badwords_count),1)) test_reshaped_bad_ratio = np.array(test_bad_ratio).reshape((len(test_badwords_count),1)) train_reshaped_exclamation = np.array(train_exclamation).reshape((len(train_badwords_count),1)) test_reshaped_exclamation = np.array(test_exclamation).reshape((len(test_badwords_count),1)) train_reshaped_addressing = np.array(train_addressing).reshape((len(train_badwords_count),1)) test_reshaped_addressing = np.array(test_addressing).reshape((len(test_badwords_count),1)) train_reshaped_spaces = np.array(train_spaces).reshape((len(train_badwords_count),1)) test_reshaped_spaces = np.array(test_spaces).reshape((len(test_badwords_count),1)) X_train_transform = np.hstack((X_train_transform.todense(), train_reshaped_badwords))
X_test_transform = np.hstack((X_test_transform.todense(), test_reshaped_badwords))
X_train_transform = np.hstack((X_train_transform, train_reshaped_n_words))
X_test_transform = np.hstack((X_test_transform, test_reshaped_n_words))
X_train_transform = np.hstack((X_train_transform, train_reshaped_allcaps))
X_test_transform = np.hstack((X_test_transform, test_reshaped_allcaps))
X_train_transform = np.hstack((X_train_transform, train_reshaped_allcaps_ratio))
X_test_transform = np.hstack((X_test_transform, test_reshaped_allcaps_ratio))
X_train_transform = np.hstack((X_train_transform, train_reshaped_bad_ratio))
X_test_transform = np.hstack((X_test_transform, test_reshaped_bad_ratio))
X_train_transform = np.hstack((X_train_transform, train_reshaped_exclamation))
X_test_transform = np.hstack((X_test_transform, test_reshaped_exclamation))
X_train_transform = np.hstack((X_train_transform, train_reshaped_addressing))
X_test_transform = np.hstack((X_test_transform, test_reshaped_addressing))
X_train_transform = np.hstack((X_train_transform, train_reshaped_spaces))
X_test_transform = np.hstack((X_test_transform, test_reshaped_spaces))
Scoring Our New Model.
Finally We get to see our new score. It seems to be doing a bit better but to start to look at recall and other measures we can set our predictions and then see how many items we missed and in what ways we missed them. Execute the final cells to see the scores.
In [8]:
clf.fit(X_train_transform,y_train)
predictions = clf.predict(X_test_transform) clf.score(X_test_transform, y_test)In [9]:
# Run this cell to get a print out of each of our wrong predcitions, and how they were predicted incorrectly. wrong_predictions_number = 0
false_negatives = 0
false_postitives = 0 for i,p in enumerate(predictions):
if p != y_test[i]:
wrong_predictions_number += 1
if p > y_test[i]:
false_postitives += 1
print “++FALSE POSITIVE++”
if p < y_test[i]:
false_negatives += 1
print “—FALSE NEGATIVE—”
print “predicted:”, p, “actual”, y_test[i]
return:
—FALSE NEGATIVE—
predicted: 0 actual 1
++FALSE POSITIVE++
predicted: 1 actual 0
—FALSE NEGATIVE—
predicted: 0 actual 1
++FALSE POSITIVE++
predicted: 1 actual 0
—FALSE NEGATIVE—
predicted: 0 actual 1
—FALSE NEGATIVE—
predicted: 0 actual 1
—FALSE NEGATIVE—
predicted: 0 actual 1
—FALSE NEGATIVE—
predicted: 0 actual 1
—FALSE NEGATIVE—
predicted: 0 actual 1
—FALSE NEGATIVE—
predicted: 0 actual 1 In [10]:
# Execute this cell to see a final scoring of our new model
print “total number of instances:”, len(predictions),
print “| total number wrong:”, wrong_predictions_number,
print “| total false negatives:”, false_negatives,
print “| total false positives:”, false_postitives
print “ THE FINAL STATS”
print “————————————————————”
print “Percent Right:”, 100 — (float(wrong_predictions_number) / float(len(predictions))) * 100,”%”
print “Percent Wrong:”, (float(wrong_predictions_number) / float(len(predictions))) * 100,”%”
print “Percent False Negatives:”, (float(false_negatives) / float(len(predictions))) * 100,”%”
print “Percent False Positives:”, (float(false_postitives) / float(len(predictions))) * 100,”%”total number of instances: 2647 | total number wrong: 67 | total false negatives: 54 | total false positives: 13 THE FINAL STATS
————————————————————
Percent Right: 97.4688326407 %
Percent Wrong: 2.53116735927 %
Percent False Negatives: 2.04004533434 %
Percent False Positives: 0.491122024934 %
In [11]:
# Uncomment this and execute to get a list of insults to look through for more potential patterns
# corpus.Comment[corpus.Insult == 1].to_csv(“insults.csv”)Now Time To Pull In Data From Hacker News.
Thanks Hacker News for your great API! Here I will be using urllib2 and json to pull in and organize the data I get back from HN.
In [12]:
# Setting up a connection to the internetz
from urllib2 import urlopen
# Getting things to come back
from json to a dict import json
# Making that json pretty if needed
from pprint import pprint
# Dealing with stupid Unicode issues
from django.utils.encoding import smart_str # Let’s make sure things work by getting a magical kitten from the interwebs
kittens = urlopen(‘http://placekitten.com/')
response = kittens.read()
kitty = response[559:1000] print “MY PRETTY KITTY. HIS NAME IS FRANK.”
print kitty
print # Now time to try with Hacker News
username = “testhater”
strt_item = “8478757"
url_usr_strt = “https://hacker-news.firebaseio.com/v0/user/"
url_itm_strt = “https://hacker-news.firebaseio.com/v0/item/"
url_end = “.json”
kevin = urlopen( url_usr_strt+username+url_end )
post = urlopen( url_itm_strt+strt_item+url_end )
kevin = json.loads(kevin.read())
post = json.loads(post.read()) print “MY HACKER NEWS ACCOUNT:”
print “———————————-”
pprint(kevin)
print “MY HACKER NEWS POST:”
print “———————————-”
pprint(post)
return:
MY PRETTY KITTY. HIS NAME IS FRANK.
__ __,
\,`~"~` /
.-=-. / . .\
/ .-. \ { = Y}=
(_/ \ \ \ /
\ \ _/`'`'`b
\ `.__.-'` \-._
| '.__ `'-;_
| _.' `'-.__)
\ ;_..-`' / // \
| / / | // |
\ \ \__) \ // /
\__)
MY HACKER NEWS ACCOUNT:
-----------------------
{u'created': 1413737356,
u'delay': 0,
u'id': u'testhater',
u'karma': 1,
u'submitted': [8478765, 8478763, 8478761, 8478759, 8478757]}
MY HACKER NEWS POST:
-----------------------
{u'by': u'testhater',
u'id': 8478757,
u'parent': 8478079,
u'text': u'nope. everything is terrible.',
u'time': 1413737378,
u'type': u'comment'}Building Out Some Functions to Ping The Hacker News API and Return Back Some Hate.
The three main ones I use are to get the comment ID’s accociated with a username, Get all the text from each of the comments and feed them into my model to get predictions, and finally run them through a function that gives me my “hater score”.
In [13]:
# Get all of a user’s comments
def get_user_comments(username):
comments = []
url_usr_strt = “https://hacker-news.firebaseio.com/v0/user/"
url_itm_strt = “https://hacker-news.firebaseio.com/v0/item/"
url_end = “.json”
user = urlopen( url_usr_strt+username+url_end )
user = json.loads( user.read() )
if len(user[‘submitted’]) > 100:
for c in user[‘submitted’][:101]:
item = urlopen( url_itm_strt+str(c)+url_end )
json_item = json.loads( item.read() )
if ‘text’ in json_item:
comments.append(smart_str(json_item[‘text’]))
return comments # Get all of an item’s comments
def get_item_comments(itemid):
comments = []
url_itm_strt = “https://hacker-news.firebaseio.com/v0/item/"
url_end = “.json”
item = urlopen( url_itm_strt+itemid+url_end )
item = json.loads( item.read() )
if ‘kids’ in item: for c in item[‘kids’]:
comment = urlopen( url_itm_strt+smart_str(c)+url_end )
json_comment = json.loads( comment.read() )
if ‘text’ in json_comment: comments.append(json_comment[‘text’])
if ‘kids’ in json_comment:
for c in json_comment[‘kids’]:
kcomment = urlopen( url_itm_strt+smart_str(c)+url_end )
kjson_comment = json.loads( kcomment.read() )
if ‘text’ in kjson_comment:
comments.append(kjson_comment[‘text’])
if ‘kids’ in kjson_comment:
for c in kjson_comment[‘kids’]:
k2comment = urlopen( url_itm_strt+smart_str(c)+url_end )
k2json_comment = json.loads( k2comment.read() )
if ‘text’ in k2json_comment:
comments.append(k2json_comment[‘text’])
if ‘kids’ in k2json_comment:
for c in k2json_comment[‘kids’]:
k3comment = urlopen( url_itm_strt+smart_str(c)+url_end )
k3json_comment = json.loads( k3comment.read() )
if ‘text’ in k2json_comment:
comments.append(k3json_comment[‘text’])
return comments # Get a final Hater Score. 100 is the worst, 0 is the best.
def calculate_score(predictions):
total_score = []
for s in predictions:
total_score.append(s[1])
return np.mean(total_score) * 100 # Get all a users comments and run them through my model
def user_score(username, my_vect, clf):
comments = filter(None, get_user_comments(username))
badwords = set(pd.read_csv(‘data/my_badwords.csv’).words)
badwords_count = []
for el in comments:
tokens = el.split(‘ ‘)
badwords_count.append(len([i for i in tokens if i.lower() in badwords]))
n_words = [len(c.split()) for c in comments]
allcaps = [np.sum([w.isupper() for w in comment.split()])
for comment in comments]
allcaps_ratio = np.array(allcaps) / np.array(n_words, dtype=np.float)
bad_ratio = np.array(badwords_count) / np.array(n_words, dtype=np.float)
exclamation = [c.count(“!”) for c in comments]
addressing = [c.count(“@”) for c in comments]
spaces = [c.count(“ “) for c in comments]
re_badwords = np.array(badwords_count).reshape((len(badwords_count),1))
re_n_words = np.array(n_words).reshape((len(badwords_count),1))
re_allcaps = np.array(allcaps).reshape((len(badwords_count),1))
re_allcaps_ratio = np.array(allcaps_ratio).reshape((len(badwords_count),1))
re_bad_ratio = np.array(bad_ratio).reshape((len(badwords_count),1))
re_exclamation = np.array(exclamation).reshape((len(badwords_count),1))
re_addressing = np.array(addressing).reshape((len(badwords_count),1))
re_spaces = np.array(spaces).reshape((len(badwords_count),1))
vect = my_vect.transform(comments)
features = np.hstack((vect.todense(), re_badwords))
features = np.hstack((features, re_n_words))
features = np.hstack((features, re_allcaps))
features = np.hstack((features, re_allcaps_ratio))
features = np.hstack((features, re_bad_ratio))
features = np.hstack((features, re_exclamation))
features = np.hstack((features, re_addressing))
features = np.hstack((features, re_spaces))
predictions = clf.predict_proba(features)
return calculate_score(predictions)
In [78]:
# Testing out my function. Put in a User Name here (example: ‘pg’, ‘vegabook’ or ‘KevinMcAlear’)
username = ‘CmonDev’# Here is there hater score! 100% is a pure hater.
print “The Hater Score of”,username,”:”,user_score(username, vect, clf),”%”
# Here is there most reacent comments maxed out at 100. user_comments = get_user_comments(username) for c in user_comments:
print c
print “——————————-”
return:
The Hater Score of CmonDev : 4.53430404659 %The damn thing hasn't even been released yet.
——————————-
"Unity brings the cutting edge graphics" — huh?<p>Great initiative though.
——————————-
"I've received nothing but positive feedback…"<p>People are generally nice because it is considered polite.
——————————-
More like: "What the fuck is wrong with this guy. He poses a complex ethical dilemma. It makes me think and thinking hurts my brain. I will better pretend he is some sort of creep."
——————————-
And why would an intelligent being with a free will still hang around?
——————————-
As calculated as the Nexus-muting "leak".
——————————-
We are at the high of the HTML5/JS hype cycle.
——————————-
If it's just an expression than maybe "forking" is also just an expression with a random meaning? Either that or he never was a BDFL.
——————————-
Doing Everything Manually To One Time To Be Safe.
Everything seemed to work great with my nice functions! But to be sure let’s manually do what my functions just did to be sure.
In [51]:
# Starting the transforming of my comments
test = vect.transform(user_comments)In [74]:
# Running through my feature manipulation manually
# *NOTE: This time I printed the features out each transformation I do to them so you can kind of see how they change.
p_badwords_count = [] for el in user_comments:
tokens = el.split(‘ ‘)
p_badwords_count.append(len([i for i in tokens if i.lower() in badwords])) p_n_words = [len(c.split()) for c in user_comments]
p_allcaps = [np.sum([w.isupper() for w in comment.split()]) for comment in user_comments]
p_allcaps_ratio = np.array(p_allcaps) / np.array(p_n_words, dtype=np.float)
p_bad_ratio = np.array(p_badwords_count) / np.array(p_n_words, dtype=np.float)
p_exclamation = [c.count(“!”) for c in user_comments]
p_addressing = [c.count(“@”) for c in user_comments]
p_spaces = [c.count(“ “) for c in user_comments]
p_re_badwords = np.array(p_badwords_count).reshape((len(p_badwords_count),1))
p_re_n_words = np.array(p_n_words).reshape((len(p_badwords_count),1))
p_re_allcaps = np.array(p_allcaps).reshape((len(p_badwords_count),1)) p_re_allcaps_ratio = np.array(p_allcaps_ratio).reshape((len(p_badwords_count),1))
p_re_bad_ratio = np.array(p_bad_ratio).reshape((len(p_badwords_count),1))
p_re_exclamation = np.array(p_exclamation).reshape((len(p_badwords_count),1))
p_re_addressing = np.array(p_addressing).reshape((len(p_badwords_count),1))
p_re_spaces = np.array(p_spaces).reshape((len(p_badwords_count),1)) # Adding In badwords
X_p_transform = np.hstack((test.todense(), p_re_badwords))
# Adding In n_words
X_p_transform = np.hstack((X_p_transform, p_re_n_words))
# Adding In allcaps
X_p_transform = np.hstack((X_p_transform, p_re_allcaps))
# Adding In allcaps_ratio
X_p_transform = np.hstack((X_p_transform, p_re_allcaps_ratio))
# Adding In bad_ratio
X_p_transform = np.hstack((X_p_transform, p_re_bad_ratio))
# Adding In exclamation
X_p_transform = np.hstack((X_p_transform, p_re_exclamation))
# Adding In addressing
X_p_transform = np.hstack((X_p_transform, p_re_addressing))
# Adding In spaces
X_p_transform = np.hstack((X_p_transform, p_re_spaces))
In [75]:
# Predicting the comments insult probaility
test_predictions = clf.predict_proba( np.nan_to_num(X_p_transform) )In [76]:
# Seeing what some predictions look like
test_predictions[:5]Out[76]:
array([[ 0.93827726, 0.06172274], [ 0.99678827, 0.00321173], [ 0.93641375, 0.06358625], [ 0.94203673, 0.05796327], [ 0.99731428, 0.00268572]])In [77]:
# Finally Evaulating the hater score manually.
print “The Probability of”,username,”being a hater is:”, calculate_score(test_predictions),”%”The Probability of pg being a hater is: 2.76497951931 %
Some Final Thoughts.
- Sarcasm is hard to detect.
- Using Pipelines when building my own meta-features is difficult.
- Most people on Hacker News are pretty nice people.
- I want to build in more training data from comments in Hacker News using the web app.
Well, that’s it for now. I hope this was informative. Feel free to reach out and ask me anything. And remember, all of my code for this project is in My GitHub Repo. I made a detailed README.md file so you can run the app locally and try it out for yourself. You can even mess with pickling, but that is for another day…
Originally published at kevinmcalear.com on October 27, 2014.
