Week 2— Toxic Overflow: Detecting Hateful Comments
Our Goal: Can we automatically detect the toxicity on the web for good?
Our Previous Blog : Week 1 — Toxic Overflow: Detecting Hateful Comments.
We are Nur Bengisu Çam, Furkan Çağlayan & Ahmet Burak Kahraman. For our semester project in Machine Learning we are trying to detect potentially hurtful & toxic comments.
This week we propose you the related works in the literature. It’s always a good idea to do some research so that you will be much more familiar with the topic. Keep on reading our weekly blog, we’ll update you with the progress.
As you can see we mentioned about the possible methods in the literature. Now let’s explain them detaily!
Text classification has some difficulties such as understanding the topic from the context with a limited key words and high dimensionality of the datasets. Due to this high dimensionality ,the importance of the information of each feature reduces, so it becomes harder to predict the true class. Even the feature reduction which is known as Dimensionality Reduction has its own challenges. So this is not an easy peasy task! In the literature, there are lots of different kind of classifiers applied to achieve success on topic. These are Naive Bayes classifier, Decision Trees, Nearest Neighbour classifier, Linear Discriminant Analysis (LDA), Logistic Regression and Deep Neural Networks.
Toxic Comment Classification has its own difficulties such as long-range dependencies or misspelled and kook words.There are mainly 2 common datasets about this topic and most of the researches are made upon them.
The first dataset is comments on Wikipedia talk pages which is represented by Google Jigsaw during the Kaggle’s Toxic Comment Classification Challenge . this dataset includes 223,549 user comments collected from Wikipedia talk pages. It is the largest available dataset on this topic. The class labels in this dataset are ‘toxic’, ‘severe toxic, ‘insult’, ‘threat’, ‘obscene’ and ‘identity hate’. So any comment in the dataset can be classified as multiple classes so it makes it an multi-label classification problem.
From the Table 1, you can see that the Wikipedia Talk Pages dataset is unbalanced because the number of samples are different for each of the classes. This leads to additional difficulties to the problem that is because most of the ML techniques are created for balanced datasets. Further information can be found here.
Other datsset is a Twitter Dataset presented by Davidson et al.(2017). This one contains 24,783 Tweets from the Twitter API and labelled by CrowdFlower workers. The labels are‘hate speech’, ‘offensive’ or ‘clean’. So the comments cannot be labeled as both of these classes. As you can see from the Table 2, this dataset is also unbalanced. Let’s see the results of the articles in the literature.
Article #1: Classification of text documents
Text classification become more popular with growth of World wide web(web).We need to classify text documents to sort text based information. In this paper (Classification of Text Documents Y. H. LI & A. K. JAIN 1998) researchers compared 4 different classification algorithm Naive Bayes classifier, Nearest neighbour classifier, Decision tree classifier and Subspace method.
From the figure below(we can see accuracy of different classifier algorithms given paramater feature count.
Let’s go over all used algorithms here briefly.
Naive Bayes Classifier : The naive Bayes classifier is an overall successful text classification method. That uses Bayes Theorem and it assumes that all features are independent, this is not true in real-world but it works well on practice.
Nearest Neighbour Classifier: NN for short is an algorithm that calculates similarities between all other class using cosine similarity function.
Decision Tree: Decision trees get built upon selecting the highest impact feature on each tree-level with a given ruleset.
Subspace Method: Subspace Method splits given features into m subgroups where every region still corresponds to classes. In the end, researchers decided to use a combination of multiple classifiers(CMC) using 3 different methods.
Simple Voting: Get the majority vote of the classifiers we mentioned above.
Dynamic classifier selection: Similar to the k-nearest neighbour method we get the values from classifiers and use “leave-one-out ” method to training data optimize the result to get the highest local accuracy classifier’s result.
Adaptive classifier combination: Instead of getting the highest local accuracy classifier first we denote the unlabeled data using KNN algorithm for each class calculate the a posteriori probability. Local accuracy can be obtained using dynamic classifier selection method above.
Keep on reading, things will become more exciting!
Article #2: Challenges for Toxic Comment Classification: An In-Depth Error Analysis n-Depth Error Analysis
In this article both of the datasets mentioned earlier is used. They found common problems between those datasets such as Out-of-Vocabulary Words, Long-Range Dependencies and Multi-word Phrases. Out-of-Vocabulary Words means there are different words that are usually misspelled or slag words that do not occur in the vocabulary. Long-Range Dependencies means that the toxicity of the document depends on the earlier parts of the text but if the document is long then the earlier parts vanishes. Multi-Word Phareses means that toxicity of the document can depend on the phares that consists of more than one word.
Their methods are Logistic Regression, Recurrent Neural Networks, Convolutional Neural Networks, (Sub-)Word Embeddings. You can read the details of these model from the article’s itself. Let’s see the results of these models now!
As you can see from the Table 3, they used precision, recall, F1-measure, and ROC AUC scores of the models on two datasets.
False negative results of the experiment with Wikipedia dataset is 23% and with the Twitter dataset is 9%. There are some comments that do not include swear words but still toxic. Here is an example comment: “she looks like a horse”. Or in some comments they caused toxicity with rhetorical questions such as example is “have you no brain?!?!”. Or sometimes toxicity can be made with the irony, for example “hope you’re proud of yourself. Another milestone in idiocy.” These kind of comments labeled as toxic or one the the classes that causes toxicity because these sentences are hard to classify since they need too much semantic information which is the hardest thing in NLP.
False positive results of the experimentwith Wikipedia dataset is 53% and with the Twitter dataset is 10%. In some comments, there are swear words that does not hurt anyone because of the meaning and usage. But the models usally indicate swear words as a highly toxic. Such an example is “Oh, I feel like such an asshole now. Sorry, bud.” Some comments refer to some other things which may include swear words but do not cause toxicity. Such an example is “I deleted the Jews are dumb comment.” These kind of comments labeled as toxit but they are actually not.
Article #3: The effect of Ensemble Learning Models on Turkish Text Classification
In this article Kilinc tried to classify texts from TTC-3600(link) Turkish news dataset. Kilinc used ensemble learning methods with 4 different Machine Learning classifiers. Naive Bayes classifier, Support Vector Machines, K-NN and Decision Trees(J48). TTC-3600 Dataset consist news from 6 news portals and agencies
“TTC-3600 Dataset consist news from 6 news portals and agencies.”
But, what is Ensemble Learning ?
Ensemble learning is the method of using more than one default classification models in the learning process. There are various ensemble learning algorithms such as Bagging, Boosting and Rotation Forest which are used in this article.
Most common Ensemble Learning Algorithms :
Bagging: Parallel learning process. Each default classifier is trained with different samples.
Boosting: Sequential learning process. Each model will choose what features the next model will focus on
Results from the article show that rotation forest and boosting methods gives the best accuracies for the dataset. Without the ensemble learning
SVM among the default models gives the best accuracy with 82.38%.
Thanks for reading, see y’all next week! Don’t forget to clap. :)
