Learning Emotions from Reddit
Introduction
Everyday, our inboxes are flooded with a ton of computer-generated messages, what we internally call ‘botmails’, letting us know about updates for services we use, purchases we have made, promotions trying to convince us to buy the latest and shiniest products, etc. However, many of us also use email to communicate with our family and friends as well as a our co-workers. This type of communication is what we call “conversational email.”
In conversational email, just as in any other type of human-to-human form of communication, emotion is an essential aspect that needs to be taken into account when trying to fully understand the meaning of human produced messages.
When communicating, humans not only exchange basic information or their needs and desires, but also their emotional state, reactions, and opinions towards entities and events.
But what is emotion? Emotion is a multifaceted concept very closely related to, and frequently confused with, mood and sentiment. Here’s a few technical definitions to help us narrow down the theoretical scope of the concepts we are dealing with:
- Mood is an “overall state of emotion […] sustained over longer periods of time” (Lin et al, 2012: 60).
- Sentiments (sometimes referred to as appraisals or opinions), unlike emotions and moods, “are not states of an individual, but assigned properties of an object.” (Brave, 2011: 56).
- Emotions are “evaluative judgments of the environment, the self and other social agents, in light of the agent’s goals and beliefs.” (Hudlicka, 2008: 53)
Even though we agree with the fact that emotion and sentiment/appraisal are two distinct concepts, we have decided for this project to model them jointly. As Brave (2011) states:
“When people say that they “like” an interface or find an interface to be “frustrating,” what they really mean is that that they associate the interface with a positive or frustrating emotional state; in other words, they expect interaction with the interface to lead to positive or frustrating emotions.” (Brave, Idem)
This means that when people make evaluative statements about particular objects and/or their properties, they are consciously or unconsciously stating that those objects and properties have the potential to evoke emotions for them or any other individual that shares the same needs, goals or concerns. From this point of view, appraisals are an indirect form of conveying emotion.
So now that we have narrowed down what emotion is, what different types of emotion are there?
Even though there are different theories that try to state which are the most basic emotions for human beings, we have chosen to align with the concise and most widely accepted set of six basic emotions enumerated and defined by Paul Ekman (1992): joy, sadness, anger, fear, disgust and surprise. Sound familiar?
Emotion Corpus
To train our emotion classifier we used one of the earlier, smaller versions of the Reddit corpus. This is a diverse topic-focused informal style corpus of online comments from Reddit which contains millions of JSON objects that encapsulate user comments on a myriad of topics along with several metadata points that contextualize those comments.
We randomly extracted 9296 sentences from this large corpus, which has 104,861 tokens and 10,625 types, and manually annotated them as to whether they convey one or more of the six basic emotions listed above or none of them.
Statistics regarding class distribution for this corpus are as follows:
The following are just a few examples of annotated sentences containing one or more emotions from this corpus. Be advised there’s some colorful language in some of these sentences.
“What the fuck is wrong with you idiots?” Anger
“how dare he mispronounce the Zemgod’s name like that 😤 😤 😤 💥 🔫 🔪” Anger
“He’s half scary looking and half retarded looking … which makes him even scarier.” Fear, Disgust
“It sucks just waiting for the other shoe to drop.” Disgust
“I’m just worried how packed the gym is gonna be when I go back to college on Monday.” Fear
“I was legitimately scared when encountering drunk Russians.” Fear
“Love the bush 😻” Joy
“Girl, that is seriously impressive 💋” Joy, Surprise
“Goddammit just when I was getting hope 💢 😢 😭 😭” Anger, Sadness
“It was a great deck and I’m particularly sad it’s been nerfed into the ground.” Joy, Sadness
“How do you do that it looks so cool” 😮 Joy, Surprise
“oh my god this is the top fucking comment?” Anger, Surprise
Training An Emotion Classifier
Our emotion classifier analyzes sentences and tries to determine whether they convey one or more emotions or no emotion at all. From a Machine Learning architecture point of view, this is a multilabel setup where each vectorized unit (sentence) needs to be allowed to be labeled as belonging to one or more classes.
In order to allow for this specific setup, we designed an emotion classifier using the Keras+Tensorflow combo. This classifier is different than any other classifier that we have trained because it uses one common encoder and six different output layers. This architecture saves a lot of processing time because data only needs to pass through one single stack of embedding, lstm and attention layers. Plugging this single stack to each of the different output layers accounts in fact for six different classifiers, one for each basic emotion.
Each of these classifiers is a binary classifier that determines whether a vectorized unit belongs to the class it models or not. If no positive class has been assigned to a particular vectorized unit after analyzing it with all six emotion classifiers, then the “No emotion” label is assigned.
One last point that we would like to touch upon is the fact that the embedding layer uses a matrix of pre-trained embeddings that were trained using word2vec (Mikolov et al., 2013) on very large sample of the Reddit corpus. In particular we used the entire month’s worth of Reddit comments for the following months and years: December 2005, June 2007, May 2009, September 2011, July 2013, January 2015 and April 2017.
This allows us to enrich the vocabulary for the above described architecture by bring in word distribution knowledge in the shape of continuous word representations that is not contained in the small annotated corpus used as training corpus.
Results Discussion
The results we are able to achieve on the development corpus are as follows:
Even though these results are relatively good, experience tells us that threshold results help with maintaining high precision when trained classification models are generalized to new, unseen data.
We conducted another series of experiments to try to determine the probability threshold that each classifier should use as an indication to whether predictions should be deemed reliable or not. One criterion guided our search for optimal probability thresholds: precision should be high for the positive class even at the expense of some recall for this class, but without greatly degrading it.
Results and optimal probability thresholds for each binary classifier are as follows:
Finally, in order to get a better idea of how well these set of classifiers should be expected to generalize to new, unseen data, we present the results achieved on our testing corpus using the thresholds found during the previous step:
For each of the different classifiers in the table above: class 0 (negative class) refers to sentences that don’t contain each particular emotion; class 1 (positive class) refers to sentences that contain each particular emotion.
Conclusion
In this post we’ve showed how a complex emotion classifier can be learned if one finds and annotates the right data. Reddit users, as many other online communities, convey many emotions when discussing a myriad of topics, so this type of corpora are perfect candidates to automatically learn emotions from topic discussions.
Emotions are essential to human communication. Humans interact with each other not only to exchange or request information or to compel others to perform certain actions, but also to let other people know of their emotional opinions and state of mind.
Emotion classifiers can be used as a more fine-grained extension to traditional sentiment analyzers, which don’t provide as much depth into what users are thinking and/or feeling.
References
Brave, S. (2011). Emotion in Human-Computer Interaction. In J. A. Jacko & A. Sears (Eds.), The human-computer interaction handbook (pp. 81–96). Hillsdale, NJ: Lawrence Erlbaum Associates.
Ekman, P. (1992). An Argument For Basic Emotions. Cognition and Emotion, 6 (3/4), 169–200.
Hudlicka, E. (2008). What Are We Modeling When We Model Emotion?. Proceedings of the AAAI Spring Symposium “Emotion, Personality, and Social Behavior,” 52–59.
Lin, J., Sprarangen, M., & Zyda, M. (2012). Computational Models of Emotion and Cognition. Advances in Cognitive Systems, 2, 59–76.
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. ICLR 2013 Conference Track, cs.CL.
