NLP: How does NLTK.Vader Calculate Sentiment?

NLTK.Vader is one of the more popular tools for sentiment analysis

Ying Ma
Ying Ma
Feb 5 · 4 min read

Sentiment analysis is one of the most popular field in Natural Language Processing (NLP) that automatically identifies and extracts opinions from text. This technique transforms large-scaled unstructured text data into structured and quantitative measurements of the sentimental opinions expressed by the text. Sentiment analysis has been widely applied to monitor the sentiment trend in product reviews, social media comments, news and blog articles.

Researchers have devoted more than a decade to solve this problem, and a few NLP-based sentiment analysis algorithms are readily available. In this article, I will review one of the most popular sentiment analysis tool NLTK.Vader, break down the technical details of this algorithm and discuss how we can make the best use of it.

Lexicon and Rule-based Model

Valence Aware Dictionary for sEntiment Reasoning, or Vader, is a NLP algorithm that blended a sentiment lexicon approach as well as grammatical rules and syntactical conventions for expressing sentiment polarity and intensity. Vader is an open-sourced package within the Natural Language Toolkit (NLTK) and here are the source code and the original publication if you are interested to check them out.

What is Sentiment Lexicon?

The lexicon approach means that this algorithm constructed a dictionary that contains a comprehensive list of sentiment features. This lexical dictionary does not only contain words, but also phrases (such as “bad ass” and “the bomb”), emoticons (such as “:-)”) and sentiment-laden acronyms (such as “ROFL” and “WTF”). All the lexical features were rated for the polarity and intensity on a scale from “-4: Extremely Negative” to “+4 Extremely Positive” by 10 independent human raters. The average score is then used as the sentiment indicator for each lexical feature in the dictionary. For example, in Vader, the word “okay” has a positive rating of 0.9, “good” is 1.9 and “great” is 3.1, whereas “horrible” is -2.5, the frowning emoticon “:(“ is -2.2, and “sucks” is -1.5. Vader’s lexicon dictionary contains around 7,500 sentiment features in total and any word not listed in the dictionary will be scored as “0: Neutral”.

Grammatical Rules

Besides the sentiment lexicons, there are structures that are neutral inherently but can change the polarity of sentiment (such as “not” and “but”) or modify the intensity of the entire sentence (such as “very” and “extremely”). In Vader, the developers incorporated several heuristic rules that handles the cases of punctuation, capitalization, adverbs and contrastive conjunctions. Below are a few examples of how the degree modifiers boosted the positivity in the compound score of a sentence.

Image for post
Image for post

Calculate the Compound Score

To calculate the sentimental score of the entire text, Vader scans the text for known sentimental features, modified the intensity and polarity according to the rules, summed up the scores of features found within the text and normalized the final score to (-1, 1) using function:

In Vader, alpha is set to be 15 which approximates the maximum expected value of x.

In addition to the compound score of the sentence, Vader also returns the percentage of positive, negative and neutral sentiment features, as shown in the previous example.

Use Cases

Since the development of this algorithm in 2014, Vader has been widely used in various forms of sentiment analysis to track and monitor social media trends and public opinions.

Based on the heuristic rules and the normalization calculation, we can tell Vader will average out the sentiment if the input text is relatively long or has several transition in term of tones and sentiment. This is because by design Vader is attuned to microblog-like contexts, which is usually no more than 280 words and has singular sentimental theme. Validation of the algorithm also attested that Vader performs exceptionally well in the social media domain, and outperforms human raters at classifying the sentiment of tweets.

Because of the embedded lexicon and rules, Vader is computationally economical especially comparing to the machine learning algorithms that requires massive operation for word embedding and training. Even though the sentiment features are restricted within the built-in lexicon and rules, it is relatively easy to modify and extend the sentimental vocabulary and tailored the Vader to specific contextual use cases.

Ro Data Team Blog

Ro Data Team Blog: data analytics, data engineering, data…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store