Powerful Twitter Sentiment Analysis in Under 35 Lines of Code

Python, Tweepy, and Flair have made it dead simple.

Photo by Souvik Banerjee on Unsplash

Sentiment analysis can be as hard or as simple as you make it. Just like with any ML project you can pick the algorithm, train the model, and handle all of the issues that come along with it (hello, overfitting 😒).

I’ve been through that and I try to choose the easy road where possible. In the case of sentiment analysis, the work has been done. No need to reinvent the wheel.

So let’s have at it. We’ll only be using two libraries:

• Tweepy
• Flair

Flair is a NLP framework developed by Humboldt University of Berlin. They have models trained on social media data, which is perfect for our use here.

Tweepy is an open-source Python library for accessing the Twitter API.

Setup

We’ll need a few libraries installed to get this project off the ground:

pip install tweepy
pip install flair

Flair takes a while to install, it’s quite a large library. While it’s installing, head over to Twitter’s developer platform and sign up for an account.

Once you have an account, open the developer portal and make a new project and click the key icon next to it to get your API keys and authentication tokens.

Developer Dashboard

We’ll only be using v1.1 endpoints, so no need to worry about getting v2 access or any elevated access. The most basic account will do.

You’ll need four keys for this project, they’ll be labeled:

• API Key
• API Secret
• Access Token
• Access Secret

Keys and Tokens Page

Once you have those saved somewhere, our setup is done. We can dive into a Python file and start putting everything together.

Getting Tweets

Setting the API Handler

We’ll start by making sure we can get tweets from Twitter’s API. Let’s import tweepy and set those keys and tokens we generated:

import tweepy
API_Key = 'Your API Key'
API_Secret = 'Your API Secret'
Access_Token = 'Your Access Token'
Accesss_Secret = 'Your Access Secret'

Alright, now that we have our variables set, let’s set up an API handler. We can do this by using tweepy’s OAuthHandler class:

auth = tweepy.OAuthHandler(API_Key, Secret_Key)

With that new auth object, we want to use the set_access_token method to set our access tokens:

auth.set_access_token(Access_Token, Access_Secret)

Finally, we can set finish our API handler setup by creating a tweepy API class using that auth object:

api = tweepy.API(auth)

Let’s have a look at what our code should look like at this point:

API Handler Set

Searching Tweets

The Twitter API is pretty robust, there are many ways to gather tweets from certain users, followers, etc. In this example, we’ll just search tweets. This works the same as using the search feature on Twitter.

Twitter Search

To search tweet’s we’ll use our api object’s search_tweets method. This takes in a few different arguments but we’ll just set a query, language, count, and the tweet_mode:

tweets_data = api.search_tweets(
    q = 'test -RT',
    lang = 'en',
    count = 10,
    tweet_mode = 'extended'
)

If you noticed that our query has “-RT” in it, that is to exclude Retweets. That just makes things slightly simpler to work with.

That should be everything you need to search for tweets. Just print the tweets_data to make sure everything is up and running.

Tweet Data Printed

You should get a big list of all kinds of data. The Twitter API will send everything back, including all kinds of information we won’t use.

If you didn’t get that, you likely got an authentication error:

Authentication Error

This means you’re having issues with your keys and tokens. Make sure they’re in the right spot. If they are, try regenerating them because there was likely an error when you copied them over.

Getting the Right Tweet Data

So now that we’ve got a bunch of data printed, we need to trim it down to just the data we want. For now, we’ll just grab the text to run sentiment on.

We can do this pretty easily. Our tweet search outputs a list of tweets and we can use an attribute in tweepy’s model to grab the text:

tweet_text = [tweet.full_text for tweet in tweets_data]

That will output only the text associated with the tweet. If we print out tweet_text we’ll get a list of tweets (as strings):

Tweet Text Printed

From here we can start analyzing the sentiment of these tweets.

Sentiment Analysis

Flair is a pretty large library. There are only a few pieces of the library we are going to use, so we’ll specify those imports to make our code as readable as possible:

from flair.models import TextClassifier
from flair.data import Sentence

The first thing we’ll need to do with flair is set up our classifier:

classifier = TextClassifier.load('sentiment')

Next, we’ll set up the sentences which will be used for analysis:

tweet_sent = [Sentence(tweet) for tweet in tweets]

Finally, we’ll use the classifier to predict each tweet sentiment:

classifier.predict(tweet_sent, mini_batch_size=32)

So just for a recap on where we are in our program, here’s a screenshot with the tweet_sent printed out:

Sentences Printed

As you can see, Flair gives us the tokenized sentence. What we are really after though, are the labels. Let’s separate out just the “POSITIVE” and “NEGATIVE” labels:

tweet_sent = [sent.labels[0].value for sent in tweet_sent]

So now we get a printout of just the label given to the tweet:

Labels Printed

Now we have the tweets pulled and sentiment analyzed. We can simply combine the text with the sentiment.

analyzed_tweets = list(zip(tweet_text, sent_values))

Printing those out, we can see which tweets ranked positively and negatively.

Tweet Sentiment

Conclusion

So this covers a lot of ground, we’ve:

• Generated API keys and authorization tokens for Twitter
• Used tweepy to gather tweets
• Used Flair to analyze the sent

Just one more check on how much code that took:

Final Code

Right at 33 lines of code. That is an incredibly succinct program.

This is just scraping the surface of what both tweepy and Flair can do. You could build applications to analyze the sentiment of responses to tweets, followers sentiment on a subject, or even analyze sentiment in real time with streamed tweets!

If this helped you out, the best way to support me is by following me on Twitter or here on Medium!

Feedback is my friend, so feel free to reach out and tell me that you liked my story, want some topic covered, or that some part of this could be done better.

Sign up for our Free Weekly Newsletter. Fresh content from developers all over the world, fresh and direct from the field! Don’t forget to follow our publication The Dev Project.