Building a Twitter Sentiment-Analysis App Using Streamlit

“The fastest way to build custom ML tools”

Sebastian Theiler
Dec 14, 2019 · 7 min read
The final app we develop

Not everyone is a data scientist, but everyone needs the power data science brings.

Streamlit helps us solve that problem.

The Streamlit framework makes deploying ML models just as easy as a few function calls.

For example, if you run:

Streamlit will build an interactive slider just like:

All credit to the Streamlit GitHub for both the code and this image

To install Streamlit, you can run:

pip3 install streamlit

And then you can run your apps with:

streamlit run <FILE>

Typing python <FILE> will not work.

Check out the Streamlit GitHub, documentation, PyPi page, and home page for more information.


All the code we will be developing in this tutorial will be available on my GitHub, here.

Sentiment Classification

Sentiment classification is a classic problem in NLP regarding understanding whether a sentence is positive or negative.

For example, “I love Python!” should be classified as a positive sentiment while “Python is the worst!” should be classified as a negative sentiment.

Flair

There are plenty of ways to implement a sentiment classifier in all popular machine learning libraries. Still, for the sake of ease (and better results too!), we will be using Flair, a library for state-of-the-art NLP classification.

If you don’t already have Flair, you can install it with:

pip3 install flair

Sentiment140

For any data science project, we need, well, data. Sentiment140 is perfect for that.

The Sentiment140 dataset is a collection of 1.6 million tweets labeled as 0, negative sentiment, or 4, positive sentiment.

You can download Sentiment140 at its Kaggle page, here.

Loading the Data

Once you’ve downloaded Sentiment140, you can load it into memory with:

You may need to change data_path depending on where you’ve installed Sentiment140

The above code should output something like:

        sentiment                                               text
1459123 4 @minalpatel Any more types of glassware you'd...
544833 0 I was a bit puzzled as to why it seemed to it...
398665 0 Yay...my car is ready....Was about 2500 miles...
708548 0 @JoshEJosh How ya been? I MISS you!
264000 0 @MrFresh0587 yeah i know. well...i'm going to...

However, since we are shuffling the order of the data with .sample(frac=1), you will get something a bit different.

Right now, the data is pretty messy. To fix this, we need to pre-process it.

The above function is a bit tedious, but in short, it removes all unrecognized characters, pads out punctuation with spaces (‘.’‘ . ’ ), removes links, and cuts the string to 280 characters. There are much better ways to remove links, but this naïve approach works fine.

Flair requires a specific format for the data to be in; it wants three CSV’s that look like:

__label__<LABEL>    <TEXT>

In our case, the data will end up looking like:

__label__4    <PRE-PROCESSED TWEET>
__label__0 <PRE-PROCESSED TWEET>
...

Getting the data to look like this takes three steps:

  1. Apply the pre-processing function:

2. Put __label__ in front of each sentiment:

3. Save the data:

Above, you may have noticed two things:

  1. We are only saving a percentage of the data. Like I said earlier, Sentiment140 is enormous and takes a lot of RAM when Flair loads it.
  2. We are splitting up the data into train, test, and dev files. When Flair loads the data, it wants it split up into these three files (which is standard for most data science). By default, the split would be 80%-10%-10%, but this is, of course, scaled-down for RAM reasons.

Now the data is all ready to go!

Classification

In this tutorial, we will only go over the basics of Flair. If you would like more detail I would recommend reading this article by Tadej Magajna or the official Flair tutorials on their GitHub (tutorial #7 was the most helpful for this project).

Loading the Data (again)

Now that we have three CSV’s for our test, train, and dev sets, we can load them with Flair’s NLPTaskDataFetcher class.

Then we can make a label dictionary to hold all the labels assigned to the text in the corpus with:

And after that, we can load Flair’s provided GloVe embeddings:

The two commented-out lines are extra word embeddings provided by Flair for superior results. They will most likely improve the performance of the model; however, due to limited RAM, I have not been able to test them.

After loading the embeddings, we can initialize them with the following:

With the word embeddings and label dictionary in place, we can combine them into a TextClassifier model:

Then we can create a ModelTrainer instance to train the classifier on the corpus we loaded.

And the moment we’ve all been waiting for, train the model:

The default hyperparameters work fine, but feel free to test different variations to see what results you get.


After a while, the model will finish training (or you can stop it early with ctrl+c, it will still save), and Flair will save it in the path model-saves/final-model.pt (which we specified with the first argument).

If you would like to test out the model, you can run something like:

Which should print out something like:

[4 (0.9758405089378357)] [0 (0.8753706812858582)]

Correct predictions!

Twitter Scraping

Great, now we have a model that can predict whether a single tweet is positive or negative. That’s not too useful by itself. So what’s a good next step?

My idea was to scrape Twitter for recent tweets with a given query, classify each one into a positive/negative sentiment, and calculate the ratio of positive to negative tweets, however, feel free to do whatever you’d like.


Personally, I like twitterscraper by taspinar above all Twitter-scraping options. Although it isn’t as fast or pretty as the Tweepy API, you can bypass the request limit Twitter sets.

To install twitterscraper, you can run:

pip3 install twitterscraper

And you’re all set!

We’ll get to using twitterscraper in a bit.


Streamlit

In a new file, that I’ve personally decided to call main.py, we can start off by defining a few imports:

Next, we can do a few basic things like setting the page title and loading the classification model:

The with st.spinner block allows us to display text while something computes. In this case, we display Loading classification model…, as the model loads.

Next, we can copy-paste the pre-process function we defined earlier:


For the first example on our app, we will classify individual tweets.

First, we set a subheader and take text input from the user:

Then, as long as the input string is not empty, we:

  1. Pre-process the tweet
  2. Make predictions (with a spinner as we wait)
  3. And show the predictions
  • st.write allows us to write any text to the user. It can even display Pandas DataFrames without any modification.

If you go ahead and run that file with:

streamlit run main.py

You should see something like:


Next, we can implement the idea we touched earlier, searching for a topic on Twitter and finding the positive to negative tweet ratio.

And finally, we can show the data we just collected (assuming it has been defined):

Run the app one more time (or refresh the tab if it is still running) and you should see:

As you can see in the above results, the tweets classified as negative are very close to 0.5, meaning the model is unsure whether the tweet is positive or negative, and the tweet most likely neutral. You may want to set a threshold, e.g., 0.45 < x < 0.55, for which tweets are classified as neutral.


As a recap, here is the full Streamlit app we just developed:

And running that will create the following app:

This footage has been sped up, it will likely take longer to do the query than shown

Conclusion and Final Thoughts

Streamlit is powerful.

Really powerful. Single-handedly, it can take AI and data science a step closer to being accessible by the public.

This tutorial serves as an introduction to how Streamlit works (and how to do basic text classification), and I hope that with the knowledge you’ve gained here, you go out and turn some of your ML projects into Streamlit apps (it’s really easy).

At the same time, the sentiment classification technology powering this app is dangerous. Text classification can be used to understand personal beliefs, political stances, and a whole host of other personal ideologies at a scale and accuracy never seen before —use wisely!

On a different note, if you would like to learn more about deploying Streamlit apps to the web, I would recommend this tutorial by Rahul Agarwal about using free Amazon ec2 instances.


Anyway, until next time,

Happy coding!


Source code for everything we built in this tutorial: https://github.com/EvolvedSquid/tutorials/tree/master/twitter-sentiment

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Sebastian Theiler

Written by

I write articles about data science, artificial intelligence, and machine learning

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade