Building a Twitter Sentiment-Analysis App Using Streamlit
“The fastest way to build custom ML tools”
Not everyone is a data scientist, but everyone needs the power data science brings.
Streamlit helps us solve that problem.
The Streamlit framework makes deploying ML models just as easy as a few function calls.
For example, if you run:
Streamlit will build an interactive slider just like:
To install Streamlit, you can run:
pip3 install streamlit
And then you can run your apps with:
streamlit run <FILE>
python <FILE> will not work.
All the code we will be developing in this tutorial will be available on my GitHub, here.
Sentiment classification is a classic problem in NLP regarding understanding whether a sentence is positive or negative.
For example, “I love Python!” should be classified as a positive sentiment while “Python is the worst!” should be classified as a negative sentiment.
There are plenty of ways to implement a sentiment classifier in all popular machine learning libraries. Still, for the sake of ease (and better results too!), we will be using Flair, a library for state-of-the-art NLP classification.
If you don’t already have Flair, you can install it with:
pip3 install flair
For any data science project, we need, well, data. Sentiment140 is perfect for that.
The Sentiment140 dataset is a collection of 1.6 million tweets labeled as 0, negative sentiment, or 4, positive sentiment.
You can download Sentiment140 at its Kaggle page, here.
Loading the Data
Once you’ve downloaded Sentiment140, you can load it into memory with:
The above code should output something like:
1459123 4 @minalpatel Any more types of glassware you'd...
544833 0 I was a bit puzzled as to why it seemed to it...
398665 0 Yay...my car is ready....Was about 2500 miles...
708548 0 @JoshEJosh How ya been? I MISS you!
264000 0 @MrFresh0587 yeah i know. well...i'm going to...
However, since we are shuffling the order of the data with
.sample(frac=1), you will get something a bit different.
Right now, the data is pretty messy. To fix this, we need to pre-process it.
The above function is a bit tedious, but in short, it removes all unrecognized characters, pads out punctuation with spaces (
‘ . ’ ), removes links, and cuts the string to 280 characters. There are much better ways to remove links, but this naïve approach works fine.
Flair requires a specific format for the data to be in; it wants three CSV’s that look like:
In our case, the data will end up looking like:
__label__4 <PRE-PROCESSED TWEET>
__label__0 <PRE-PROCESSED TWEET>
Getting the data to look like this takes three steps:
- Apply the pre-processing function:
__label__ in front of each sentiment:
3. Save the data:
Above, you may have noticed two things:
- We are only saving a percentage of the data. Like I said earlier, Sentiment140 is enormous and takes a lot of RAM when Flair loads it.
- We are splitting up the data into train, test, and dev files. When Flair loads the data, it wants it split up into these three files (which is standard for most data science). By default, the split would be 80%-10%-10%, but this is, of course, scaled-down for RAM reasons.
Now the data is all ready to go!
In this tutorial, we will only go over the basics of Flair. If you would like more detail I would recommend reading this article by Tadej Magajna or the official Flair tutorials on their GitHub (tutorial #7 was the most helpful for this project).
Loading the Data (again)
Now that we have three CSV’s for our test, train, and dev sets, we can load them with Flair’s
Then we can make a label dictionary to hold all the labels assigned to the text in the corpus with:
And after that, we can load Flair’s provided GloVe embeddings:
The two commented-out lines are extra word embeddings provided by Flair for superior results. They will most likely improve the performance of the model; however, due to limited RAM, I have not been able to test them.
After loading the embeddings, we can initialize them with the following:
With the word embeddings and label dictionary in place, we can combine them into a
Then we can create a
ModelTrainer instance to train the classifier on the corpus we loaded.
And the moment we’ve all been waiting for, train the model:
The default hyperparameters work fine, but feel free to test different variations to see what results you get.
After a while, the model will finish training (or you can stop it early with
ctrl+c, it will still save), and Flair will save it in the path
model-saves/final-model.pt (which we specified with the first argument).
If you would like to test out the model, you can run something like:
Which should print out something like:
[4 (0.9758405089378357)] [0 (0.8753706812858582)]
Great, now we have a model that can predict whether a single tweet is positive or negative. That’s not too useful by itself. So what’s a good next step?
My idea was to scrape Twitter for recent tweets with a given query, classify each one into a positive/negative sentiment, and calculate the ratio of positive to negative tweets, however, feel free to do whatever you’d like.
To install twitterscraper, you can run:
pip3 install twitterscraper
And you’re all set!
We’ll get to using twitterscraper in a bit.
In a new file, that I’ve personally decided to call
main.py, we can start off by defining a few imports:
Next, we can do a few basic things like setting the page title and loading the classification model:
with st.spinner block allows us to display text while something computes. In this case, we display
Loading classification model…, as the model loads.
Next, we can copy-paste the pre-process function we defined earlier:
For the first example on our app, we will classify individual tweets.
First, we set a subheader and take text input from the user:
Then, as long as the input string is not empty, we:
- Pre-process the tweet
- Make predictions (with a spinner as we wait)
- And show the predictions
st.writeallows us to write any text to the user. It can even display Pandas
DataFrameswithout any modification.
If you go ahead and run that file with:
streamlit run main.py
You should see something like:
Next, we can implement the idea we touched earlier, searching for a topic on Twitter and finding the positive to negative tweet ratio.
And finally, we can show the data we just collected (assuming it has been defined):
Run the app one more time (or refresh the tab if it is still running) and you should see:
As you can see in the above results, the tweets classified as negative are very close to
0.5, meaning the model is unsure whether the tweet is positive or negative, and the tweet most likely neutral. You may want to set a threshold, e.g.,
0.45 < x < 0.55, for which tweets are classified as neutral.
As a recap, here is the full Streamlit app we just developed:
And running that will create the following app:
Conclusion and Final Thoughts
Streamlit is powerful.
Really powerful. Single-handedly, it can take AI and data science a step closer to being accessible by the public.
This tutorial serves as an introduction to how Streamlit works (and how to do basic text classification), and I hope that with the knowledge you’ve gained here, you go out and turn some of your ML projects into Streamlit apps (it’s really easy).
At the same time, the sentiment classification technology powering this app is dangerous. Text classification can be used to understand personal beliefs, political stances, and a whole host of other personal ideologies at a scale and accuracy never seen before —use wisely!
Anyway, until next time,
Source code for everything we built in this tutorial: https://github.com/EvolvedSquid/tutorials/tree/master/twitter-sentiment