Live Twitter Sentiment Analysis (With Deployment)

This story will be divided into 4 parts :

1. Connecting with Twitter API and extracting the data.
2. Preprocessing the Data, and Using TextBlob for sentiment analysis.
3. Create an API using Streamlit and Flask.
4. Deploy the project on Heroku.

• Our Goal will be to Create an API where the user will Enter a Topic, which we will search on Twitter and Extract tweets related to that Topic.
• We will then do sentiment Analysis on the extracted tweets and classify them into Positive, Negative, Neutral.
• Further, we will provide visualizations so the Data can be further analyzed by the user.
• Here is a link to the Project i have Deployed, just so we are clear what we are working towards.

Link : https://twitter-sentiment-analysis07.herokuapp.com/

All the Steps we are going to discuss can be found here in the TwitterSentimentAnalysis.ipynb file. I will only be covering the important concepts here, the rest is just standard Python code.

Github Link: https://github.com/DarekarA/TwitterSentimentAnalysis_Public

1.Connecting with Twitter API and extracting the data.

• For this you will need a Twitter Developers account, if you don't have one it is pretty easy to get, Just follow these steps :

Apply for a Twitter Developer Account.

https://www.extly.com/docs/autotweetng_joocial/tutorials/how-to-auto-post-from-joomla-to-twitter/apply-for-a-twitter-developer-account/

• Once you get the account Just create a dummy app, and from that app we will get the necessary Keys & Tokens which we need for the API.
• The 2 Steps for this are :

1. A Twitter app can be created via the Twitter app dashboard page with an approved developer account.

2. Generate access tokens on the “Keys and Tokens” tab in an app’s “Details” section within the Twitter app dashboard. Click the “Create” button in the “Access token & access token secret” section.

• After the connection is established, we can query the Twitter Api and ask for the data we want.
• We will have a DataFrame “df” ready to store this extracted data.

Fetch Data using tweepy.Cursor()

• Using tweepy.Cursor() we will fetch the tweets for the Topic we selected, Every tweet will come with information such as username of the person who tweeted it, Likes/Retweets for that tweet, Location of the User and so on.
• We can just use whichever attributes that interest us and store it in a DataFrame so we can further process it.

2. Preprocessing (Clean)the Data, and Use TextBlob for sentiment analysis.

Image from : https://dimensionless.in/

• Like any other project, we will need to clean the data before we perform any analysis on this.
• Cleaning stage will be straightforward , we will remove any tags like “@,#” . We will also remove Special characters and any links.
• All this can be done using a simple Function

Function to clean all the extracted Tweets.

• Now that the data is clean, we can use this data to analyze the sentiment using TextBlob.

• TextBlob is a big library for processing textual data, however we are for now only interested in the “Polarity” score provided by TextBlob, Using this we can classify the tweets into Positive, Negative or Neutral.
• TextBlob gives Polarity of a sentence. Polarity range is [-1,1].
• -1=Negative
• 0 =Neutral
• 1 =Positive

Function to analyze the sentiments

• Once we get the sentiments, we can analyze the data as usual.
• Below are some examples of how we can analyze/visualize the data.

Overall Distribution of Sentiments.

Pie Chart for Different Sentiments.

Most frequently used Words from the tweets we extracted.

• Similarly we can go ahead and do many such visualization and different Analysis based on the sentiments we derived from the tweets to get an idea about what the discussion on Twitter is happening about any topic we selected.

3.Create an API using Streamlit and Flask.

• The API part is simple, Just refer the App_Streamlit.py for the Code.
• https://github.com/DarekarA/TwitterSentimentAnalysis_Public has this and all other required files.
• You can run this code on local and test it at http://localhost:5000/
• After this, Upload this to your GitHub account which will make it ready to deploy it on Heroku.

4.Deploy the project on Heroku.

• In addition to our .py file, we will be needing a :

1.Procfile , 2.setup.sh , 3.Requirement.txt (in CMD Just type Pip Freeze to get the requirement file). All these files are available in the GitHub link.

• 1. Update the procfile (this is starting point of the program)
  2. update the setup.sh file
  3. Dump all this files directly in Github Repository.(Not inside folder)
• Then just go to the Heroku app and Link you GitHub, choose the correct directory and Deploy the Project .
• And just like that your Project is ready to use for everyone!!!

• Hope these steps help you in the process of Deploying the end-to-end project and it goes seamlessly. In case of any queries, any addition you feel we can do to this , do reach out to me!