Twitter Sentiment Analysis using Tweepy, TextBlob, and Django

Implementing normal sentiment analysis code in the Django Framework

Did you know that Twitter has its own API for letting the public to access the twitter Platform and its databases?

The answer is YES!

The Twitter API allows you to not only access its databases but also lets you read and write Twitter data. One can further use this information to do the following:

1. Listen and Analyse: This is where you can analyze past tweets and use them in an application.
2. Programmatically create and manage Twitter Ads Campaign
3. Publish and curate: There are several tools at your disposal to employ twitter data into apps and websites.
4. Engage: Create human and chat-bot applications for businesses etc for a fast and convenient way for communication.

To access the Twitter API the following are required:

1. A Twitter account
2. Twitter Developer account

One needs to apply to get access to a twitter developer account and it is not at all difficult.

Here is the link to apply: https://developer.twitter.com/en/apply-for-access

The main idea of analyzing tweets is to keep a company in check about the feedback for its products or just to get interesting insights about the latest issues. It is important to listen to your community and act upon it.

Main modules used and why to use them:

Tweepy: Its an open-source python package that gives certain methods and classes to seamlessly access the twitter API in the python platform. To install tweepy module in the python environment, we simply write in the command prompt the following line:

pip install tweepy

TextBlob: Its a library for processing text data. It provides simple functions and classes for using Natural Language Processing (NLP) for various tasks such as Noun Phrase extraction, classification, Translation, and sentiment analysis. We are concerned with the sentiment analysis part of the text blob. It can be installed by writing in cmd :

pip install textblob

Regular Expression(re): A regex is a special sequence of characters that defines a pattern for complex string-matching functionality. In this project, we will use regex’s to clean our tweet before we can parse it through our sentiment function.

Now down to the main act :

1. We need to import the libraries that we have to use :

import re
import tweepy
from tweepy import OAuthHandler
from textblob import TextBlob

2. Now before we start parsing our tweets, we need to get the access and authorization from the twitter API. This is done OAuthHandler() method of tweepy module. When we go to our Developer portal and copy the keys from our API and access keys and token /secret options.

3. Always use a try and catch block when dealing with data received from the internet as:

try:
     self.auth = OAuthHandler(API_key, 
                                     API_secret)
     self.auth.set_access_token(access_token,
                                       access_token_secret)
     self.api = tweepy.API(self.auth)
     print('Authenticated')
except:
     print("Sorry! Error in authentication!")

4. Now there is a need to define some functions so that they can we called in the main function where we give our predictions. These functions are the cleaning_process(self,tweet) and the get_sentiment(self,tweet). Now let's discuss these methods.

5. Cleaning_process(): This function uses the sub-method of re module to remove links and special characters from our tweets before it can be parsed into TextBlob.

def cleaning_process(self, tweet):
        return ' '.join(re.sub("(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|     (\w+:\/\/\S+)", " ", tweet).split())

6. Get_sentiment(): This function takes in one tweet at a time and using the TextBlob we use the .sentiment.polarity method. The data is trained on a Naïve Bayes Classifier and gives the tweet a polarity between -1 to 1 (negative to positive).

7. Now comes our getting the part of the tweet. In the method get_tweets() we pass the twitter id and the number of tweets we want.

8. For each tweet, we analyze the tweet and put the tweet and its corresponding sentiment in a dictionary and then put the dictionary in an array containing all the tweets.

for tweet in fetched_tweets:
     parsed_tweet = {}
     parsed_tweet['text'] = tweet.text
     parsed_tweet['sentiment'] =self.get_sentiment(tweet.text)
          if tweet.retweet_count > 0:
                if parsed_tweet not in tweets:
                    tweets.append(parsed_tweet)
                else:
                    tweets.append(parsed_tweet)

9. In our main function, we create an object for the TwitterSentClass() which gets authenticated on initiation. We put the output(Negative and Positive percentages) in an array ‘arr_pred’ and put 5 positive and negative tweets in the arrays ‘arr_pos_txt’ and ‘arr_neg_txt’.

# adding the percentages to the prediction array to be shown in the html page.
        arr_pred.append(pos)
        arr_pred.append(neg)
        
        # storing first 5 positive tweets
        arr_pos_txt.append("Positive tweets:")
        for tweet in ptweets[:5]:
            arr_pos_txt.append(tweet['text'])
# storing first 5 negative tweets
        arr_neg_txt.append("Negative tweets:")
        for tweet in ntweets[:5]:
            arr_neg_txt.append(tweet['text'])

Implementation in Django

1. Install Django frameworks using the command:

pip install django

2. In the cmd create a project in your desired directory, further we create an app and name them as per your wish.

django-admin startproject twittersentiment
cd twittersentiment
py manage.py startapp myapp

3. Add the app in INSTALLED_APP in the settings.py file

myapp added at the end of INSTALLED_APPS array.

4. Create a forms.py in your app folder and create the fields for the form to be shown on your page.

5. In the views.py file add the TwitterSentClass() code and call it in the prediction function.

6. View.py file contains two functions show() and prediction(). The show() function creates the form that u coded earlier and displays it onto the starting page of the site. The prediction.py function takes the twitter id received from the form and after prediction, the output sends all the information via arrays to the next HTML page where you will show the output.

7. The code for the HTML pages are shown below.

<!-- Code for the index page -->
<html>
 <body>
  <form action="hit" method="post">
   {% csrf_token %}
   {{ ff.as_p }}<br>
   <button type="submit">Predict</button>
  </form>
 </body>
</html>
<!-- Code for the prediction page -->
<html>
 <head>
  <title>Predicitons based on tweepy module</title>
  
 </head>
 <body>
  <form action="" method="post">
   {% csrf_token %}
{% for x in arr_pred %}
    {{ x }}<br>
   {% endfor %}
   <p>
   {% for x in arr_pos_txt %}
    {{ x }}<br>
   {% endfor %}
   </p>
   <p>
   {% for y in arr_neg_txt %}
    {{ y }}<br>
   {% endfor %}
   </p>
  </form>
 </body>
</html>

8. Add the HTML in the templates folder in your app folder.

9. Add the URLs in the urls.py file as:

Path added inside the URL patterns array.

10. To run the project in cmd write the lines:

py manage.py migrate
py manage.py runsever

11. Copy the IP given in the cmd and paste it onto any browser and using the tweet URL, open the forms page. The rest is self-explanatory.

12. The output is as follows:

Index.html page

prediction.html page

This concludes our project. This project is subjected to modifications and creativity as per the knowledge of the reader.

To access the project, here is the GitHub link:

avigautam-329/Django-projects

Thank you !! And happy coding!