Python script to collect data of your Twitter connections!!! | Daily Python #4

This article is a tutorial on how to collect data of your twitter connections using ‘tweepy’ and Twitter APIs in Python.

This article is a part of Daily Python challenge that I have taken up for myself. I will be writing short python articles daily.

The fact that data is so readily available nowadays is not surprising. This article focuses on how data can be fetched from a social media platform. This data can be used for analysis, to generate graphs based on your connections. Students can use this as a mini-project to show how they are connected to each other on Twitter and show these relationships in the form of graphs.

Requirements:

1. Python 3.0
2. Pip

Install the following packages:

1. tweepy — Library that provides easy access to Twitter APIs

pip install tweepy

Let’s start by importing the libraries in our code.

import tweepy
import time

In order to access Twitter APIs, we need to create a Twitter developer account and create an application that will provide us the API keys and Access tokens.

You can go through my previous article — Python script to fetch tweets based on hashtags(#), to understand how to create a twitter application.

Python script to fetch tweets based on hashtags(#)| Daily Python 02

Store your API keys and Access tokens in the variables as given below:

consumerKey = “YOUR_CONSUMER_KEY”
consumerSecret = “YOUR_CONSUMER_SECRET_KEY”
accessToken = “YOUR_ACCESS_TOKEN”
accessTokenSecret = “YOUR_ACCESS_TOKEN_SECRET”

First, we create an object of the OAuthHandler provided by ‘tweepy’. Here, we use the Consumer Key and the Consumer Secret Key that we got after creating an application on Twitter. We also, provide this OAuthHandler the access tokens that we generated for our application.

auth = tweepy.OAuthHandler(consumerKey, consumerSecret)
auth.set_access_token(accessToken, accessTokenSecret)

Now, let’s create the API object which will be used to fetch your followers list and then to like the latest tweet of each of your followers. To create an object of the API we simply provide the ‘auth’ variable to ‘tweepy’.

api = tweepy.API(auth)

Fetching the followers list

For fetching the followers list we use the followers API provided by Twitter.

GET followers/list

def get_followers(screen_name):
    print('Getting Follower list of ',screen_name)
    followers = []
    followers_screenNames = []
    users = tweepy.Cursor(api.followers, screen_name='@'+screen_name, wait_on_rate_limit=True,count=200)
    for user in users.items():
        try:
            followers.append(user)
            followers_screenNames.append(user.screen_name)
        except tweepy.TweepError as e:
            print("Going to sleep:", e)
            time.sleep(60)
    
    print('Fetched number of followers for '+screen_name+' : ',len(followers))
    return followers,followers_screenNames

The ‘screen_name’ input parameter is to specify your Twitter account’s username. ‘count’ limits the results of the request. ‘wait_one_rate_limit’ — Whether or not to automatically wait for rate limits to replenish.
Change the count limit as per your account’s number of followers.

Fetching the following list

For fetching the following list we use the friends API provided by Twitter.

GET friends/list

def get_following(screen_name):
    print('Getting Following list of ',screen_name)
    friends = []
    friends_screenName = []
    users = tweepy.Cursor(api.friends, screen_name='@'+screen_name,
            wait_on_rate_limit=True,count=200)
    for user in users.items():
        try:
            friends.append(user)
            friends_screenName.append(user.screen_name)
        except tweepy.TweepError as e:
            print("Going to sleep:", e)
            time.sleep(60)
    print('Fetched number of following for '+screen_name+' : ',len(friends))            
    return friends,friends_screenName

Function to fetch user data

Let’s write a function to go through each of my followers & following and store their data in a python dictionary. After traversing each of the users from my followers and the following list, we’ll dump the data into a file.

def fetch_data():
    people = list(get_followers('sonawaneajinks'))[0]
    people.extend(list(get_following('sonawaneajinks')[0]))
    data = {}
    for user in people:
        try:
            print('Fetching data of ',user.name)
            ob = {
                    'ID':user.id,
                    'Name':user.name,
                    'StatusesCount':user.statuses_count,
                    'Follower_Count':user.followers_count,
                    'Following_Count':user.friends_count,
                    'Followers':list(get_followers(user.screen_name))[1],
                    'Following':list(get_following(user.screen_name))[1],
            }
            print(ob)
            data[user.screen_name].append(ob)
            print('Data saved, moving on to next user\n\n')
        except Exception as ex:
            print(ex)
            pass
    
    with open('data.txt', 'w') as outfile:
        json.dump(data, outfile)

First, we fetch the followers, then the following and store them in a Python list. We traverse each of the users and store their name, id, # of tweets, followers counts, the following count, follower accounts, following accounts.

We create a dictionary ‘ob’ and store the above values in it. It is then added to the main dictionary ‘data’ which will contain information of all users after traversing all the users. The ‘data’ dictionary is then dumped into a text file for future use.

Let’s call this function and see the output:

fetch_data()

Snip of the output

This was a really simple tutorial on how to collect user data from Twitter. I hope this article was helpful, do leave some claps if you liked it.

Open to suggestions, do not hesitate to give your valuable feedback.

Ajinkya-Sonawane/Python

Login * Instagram