reference: https://github.com/jeremygottfried/twitter-api-import-followers-CLI
How hard is it to store your Twitter followers in a database? Not very hard. Facebook’s another story.
With all the media hype about Cambridge Analytica, I was questioning my lifelong Facebook addiction. Were all the rumors about Facebook’s weak level of security true? I wanted to know how difficult it would be to store all my Facebook friends data in a database using the Facebook Graph API. Just like Bruce Almighty (photo above), I got to work to find answers to my questions.
What is an API?
API stands for Application Programming Interface. Most social media websites have an API that allows apps to incorporate users’ data.When you first log into an app like Grubhub you’d see the options in the photo below. If you log on through Facebook, Grubhub gets your Facebook profile information through Facebook’s API, then stores it in cache and local storage.
I was on a quest to learn how to utilize this powerful technology. Before making any requests to the Facebook API, I had to create an app on Facebook. That’s how I’d get an authentication key to ask for data from the Facebook API. I created a free website for my app through webnode.com, and set my background photo to a closeup of pasta (see photo below).
Here’s the current list of approved requests for my app, aptly named “Jeremy Gottfried Facebook API Tester.”
How I set up the Facebook Graph API
To get started making API requests, I installed the Koala Ruby gem, a library that helps you make requests to the Facebook API in Ruby. But as I began sending requests to the Graph API, I discovered that Facebook would only send back the name and ID of one friend from my friend’s list — my brother Jon.
Mark Zuckerberg explained in a recent Facebook post that Facebook has had strict API rules since 2014. You cannot request data about a user’s friends unless those users authorize your app. That means I couldn’t request my friends’ data even though I was a user of my app.
In order to gain new users, I had to make my Facebook app live. For that, I had to give Facebook a link to my app’s privacy policy. I automated writing my privacy policy through a nifty website called termsfeed.com.
To request any data more than users’ public profile and email, Facebook’s dev team required a video of how your app utilizes that specific data. They also require a list of what platforms you use it on, and a written explanation of how you use the data.
Facebook API was a no-go
Since I haven’t developed a functional app, I could not get approved for requesting private info from users. Furthermore, Facebook is not reviewing any new data requests until further notice as a result of Cambridge Analytica.
Getting user data from Twitter is much easier
I was out of luck with Facebook, but Twitter makes it easy to get your friends data into a database. Twitter has it’s own API, along with a well documented Ruby Gem. The Twitter API protects against misuse through rate-limiting. That means an unverified app can only make 15 list requests per user every 15 minutes. Twitter breaks up follower lists into multiple pages, so you have to make a new API request for every 20 followers. That means you can only get 1200 followers per hour.
sleep(3.5) TIP: If you have a large number of followers, you can avoid hitting a rate limit error by using the ruby sleep method. My program sleeps for 3.5 seconds before adding the next user to my database.
How I got my Twitter followers’ data into a database
Here is the link to my code if you’d like to try my CLI on your own Twitter followers: https://github.com/jeremygottfried/twitter-api-import-followers-CLI
To send requests to twitter, you need your own Twitter access tokens. I got my own here by creating an app. Using Active Record and sqlite3 gems, it was relatively easy to automate the process of creating tables and adding my followers info to tables.
My method for importing followers to my database is simple given the amount of data. If the follower already exists in my table, I do nothing. If it does not exist, I create a new instance of the follower class and save it in my table. Then I sleep for 3.5 seconds to avoid the Twitter API rate limit. There are a few more lines of code not shown, which get some of the data from the nested hashes before deletion.
My ‘Create_Followers’ migration file iterates through all the attributes of one follower. It checks the class of each attribute and creates a column of that type in the followers table. The schema and database are already created for you in my github repo, so don’t worry about running the migration yourself. It will return errors if you try because I reformatted the TwitterAPI class to receive its auth keys from gets.chomp.
Once I ran the code, I had a database full of data for 586 Twitter users. I wrote some methods that you can try on your own followers. Here are the most popular Hashtags in the recent tweets from my followers. It makes sense that #music and #soundcloud got the highest counts since I am a musician.
Here are the most common locations of my Twitter followers:
And lastly, the most common keywords used in my twitter followers descriptions. I was pleasantly surprised that the word “music” earned fifth place.
Try it out for your own followers by cloning my github repo and running bin/run.rb. You can build off my methods to analyze your followers’ data further.
