Is a Magazine Generated Using Twitter’s Trending Topics Worth Reading?

I was curious to understand how Trending Topics come to life. Is popularity generated by the media, or are popular topics the drivers for media coverage? Unfortunately, I chose the wrong moment to start monitoring this: the worst US presidential campaign in history.

The concept

Something that I noticed is that very often, in Italy, social media drive traditional media. The feeling is that, at the end of the day, journalists read Twitter and write about the trending topics instead of generating Twitter’s trending topics through local news coverage. What I wanted to check was if the relationship between social and traditional media was the same even in the US.

… the website became very soon an additional weapon in the hands of conspirationists, alt-right activists, and fake news writers…

I started monitoring international and US trending topics during the summer before the US Presidential Election of 2016. Remember that campaign? Russiagate, sex scandals, fake news and so on. The scrapers ran few weeks, and then I stopped them: with the exception of a few big news stories, the website became very soon an additional weapon in the hands of conspirationists, alt-right activists, and fake news writers. At that time, Twitter trending topics were almost a monopoly.

How it works

This web app is generated starting from the Twitter trending topics list. A bot gets the list of trending topics, and then a set of related tweets. A first check is performed and tweets are validated to check if they contain a link, and if this link has not been previously blacklisted. A second validation happens against the author of the tweet, against a list of known spammers. Once both tests are passed, the link (not the tweet), is added to a queue for further analysis.

A second bot then elaborates this queue, scraping each link to get OpenGraph tags (if available) or HTML meta-tags. Once this information has been collected, the article is added to the database, and a relation to the trending topic is created. Only when all the links that belong to a certain set of trending topics have been checked, a “new issue” of the magazine is published.

The technology

The Twitter bot and the scraper are written in PHP. Everything is built on top of my own PHP Boilerplate, a framework I built to quickly bootstrap PHP applications. Twitter APIs are consumed using Abraham Williams’ TwitterOAuth class. The scraper leverages the capabilities of Guzzle, an HTTP client for PHP. The database is MySql. The frontend is a simple HTML + VanillaJS single page Application.

Download

The complete source code (PHP Scripts and HTML, CSS, and Javascript templates) are available for download on GitHub. Please be careful if you choose to try it because it’s really resource-consuming.

My name is Simone Lippolis, after spending almost ten years as Design Technologist at frog, I am now with Cisco, as a Data Visualization Expert. This article is part of my online portfolio that you can access at: simonelippolis.com.