Twitter’s Controversial New Change: A Closer Look
How Data Scraping Has Forced Twitter Into Further Alienating Its Users
In a baffling turn of events, Twitter has decided to enforce strict rate-limiting on the number of posts you can view per day.
Essentially, users are now temporarily limited so that they can only view a set number of tweets per day.
It’s safe to say, people aren’t happy about this.
This is a… let’s say “strange” choice, considering most social media platforms aim to keep you “hooked”.
Generally, the longer you stay active on a platform, the more revenue social media websites like YouTube, Facebook or Twitter will make.
For example, this is why YouTube has such a complex recommendation algorithm: by suggesting more videos that you will like, it increases the chance that you’ll use the platform for longer.
So why has Twitter seemingly made the decision to actively prevent users from spending more time on the platform?
Here’s what you might not know about the situation.
The Big Data Scraping Problem
Twitter's rate limits were enforced to counter-act excessive data scraping and system manipulation according to Elon Musk.
Data scraping is essentially treasure hunting on the internet. It’s when a computer program searches through websites, grabbing any information that it needs or wants.
This is a big problem for Twitter.
Data scraping ultimately leads to a huge number of requests for data being made to Twitter’s backend services, which Twitter must then process.
Think of it like this, a bus full of tourists (a huge number of requests) suddenly arrives at a restaurant (Twitter).
The kitchen (Twitter’s backend services) gets overwhelmed trying to serve all of these customers at once. This puts a strain on the staff, slows down service for everyone, and can lead to errors.
This has always been a problem, but it’s become much more of a concern lately…
AI Impacts
The rise of AI magnifies this problem.
Popular AI platforms (and the tools built on top of them) like ChatGPT and BrowseAI can be configured to scrape and consume HUGE amounts of data and can do so over an insanely short period of time.
Imagine a program that is configured to pull back as many new tweets from Twitter as possible every few seconds, process the data and highlight key trends, sentiments or popular topics.
This will lead to millions upon millions of requests, all putting strain on Twitter.
Now, this problem doesn’t affect Twitter alone. Any platform can be targeted by web scrapers, particularly news and social media sites, and will have to figure out a way to deal with them accordingly.
This is why CAPTCHAs and anti-robot verifications are so prevalent.
So, is this rate-limiting the answer to Twitter’s problems?
I highly doubt it, at least not in this current form.
While it will lessen the impacts of data scraping, it also has a huge negative impact on the user experience.
In fact, many users see this only as another feeble attempt to drive users to pay up for Twitter Blue, Twitter’s premium subscription service, which has a higher allowed threshold of tweets that can be viewed per day.
Don’t get me wrong, some level of rate-limiting will be needed, that’s natural for most applications, but the current approach seems very heavy-handed.
If you enjoyed this article, please consider subscribing to my newsletter, The Debug Diary, for further insights and advice, and follow me on Medium and Twitter for ways to develop your skillset as a software developer and advance your career.
