Elon Musk vs. Physics
When we hear phrases like “cloud service” its easy to forget the physical industrial scale needed to deliver these websites. The Social Media industry relies on equal doses of physical infrastructure, algorithms and the determination of the humans that build+manage it all.
Despite existing in “cyber space” websites like Twitter exist physically in data centers. In Elon Musk’s take over of Twitter the social laws of human capital don’t appear to be the only challenges facing the embattled new CEO.
The limits of Human Capital
In the time since I last wrote about the numerous twitter Security Experts who’ve resigned twitter is now faced with a tidal wave of departing employees.
The alleged departures as of Thursday include entire Twitter teams:
- all Android developers
- all Site Reliability Engineers(SREs)
- all Payroll staff
- all US tax compliance officers
To my eyes this represents years of human capital development, over a decade of technical expertise and all key operational staff required to keep this cloud service operating. This constitutes one of the largest leadership failures I’ve ever witnessed in a large technology company.
Beyond wholesale team departures Twitter now also faces a class action lawsuit over discrimination. A fired staffer who could not comply with Elon’s new requirements for health reasons is objecting to the way layoffs were carried out. These are unnecessary internal conflicts which will undoubtedly distract the new CEO from the core business of running the company.
Data Centers Are Serious Business
I’ve been asked several questions by readers:
- “why are SREs and Android developers so important?”
- “twitter seems simple, its just a few text files that need to be sorted and served”
According to Twitter’s blog “hundreds of millions of tweets are sent each day”. This has resulted in the company retaining exabytes of tweets, photos, videos and metadata. To manage this data Twitter initially built their own custom datacenters and also contract with collocation company such as QTS Realty to warehouse the thousands-millions of servers which ingest and process this data.
In one Atlanta, GA datacenter alone twitter has contracted QTS for over 80,000 square feet of server racks and nearly 275 Mega Watts of electricity.
The exact number and location of Twitters collocation datacenters is a closely guarded trade secret.
Twitter is a already a Hybrid Cloud
One of the themes that Musk fanboys like to champion is the idea that Twitter does not need much engineering staff because they can simply outsource virtual servers in places like Google, AWS or Azure. This idea is frequently brought up by folks who don’t realize that Twitter is already a hybrid cloud. As far back as 2017 their engineers have been working to migrate a portion of their exabytes of data into Google Cloud.
Twitter is a complex web of network infrastructure, data ingestion, data storage and content delivery systems! A true tour-de-force of CS and Network theory hidden behind a deceptively simple mobile app interface. The engineers at Twitter have made this look easy for a over a decade, its anything but!
Reliability: Misfortune, Mistakes & Misconfiguration
As a sometimes tech founder I’ve developed a deep appreciation for what may motivate my co-founders and how we can stay sane. One of my favorite issues on this topic is the connection between rewards, stress and human performance. Time and again economist and HR executives have noted that offering large incentives for high performance sometimes produces the opposite!
It’s clear this lesson has been lost on Elon Musk.
“Less than half of Twitter’s remaining employees committed to Elon Musk’s ‘hardcore’ vision” — Business Insider
High stakes situations and industrial scale solutions are poor bed fellows.
Twitter has reached a scale where it must use fickle technologies like BGP(Border Gateway Protocol) and OSPF/STP(Spanning Tree Protocol). These technologies have a long history of small miss steps causing catastrophic consequences. Sometimes not just for the primary company operating them.
My undergraduate research, over a decade ago, was on these very systems. In that time not much has changed. A single BGP configuration mistake in 2021 resulted in -all- Facebook/Meta Inc properties going offline for over 24 hours!
So now imagine, if Facebook was down for over hours with thousands of active fully engaged SREs working hard to solve it, what then will happen to twitter who now has no such engineers remaining?
When worlds collide
Sadly, few twitter users appear to understand the dire situation the company is now placed in due to Elon Musk’s poor leadership. I’m especially disappointed in countless technology VCs who are currently demonstrating their lack of Due Diligence skills by down playing the risks facing Twitter. Twitter is speed running a text book case of poor risk assessment + mitigation skills.
It is Elon’s prerogative for sure to operate like this. However, its startling to me to see experts in building and investing into technology companies down playing this risk. I’m not calling anyone out today as to not get myself more cancelled by the VC class than I already am.
But, boy is this an eye opening exercise!
Chief among Twitter’s cyber security threats are nation state actors. Sometimes called APTs(Advanced Persistent Threats).
There is good reason many countries whom are not aligned with Twitter’s goals may wish to undermine the site.
In my opinion Twitter will be lucky to survive long enough for that to be a concern. Should twitter make it weeks or months without an extended outage it will be due to the herculean efforts of the people and engineers who made this site possible for the last decade. My hats off to y’all!
Internet, meet consequences
Behind the pleasant and clean mobile interfaces of modern social media lays a dark world full of dangerous content and bad actors. Out side of government hackers and accidental cascading failures are some of the worst human behavior imaginable. Modern websites must be built to withstand massive liability risks due to the constant ways that users participate on these sites.
Elon is famous for firing off a tweet calling a brave first responder a “pedo guy” after Musk’s ineffective cave rescue apparatuses was rebuffed by Thai authorities. It would be ironic and quite sad if Elon’s newly re-imagined Twitter is unable to shutdown the very abuse he “jokingly” accused British cavers of committing.
Failure to appropriately respond to disturbing user submitted content and forged surveillance requests could have dire consequences for the business its customers and especially advertisers. The departure of Twitter’s SRE teams poses a grave liability threat to the business which could easily cause government intervention. Advertisers would be wise to steer well clear of Twitter until such issues are firmly addressed by Musk and his new leadership teams. Sadly, I find it doubtful that Musk’s experience in cars and space transport can rapidly map to this challenge.
Black Twitter
As a Black American technologist, I worry about what the possible sudden degradation of Twitter services may mean for my fellow Black Americans. Twitter has been one of the few places that enabled Black Americans to connect and interact freely at world wide scale. It’s enabled many folks to build brands, businesses and entire careers. I look forward to seeing what comes next for this vibrant community. As much uncertainty as Twitter’s current state represents this also in my mind presents amazing new opportunities for Black owned technology platforms to fill this gap.
Follow and Share
If you found this story interesting please support my work as a Black Technologist by following me on Mastodon or donate to my work on Patreon, GitHub, or Liberpay.
