Why decentralized social services fail
(yet so far)
A tale of how and why
The other day, I read about another decentralized social network, all filled with great ideas, strong privacy properties, interesting cryptographic defenses and a mission to finally disrupt everything social as we know it. Seeing new ones, once in a while, I see plenty of ways in which they fail, but the discussion around it almost always misses the reason “why”, focusing on “how” instead (which is merely stating the obvious).
The “why” is not technical. It often feels like distributed and federated services provide better resiliency, risk management, and privacy than typical star-topology systems. However, Facebook et al. provide and maintain huge, single-point-of-entry, solely owned, walled gardens with hostile privacy policies and yet they successfully serve millions of users, attracting some of the best engineering talent along the way.
So, why do decentralized social services fail?
Because decentralization is hard to implement right and it’s even harder to incentivize it correctly.
It’s not about technology. We’ve been gradually learning about building and maintaining better decentralized systems for half a century— e-mail, USENET, UUCP (bang address notation!), XMPP, IRC, running on top of decentralized protocols on every layer of communication — ARP, OSPF, BGP, etc.
When it comes to modern user-oriented systems, they tend to be star-like. They tend to be served over HTTP, which is semi-decentralized implementation-wise, but quite centralized in ownership and incentives: you can balance your requests infinitely, build redundant infrastructure and CDN, but you head over to facebook.com in the end.
Why is that?
Incentives incentives incentives
Evolution of technology as we know it has never been about efficiency or resiliency. Many things are efficient enough and resilient enough — yet they fade into the forgotten pages of history. Evolution and, more frequently, revolution is about what pushes and rewards the behavior of survivors and produces more survivors: incentives and risk aversion: the crave and the worry.
Television didn’t kill the cinema, cinema didn’t kill theatre: there are still people attending both forms, then there’s much wider audience that is bound to a shiny TV screen. It’s not about the form, or the content that TV is transmitting — it’s about the content consumption scheme and the incentive to build epic infrastructure for delivery: making a single node broadcast uniform data to a number of users, who are sitting there and staring without having to make a conscious choice, which means that their attention can be controlled and wrapped into a nice package.
With the Internet, we kind of ended up in the same place — Facebook didn’t kill IRC. It’s just that everyone joined Facebook, and some left IRC. User experience? Obvious choice. Resiliency? Your Facebook group can’t be taken over by someone clever riding a netsplit.
We’re browser-bound, single-URL dependent users with browser history dominated by power distribution — few dominant locations and a tail of the rest.
The story of The Web
HTTP and surrounding protocols make great stack for border integration — point of simple compatibility between some very complex systems. In this role, it’s efficient. Yet we know that the best systems can be reliably integrated in a deeper fashion than human-oriented document pulling protocol designed for static data sets. Being a border integration protocol, HTTP enables building user experiences for cheap — so cheap that very complex controlled experiences could be wrapped into it.
However, by its very design, the Web incentivizes centralization as much as TV does.
Facebook is a quintessence of the Web’s evolution as we understand it right now. The success of the World Wide Web’s is the what evolution is all about — whatever there is, is used for providing people crave (what they desire and what they avoid, the incentives and the risk aversion), not what is good, fair, efficient or elegant.
Controlled, comfortable and predictable experience attracts users — we all know the magic of surrendering to the magic of the Apple ecosystem, where everything is well thought-out and “seamlessly” integrated (at least if you don’t dig deep enough). Even in federated designs like e-mail, GMail wins, only clients blossom (remember the multitude of ICQ clients back in the 00s?).
After some time, the availability of your social graph starts to weigh in. If your less technically savvy friends, and your Mom, are on Facebook ,— you either act as a luddite recluse, or you’re going to be on Facebook too, no matter how much you dislike it.
<Captain Obvious>
The attention of the audience in the walled harden is as easy to sell as TV. This is very practical — new medium, familiar business. No need to risk rethinking and easy to incentivize through the very design of the “new medium”, as we still call the Internet.
</Captain Obvious>
Resilience?
The intuition behind resilience hides in human body itself: for when your kidney fails, you’ve got another one and it works. You have two eyes, two lungs — plenty of spare parts to survive.
When a node in a decentralized social network fails because of some serious technical issue that takes days to fix, users will flee. If there’s no incentive for operator to operate the node — sooner or later he’ll be like ‘oh screw it’.
On the contrary, if you operate ISP and break BGP peering, you’re excommunicating yourself instantly, can’t provide services, and lose money or availability of your important assets. Because you don’t get to the responsibility of Internet routing ‘for fun’.
Centralized infrastructure is much easier to maintain and control. Having a CDN, load balancing and request routing infrastructure at the front, then routing and managing properly redundant decoupled services in the back is a fine challenge, and it resembles the human body more — the service organs are there in redundant quantities, but there’s one core module that is the bottleneck.
Incentives drive opportunities, not technologies.
Unfortunately for an engineering idealist, capital/impact distribution affects incentives and risk far more than privacy, resiliency and efficiency.
Most people will stay within current ecosystem, and will be absolutely fine supporting star topologies. Some, of course, will flee and create small closed communities around products and services, that are decentralized. I suspect, they will be absolutely beautiful to participate in and will not survive long. I’ve joined a few, because of social graph. But when they fall apart — somehow, I believe, we’ll catch up on Twitter anyway, to figure out where the next safe haven is.
And this leads to a my point.
Brilliant technical solutions are solutions to problems that people have, not problems we want to solve. As soon as there will be a serious problem, which can be solved by decentralized or federated social service — it will be solved graciously, designs are in place already. Until then, we will adapt to the system that is bent towards incentives that are important right now.
