The Inside Story of BitTorrent’s Bizarre Collapse
Jessi Hempel
1.3K44

I hosted a conference call a few years ago between a major cell carrier struggling to manage mobile internet service costs for their subscribers, and the BT team. Unfortunately it didn’t go anywhere, because neither side seemed to understand the inherent value of minimizing the aggregate systemic network load required to distribute files to users devices at the edges of the network.

That is to say, sum up all the fronthaul and backhaul required to deliver files to subscribers, estimate your network costs to support the haulage, and then calculate what % of your haulage costs you can eliminate by implementing distributed file management so that your users can pull files from the most geographically adjacent users who already have the files.

Now (finally) orgs are beginning to grasp that it’s cheaper to send 1mb 50 feet than 500 miles, and implementing edge computing and distributed data centers to enable this behavior, but a few years ago, for some reason that (fairly obvious) optimization was escaping people. One might respond, “well, but you won’t save that much per meg.” Okay, so what? A small fraction of a very large number is still a very large number.

I share this is to emphasize I’ve been thinking about the Bittorrent situation, and how to make the technology successful, ever since Cohen published the very first version of BT.

Bittorrent has always tried to live in the wrong place in the stack. As long as you look at Bittorrent as an application, it cannot truly succeed. Bittorrent should not be running on top of the OS, managing files on a case by case basis. Bittorrent is a brilliantly useful file management, identification, and distribution system, and should live inside the OS as a key component of the file system.

If Bittorrent is to be successful outside its current use cases (which, let’s face it, don’t exactly service BT’s interests well), Bittorrent needs to burrow into the file system, file allocation tables, and registry (or the OS specific analog, whatever it may be) of the operating system and hard drive, so that all files resident on a system are natively Bittorrent compatible, and the torrent hash files become equivalent to the registry keys (or, again, whatever structure that specific OS uses).

This would open up a whole new world of possibilities for users. All applications and other forms of files would be distributed by Bittorrent by default, without users having to hash each one to create a .torrent file.

Box, Dropbox, OneDrive, and Google Drive would be irrelevant and unnecessary, as long as a machine with your files is online. (I see these services as a kludge intended to simulate shoving BT down into the file system where it should have sat to begin with.) Heck, you could even colocate your files on the machines of untrusted users, as long as you allocate proportional space on your machine for other users’ information as you need allocated on other users’ machines for your information.

The missing data problem and irreversibility of a properly constructed hash proves it would be impossible for a user to replicate the completed file unless that user had both all fragments of your data, and your keys for decryption, so file security seems to be inherent to most typical attacks because we can conceivably preclude both full possession of data by a non-owner, and possession of keys by a non-owner.

Repairing a broken OS would be one-click trivial; as long as your machine is connected to a network where other machines are running (or serving) the same OS, you could stream the repaired files from that working system. (Yes, this assumes the presence of a lightweight failsafe bootstrap / recovery OS.) Heck, controlling OS files with a hash is a great way to determine if the OS files have been changed, to control those changes, and to reverse them when necessary.

I could be wrong, but I think it would be exceedingly hard to infect a system (or application operating on a system) where the filesystem was managed as a BT style hashtable.

In this model, if you log in to any computer with a BT-compatible file-system, and BOOM — there’s your desktop, your applications, and files, and they can be streamed to you in a maximally efficient manner, on-demand. The files don’t have to come from your personal machine, they just have to come from other machines with the same OS and hardware, piecemeal according to where the specific files are available from. Only your unique personal files would require transport across the entire network, and even that transport would be minimized by using a peer colo system from 2 paragraphs up.

Seeding would be a non-issue, because all files on a machine would be seeded all the time. (Leaving, of course, management over up/down speeds and caps to the user.)

I’ve always held the right place for Bittorrent isn’t on top of the stack, but on bottom. Stop trying to bridge the weaknesses, and instead play to the strengths. Bittorrent is infrastructure, not application. I wish the Bittorrent team would understand this.