Cloud Storage Solutions, Decentralised Systems, and Decoupled Architectures for Personal Use
I’ve been thinking lately about the implications of cloud storage on the basic Internet model, i.e., open, decentralized services. Cloud storage for personal usage offers several advantages to previous models of personal or home mass storage. Namely, users aren’t limited by their hardware.
Before the proliferation of public cloud storage solutions, consumers were limited to traditional means of mass storage for personal use, such as budget Network Attached Storage devices on their home network, portable storage, for example, USB drives, or even just their individual machine’s hard drives.
As well as the upfront hardware cost limitations made these not necessarily ideal options; issues such as synchronization, backup, and file versioning had to be handled by consumers themselves.
For the average non-technical person, these relatively simple technical tasks weren’t something they could or wanted to do. Also, critically it meant an option long available to enterprise and corporate users such as remote access of files at high data speeds & bandwidth wasn’t available to the public.
The proliferation of publicly available cloud storage remedied that resolving the problems described above. I have my whole personal and home infrastructure built on top of such services, in what I imagine is a fairly typical manner; my personal and business documents are stored with Google Drive, all the media I consume from TV shows & movies to books are stored with Mega, and for sharing data or files with friends I have used Dropbox.
These solutions work great and mean I no longer use or need any physical media at all, such as Blu-Ray discs or even USB sticks with personal videos on to play on our smart TV, and critically have all my data available anywhere on any device quickly at any time.
While they work great, these solutions break a fundamental aspect of the Internet. It was invented way back in the dark days of the Cold War to act as a communication system in the nuclear holocaust, that of decentralisation and communication based on open standards.
To most people, this probably is a complete non-issue; the public cloud storage services work as-is, and such concerns are more philosophical and abstract than affecting day-to-day usage. For a technologist who is building their solutions or even just tinkering, these concerns become concrete.
Building a home solution such as personal document storage based on a public cloud storage service gives you a single point of failure. It also risks tightly coupling your solution to said service.
For a technologist who relies on near real-time communication systems, even for personal use, this isn't good. While nuclear war thankfully isn’t one of our pressing concerns nowadays, suddenly finding that unforeseen problems with cloud storage provider mean your business no longer having today’s accounts, or even just not seeing that the latest episode of your favourite TV show hasn’t synchronised across from your PVR to your Home Theatre PC for you to watch that evening is annoying or in the previous case possibly disastrous.
The above problems could arise simply from issues with our public cloud storage providers synchronisation service. Their storage service and the data itself should still be accessible.
But it may not be if the unforeseen problems affect both since we have no way of knowing how their services are coupled internally. The only way to access our files remotely from their online storage is via their proprietary web interface, which could conceivably be experiencing problems also.
In this event, not only have we lost file synchronisation, but we have also lost data storage since by using a public cloud storage service, the two are tightly coupled together, and the issues described above catastrophize into being unable to access any of your business accounts data, and all your TV shows say.
So what can we do to remedy this while still having all the advantages of cloud storage, such as online synchronisation, high availability, fast connectivity and bandwidth, with large amounts of storage not dependent on our hardware? The first would be to decouple ourselves from mass storage linked to a specific service.
We need to replace the storage capacity of the public cloud services provider. Cheap, widely available Virtual Private Server hosts are ideal for this. They offer large amounts of storage and low compute power at costs about roughly half of what a fully managed cloud storage service provider on a monthly power user plan would cost.
This gives us a large amount of storage accessible at high data speeds and bandwidth comparable to the existing public cloud storage services. It decouples us from the proprietary web interface of the major cloud services for accessing our data directly. We can typically use standards such as FTP, rsync, Samba, NFS, SSHFS, and more to access our storage on our VPS directly.
It’s still missing a key aspect of public cloud storage, namely file synchronisation and sharing across multiple devices or with others. This is perhaps the killer feature of cloud storage and why services like Dropbox became so popular. Files will asynchronously synchronise across devices.
For example, when I turn my laptop on, regardless of where I am, it will automatically download any new or changed files that I modified on my bedroom HTPC the previous night, say regardless of the other device’s availability or being even switched on. An example of sharing would be that I can make a file available for a friend.
An open-source synchronisation solution such as the Syncthing project is ideal for this. It offers free peer-to-peer synchronisation for devices on a local network or across the internet with support for features such as file versioning and support for multiple platforms with a Web GUI for configuration & management. However, this particular solution is only for file synchronisation without the sharing functionality.
Critically since we are decoupled from specific storage, what synchronisation solution we use doesn’t matter. As long as it replicates the functionality we need from the public cloud; we are free to choose our solution from several competing options such as the aforementioned Syncthing, Resilio Sync, or non-peer-to-peer based synchronisation & sharing options such NextCloud, Pydio, and others that have whatever functionality we do need.
Typically the cost to power users will also be cheaper than the public cloud solutions given the affordability of VPS hosts and the open-source nature of the applications used. With peer-to-peer synchronisation and sharing applications in combination with our VPS server storage, we then have our own affordable private cloud storage service that replicates all the functionality of public cloud storage but based on open standards, using a decentralised model, and decoupled from any individual piece such as specific mass storage or synchronisation application which can be swapped in or out depending on if better options become available or removed entirely.
