The Roadmap to Permanence, v2
Editor’s note: One of the most exciting features that blockchain unlocks for journalism is the ability to store journalistic content in decentralized environments. This makes it prohibitively difficult for censorship-prone governments or other third parties to remove content with which they disagree, as it lives simultaneously across many independent nodes around the world. This post is an update to the first post on this topic, covering Civil’s progress towards achieving a permanent storage solution for its Newsrooms.
Civil uses blockchain technology to enable a journalism protocol upon which curation, licensing, archiving, identity and membership incentives are built and secured. More broadly, this enables a self-governance mechanism that ensures that only ethical publications (as defined by the Civil Constitution) are allowed. We have covered our plans for self-governance via the Civil Registry, and over the coming weeks we plan to explain each area of the protocol. Today, I would like to dig into our vision for archiving, permanence and censorship resistance. We will get a little technical, but not too deep — so I hope the non-technical folks stick around!
Why is permanence important?
Journalism adds so much value to our society: it helps maintain an informed electorate, informs and deepens within communities and holds people in power accountable. We need to ensure this work remains available even when publications go away. This became readily apparent in November 2017 when DNAinfo and Gothamist were shut down and their years of archives were threatened with extinction (for a deeper dive on this topic, read this informative CJR piece from Popula’s Maria Bustillos) . There are places throughout the world where people in power actively censor content that does not align with their worldview. This can happen through brute force, or getting “sued out of existence”.
The Civil Protocol allows publications to ensure their content remains available even if they shut down or censored. We call this “billionaire proofing.” We plan to do this using a decentralized file storage system called the “Inter-planetary File System.” IPFS is very similar to BitTorrent, where you have peers that can “seed” content (called “pinning” in IPFS parlance) and anybody on the network can access this content by communicating with their peers on the network (or via an HTTP gateway). Check out this great article written by Consensys for a comprehensive overview. If you are interested in running an IPFS node, the folks at Textile are writing some amazing tutorials.
Permanence at Launch
Our journey to a fully decentralized, permanent archive of Civil content will consist of a number of phases. Our goal is to get as many peers to “pin” content as possible (ie, store on their harddrive) — this increases the redundancy and durability. Our first phase will use a hosted IPFS provider, Infura. You can think of them like the Amazon Web Services (AWS) of the blockchain world. They run hosted Ethereum and IPFS nodes. At launch, every article published to the blockchain will also be put on IPFS. However, it will only be pinned by a single party. This is similar to BitTorrent where there is only a single Seeder for a torrent, but anybody can access the content and pin it themselves if they so choose.
Introducing the Civil Archiver
Our current development focus is building what we are calling the “Civil Archiver”. This is a software program that anybody can run that tells the program what hashes (i.e., Civil Content) to pin. Our initial implementation will use an internal API that gets a list of all Civil hashes from our Crawler API (stayed tuned for more information about this in a couple weeks). We will then run this in a couple different geographically dispersed compute zones.
At the end of this phase, all Civil content will be pinned by both Infura and Civil (in multiple compute zones). It is redundant and decentralized, however only two parties are maintaining it. We will have our initial implementation of the Civil Archiver, but at this point is not very configurable — it will pin all content published to Civil and will rely on internal APIs.
Extend Pinning Strategies
Our next effort will be to get more parties to pin content. We will need to implement more strategies for the Archiver to selectively pin content based on the content metadata, since not everybody will want to pin all of the content. They might have a smaller disk or might only be interested in certain content relevant to them. Some examples of pinning strategies that we could support would include:
- only pin for a certain Newsroom smart contract
- pin everything with “science” category / tag in the metadata
- pin the most recent data up to 1TB in disk space
- don’t pin any images
- pin based on geography (India, Florida, Latin America, etc)
- etc etc
Ultimately, Civil is on its way towards activating a global network of universities, nonprofits, and companies that support journalism archives, low-internet accessibility and free-speech advocacy (see a list of Civil’s current partners here).
A Fully Decentralized Future
When we have a backbone of nodes pinning the content, we can start to build IPFS native clients that are very censorship resistant. We would need to extend the Civil Archiver to build in p2p functionality (see libp2p). The main nodes would serve the bulk of the storage and gossip to discover hashes without having to use a centralized API. Clients (cell phones, etc) would also serve as nodes on the network to act as a cache to other clients around them. This wouldn’t need to go over HTTP, and could be encrypted end-to-end.
Summary
The Civil Protocol is uniquely suited to help preserve journalistic archives using a decentralized file storage. We intend to do this through a phased approach, becoming more decentralized over time. While our initial focus is on IPFS, our protocol supports any content-addressable URL scheme, such as Swarm.
Stay tuned over the coming weeks for more details into our plans for curation, licensing, archiving, identity, and membership incentives. We will start with a high level overview, and then deep dive into the tech. Next up is a look into how we are consuming blockchain events using Go to feed our web APIs.
Are you interested in supporting our effort to maintain journalism archives and ensure its permanence? Maybe you have some spare disk space on a server? Reach out to us on Telegram, or send an email to developers@civil.co.
