Announcing Rust IPFS

And Our Development Roadmap

Equilibrium is thrilled to announce our latest effort: Rust IPFS.

With support from Protocol Labs, we will be spearheading development and community efforts on this crucial piece of technology.

Our intents are:

1. To deliver mission-critical, IPFS-spec compliant functionality.
2. To enable Rust developers to utilize IPFS both as a Rust library, or crate, as well as interface with a Rust IPFS node via the HTTP APIs.
3. To utilize, foster, and sustain existing community efforts instead of forking and/or starting from scratch.
4. To maintain Rust IPFS in an official capacity, within reason and financial constraints.
5. To maintain conformance with the IPFS spec, using the testing suite approved and provided by Protocol Labs.

Defining “mission-critical”

It’s likely that most people interact with IPFS either as a “gateway” — an HTTP interface that allows internet users to get data from IPFS — or as an embedded library in their applications.

We chose to focus on the latter, allowing dApp developers to create performant InterPlanetary Linked Data (IPLD) applications that don’t use a lot of resources — a place where Rust inarguably shines.

This means that, at a bare minimum, the following endpoints will need to be in place:

• /id — node instantiation (think ~/.ipfs) and RSA keygen
• /version — reports version information
• /dag — for directed acyclic graph (DAG) operations, the lifeblood of IPLD
• /pubsub — allowing Rust IPFS nodes to communicate with one another
• /swarm — allowing Rust IPFS node to find and connect to others
• /block — interfacing with the underlying blockstore directly
• /refs — allows for introspection, showing data you or a peer has
• /bitswap — keeps a ledger on your node’s sharing activities

By the end of the grant work, spread out over the weeks beginning Q2 2020, we plan on having the above done, in addition to fully transparent reporting on the ipfs/devgrants repo.

The grant team

The grant team, led by Joonas Koivunen and Mark Robert Henderson from Equilibrium, will be handling the above work. We’re also extremely lucky to have David Craven, who originated the rust-ipfs code that we’ll be using as a foundation, joining us as well.

In addition, the grant team can also include… you!

Call for contributors

The landscape is changing fast, and outside of what we’ve scoped out in the grant work, it’s hard to tell what the next steps would be, and that’s where the community (read: you) comes in.

Our needs are extensive — everything from planning to scoping to admin to infrastructure, not to mention the dozen more endpoints and features to implement. You can accelerate this effort with your own, especially if it helps you or your organizations’ work!

Here’s a (non-exhaustive) list of ideas for contributions:

1. Definition of Done for remaining endpoints
2. CLI bindings
3. Flagship Rust IPFS Public Gateway Production Deployment
4. Performance and resource utilization tuning for resource-constrained devices like the Pi Zero
5. Benchmarking using existing IPLD projects such as OrbitDB

Maintaining Equilibrium’s “Definition of Done”

Our aim is to maintain a welcoming and inclusive community, while also keeping a keen eye on quality and standards as we go. Leading by example, all of the contributions by the grant team will assume that:

1. There is a working Rust implementation of the command’s functionality
2. Code is “linted” i.e. code formatting via rustfmt and language idioms via clippy
3. There is an HTTP binding for said command exposed via the IPFS daemon
4. (Optional) There is a CLI command that utilizes either the Rust APIs or the HTTP APIs
5. There are functional and/or unit tests written, and they are passing
6. There is suitable documentation. In our case, this means:
7. Each command has a usage example and API specification
8. Top-level commands with subcommands display usage instructions
9. Rustdoc tests are passing on all code-level comments
10. Differences between Rust’s implementation and Go or JS are explained
11. There are passing conformance tests, approved by Protocol Labs

Obviously all of these requirements will not be entirely relevant for every contribution, but we’ll be aiming as close as possible to these guidelines as we progress through the work.

Survey of community contributions

All of that being said, we’re not just starting from scratch. In addition to our desire to work with the community in the future, we also want to honor past and current contributions that the community has already made.

rs-ipfs/rust-ipfs
This is the groundwork that the final Rust IPFS implementation will be built on. David Craven’s work here gives us a tall pedestal to leap from, including Merkledag (dag-pb) and dag-cbor, a Blockstore trait that gives us an in-memory and filesystem store, and a baseline libp2p implementation to build from.

ipfs-rust/rust-ipld
Another useful component that Craven provided is rust-ipld, which includes dag-cbor on top of a custom encoder and decoder and even multiblock types in its sister project, rust-ipld-collections. Protobuf encoding and decoding are mature and there exists at least three solutions for the project needs with different trade-offs (rust-protobuf, quick-protobuf, prost!). In fact, Protocol Labs has joined this project to build the Rust IPLD implementations

libp2p/rust-libp2p
By far the most robust and production-ready component of the modern Rust peer-to-peer ecosystem is rust-libp2p. If you’re not familiar, libp2p is a delightful piece of functionality that sits on your system and finds creative ways to connect to anybody and anything. From there, it forms a distributed hash table of peer ids, effectively replacing address-based routing in favor of a swarm.

ferriseng/rust-ipfs-api
Although we don’t use this project directly inside our implementation, we wanted to make sure to call this one out, since before our efforts we would use this crate to interact with IPFS nodes from Rust. Trailblazing work here.

seanmonstar/warp
The async story of Rust enabling for example high performance web services is still evolving at great speed but there exists some longer running projects enabling the building of HTTP API as is required to enable testing such as warp.

multiformats/rust-*
These crates are extremely helpful to our efforts and have improved vastly over the last few months. Multihash, cid, multiaddr, and multibase: all foundational data types needed to create higher-level functionality in IPFS, and they’re all here and ready for us to utilize. Great work.

What you can do from here

• Visit the ipfs-rust organization on GitHub
• Read more about this effort on the Protocol Labs blog
• Star and watch the GitHub repo
• Back IPFS Rust on OpenCollective

Equilibrium is a venture studio focused on building core infrastructure for the distributed web.

Interested in joining us? Shoot us a message here.