The Release of the Crust Test
Update: we’ve just (07.11.18) closed v2 of the Crust Test after a huge response from the SAFE Network community — thanks to everybody who got involved!
After outlining the road to SAFE-Fleming, our next major milestone, today we’re happy to announce the release of the first stage: the Crust Test.
In this post, we’ll take a closer look at Crust — what it is, why it’s essential to a decentralised web — and how you can take part. And where better to start than by taking a look into the decentralised web.
The Decentralised, Serverless Web
Many people talk about the decentralised web. There are certain essential characteristics that must be present before any technology even attempts to tackle this goal. In the past, we’ve set out our views in the SAFE Network Fundamentals. But in this context, the requirements are very clear. Regardless of any other barriers, for a network to be truly decentralised, any two computers must be able to securely connect to each other. We then look to the Network itself to determine when it wants this to happen and to set the rules that govern each relationship.
Crust: Connections in Rust
Computers use Crust to speak to each other on the SAFE Network. It’s a generic networking library written in Rust that has been optimised for peer-to-peer communications. Pretty much any time you want to contact another peer and transfer data, the Network relies on Crust.
Multi-Protocol Support
Two of the most prominent transport level protocols used on the Internet today are TCP and UDP. Computers tend to use TCP where reliability of connection is important (perhaps you’re browsing the Web or chatting). UDP on the other hand is used mostly where it doesn’t matter quite so much if experience a bit of packet loss between the parties (for example, you might be watching video or streaming audio).
We’ve built Crust for power and flexibility. Whilst TCP is preferred over UDP, it supports both existing protocols. And what’s more, we’re continuing to look at supporting even more protocols as the Network evolves over time.
And what’s more, it’s not limited to being technology that only runs within a browser.
Making Contact
So how does one computer connect directly with another? In today’s networked world, it’s not as easy as it sounds. Most devices (mobile, laptops, tablets etc) connect to the internet via an intermediary, like a Wifi router. Although the router address is public, your device address is likely to be private — which is clearly an issue for any P2P Network trying to enable direct connections. To use a phone analogy, how can one node call another if its phone number is not even publicly accessible?
There are three ways in essence: manual port forwarding, IGD Protocol and hole punching.
Manual port forwarding requires a certain level of technical knowledge. You log into your Router dashboard and manually instruct it to direct all data that comes from a specific IP address to a specific port number pair. Hardly straightforward — and irrespective of the complexity or potential security risk, it represents an extra step for people to stumble at when you’re looking to build a Network that appeals to the greatest number of people possible (to gain the full benefits of decentralisation). IGD (Internet Gateway Device) Protocol, which allows you to programmatically set up port forwarding rules, is similarly complicated.
However, the really exciting alternative is hole punching. Where computers can’t connect directly to each other, they can use Crust to identify each other directly and ‘punch through’ any obstacles to create a direct connection. This is what you might have heard of previously as NAT Traversal. And to see just how simple the MaidSafe team has made it in practice, all you need to do is to download and run the code for the Crust Test that we’re releasing today.
Crust supports hole punching so nodes can connect directly when they might otherwise not be able to. All of these techniques are completely invisible to both Routing (the Network’s system of rules that determines the relationship between nodes) and Vaults (Network storage on your computer).
So we can rely on Crust to make these essential connections. But hold on — because it’s bringing far more to the party than simply being easy-to-set up…
Encryption — An Essential Component
Why build a new system if we simply repeat the mistakes of the past? Any new decentralised Network must have encryption baked in all the way from the very lowest level upwards. On the SAFE Network, everything is informed by three fundamental principles — privacy, security and freedom. Communications between nodes must be fully encrypted. No exceptions. Crust is essential here. So whilst today’s test is focused only on NAT Traversal, you’ll ultimately see that, unlike other projects out there, every single thing that Crust puts on the wire is encrypted.
Joining the Network
But with no central authority, how do you find other nodes to join in the first place? With this Crust Test, you can get a feel for exactly how this works — and (hopefully!) just how simple that process is.
Crust itself doesn’t think in terms of a ‘Network’. It just sees a collection of nodes. The initial connection is known as ‘bootstrapping’. This works in one of three ways. The first is Service Discovery on LAN (in other words, checking the local network to see if there are any other nodes to connect to).
The second method is by using hard-coded contacts. Here, a list of existing nodes are stored in the executable file which the node can try to connect with during bootstrapping. This is being used in the test we’re releasing today as it has the address of the dashboard as a hard-coded contact — meaning that it can send the test results to it.
But the third method — Bootstrap Cache — is the most interesting. Put simply, this describes the way that Crust itself stores a dynamic list of directly-accessible peers that a node can contact if it wants to join the Network. When a successful connection is achieved with another node, this contact is added to the list. If an attempt to connect with a contact on this list fails, it is removed from the list.
Join Us in the Test
We’d now like to understand the number of nodes that will be able to connect using hole punch connections in the real world. So we’d like to invite everyone to join us in testing Crust. You’ll find all the details about how to join in with the Crust Test on the forum post.
Dashboard
At this stage you’ll be able to view what’s going on in the Network and get a feel for the sort of peer-to-peer connections that are being enabled — across a range of geographic locations, router types and devices — by checking out the Dashboard. As a test, we’ve taken the decision to keep the information displayed basic whilst still being informative — so if you have any questions about what you’re seeing, please do just comment, either here or even better on the Forum thread itself.
Thanks for your support. We’re looking forward to seeing as many as possible of you taking part in what is a hugely exciting stage in the development of a truly decentralised, secure internet.
