RightMesh Technical Update: April 15, 2019
Our latest technical update on our pilot projects, testing, and plans
In our last Tech Update, we outlined the various types of testing needed in order to launch a robust platform that operates “at scale”. Today, we would like to give you an update on our progress.
“At Scale”
For those of you who may not have a technical background, we will start by defining “at scale”. Firstly, what we mean by “at scale” is essentially order of magnitude. This is how we refer to the scale of operation of the RightMesh platform. Basically, it means how many zeros are at the end of a number. Like saying “a six-figure salary”, the concern is not so much what the first digit is as the number of zeros that give us a sense of scale.
On one test server, we have launched a RightMesh platform consisting of one million threads (that’s ten to the sixth power, or 10⁶) Each thread does not equal one node in this case; however, it is an important step in ensuring we could operate the network effectively. Anyone who has experienced commercializing distributed carrier-grade “high availability” systems at 10⁶ scale knows this is no easy feat; that’s why estimates are revised and sometimes, as in our case, schedules do slip. It most certainly doesn’t mean we have abandoned our goal; rather we are focused on looking at all kinds of failure modes that could happen at that scale and under heavy loads…system calls that fail, operations that time out, resources that get exhausted, exception cases that arise, and designs that worked fine at tens of nodes that don’t work fine at tens of thousands. Indeed, as expected, we are finding those things, and we are working through them with thoughtful redesign so that, we hope, they won’t arise again as we scale orders of magnitude even further in future.
We would like to address some community suggestions that we just “patch and ship”. We will never do this. We have quantitative release criteria for releasing product onto Testnet and Mainnet. There is no “just ship it” authority. What this means is that we expect the platform to be robust at scale when we release. That is good news not just for token holders, but it is also the expectation of end users and the business partners we are currently discussing pilot projects with.
Transaction Testing
Our method of allowing users to be compensated for sharing bandwidth involving payment channels and RMESH tokens. The global transaction processing capacity of the Ethereum network is 10¹ transactions per second. In order to work within this limitation and perform transactions at a rate of 10⁶ per second, we first establish a channel on-chain and then subsequent exchanges of value occur within that channel. In lay terms, we need to send digitally-signed IOUs back and forth offline, and when one party wants to settle up, then and only then do we need to involve an Ethereum node.
We need to be confident that there is no failure mode in which a transaction could be lost or duplicated. In addition to our own internal design reviews and testing, as well as testing on both Ropsten and Kovan networks, we are having both our pre-existing payment channel contract as well as the off-chain processing code and the Ethereum node communications code audited by an external auditor twice: once before the Soft Mainnet Launch which involves Testnet tokens, and then again just before Mainnet Launch, at which point real RMESH tokens will be at stake. While this has been audited extensively previously and it has not changed significantly since first deployed, we want to ensure things are operating as designed. The first audit is valuable to give us maximum time to address any shortcomings uncovered by the auditor, and the second audit just before Mainnet Launch is for added assurance that the contract and code are sound for launch after fixing any bugs that arise during the beta period. We plan to publish the results of the pre-Mainnet launch audit for the peace of mind of users, partners, and backers. We are heads-down on that testing now.
Data Seller Testing
“Data Seller” is the term we use for devices that currently have a direct connection to an Internet access point. For the Soft Mainnet Launch, we are testing that each Data Seller is able to support 10³ data buyers. Some of these data buyers will be directly connected to the data seller, but others could be multiple phone hops away. Nonetheless, while the system has been designed to support multiple data sellers, we are testing as though all the traffic generated by each of those 10³ data buyers is funneled through a single data seller. In other words, we are testing the resource usage and load placed on the data seller phone in the extreme case. We’re heads down on that too.
Android Testing
We realize we have no control over end users updating their Android operating system or their apps, so we need to anticipate meshes will be made up of devices running all versions of Android from 4.0.3 to current and multiple versions of the RightMesh platform and RightMesh-enabled apps. Therefore, we need to ensure we test that the platform operates robustly across all Android API versions from API 15 to current and between phones running different versions of the RightMesh platform. We also test different manufacturers of phones and having those phones in different battery states and while running a resource-intensive foreground application. While we don’t have enough phones to emulate a full mesh with all variants, we’ve got a broad selection of devices in our collection. This is one of our current main focuses over the next quarter.
Multiple Superpeer Testing
Our private release for the Unicef-funded project only involves a single Superpeer, but by Soft Mainnet Launch, we will need to support multiple Superpeers involved in connecting multiple local meshes. In this case, there will be multiple data buyer devices on multiple different meshes wishing to communicate with one another, and there will be multiple data seller devices and multiple Superpeers involved in the off-chain transaction. We’re working through all the cases and failure modes to ensure that those transactions perform correctly at scale. As above, this is a main area of testing focus over the next quarter.
System Failure Processes
The RightMesh network has been described as “self-healing”. We mean this both in the general sense in that new devices which join the mesh “repair” connections lost as other devices leave the mesh, but also in a more technical sense that the RightMesh service running in the background on each phone is able to detect failures in itself, respawn itself, restore state, and continue operating. End Users should not be aware of system failures in the majority of cases. Of course, restarting after a failure is not the same as a failure never having happened, so we also have the ability to send the crash information to RightMesh for failure diagnosis and repair. Determining all the different failures that could happen in the field is the main purpose of the Soft Mainnet Launch, and having an automatic way of collecting that information (with user permission of course) is fundamental to being able to diagnose and repair the defect.
Load Testing
Throughput performance isn’t a goal for the Soft Mainnet Launch, but robust operation under heavy load is. So, we’re also subjecting the system to load testing. We do expect to find resource exhaustion and latency-related faults, especially during large packet transfers. There will be a performance limit reached, so our goal is for robust system operation with at least moderate throughput for Soft Mainnet Launch. By Mainnet Launch, of course, we will have had time to optimize and address any bottlenecks.
In Summary
All the testing we are doing is incremental in nature. As we find flaws, we fix them with proper engineering discipline (no patchwork quick fixes to just “get it out the door”), and then we retest on all versions of Android to ensure that the failure is gone. Those who have been through this can appreciate the scale of the task.
Finally, to address some comments on Rocket.Chat regarding our outreach activities: we are currently in the phase of market development so, in parallel to tech development, we are working to ensure end users and potential business partners are aware of our technology, ensure we have full understanding of market needs, and confirm the potential of our technology to solve various connectivity problems around the world. Bear in mind we are pushing new technology into the market and for customers for which there are long sales cycles (one to three years), and there is no “market pull” without awareness. This does not, in any way, detract from our development focus and efforts. It is supplementary activity being undertaken by our Product and Communications groups, not our developers.
We will continue to update the community as we progress. Until then, we’re heads down. But, here’s a screen shot that some of you will appreciate…
How You Can Be Involved
- Register for updates here to ensure you are kept in the loop of our Soft Mainnet Launch developments.
- Join us on RocketChat — you can get clarifications, share your own use case ideas with us, and get involved in conversations with other community members.
Stay in Touch
- Rocket.Chat: https://rocketchat.rightmesh.io
- Telegram announcements: https://t.me/rightmeshannouncements
- Website: https://rightmesh.io
- FB: https://www.facebook.com/TheRightMesh/
- Twitter: https://twitter.com/Right_Mesh
- Reddit: https://reddit.com/r/rightmesh
