Welcome to Streamr’s core dev team update for February 2020. This month, the team made a lot of progress on the Network and Data Unions (formerly Community Products). We recently published a blog post on a research study we conducted, with support from Wilsome, to learn more about user perceptions and expectations regarding the Data Unions (DU) concept. One of the example pilot projects built on top of our DU framework is, of course, Swash — check out their recently released roadmap for 2020. If you are interested in building on top of the DU framework too, or if you have some interesting use-case ideas, feel free to share them in our community forum and maybe receive a grant too ;).
Regarding progress on the Network front, we have recently removed a big roadblock in our long-planned, large-scale, network testing, comprising up to 2048 distinct nodes spread across the globe. The testing returned close to expected performance results, with 20GB of logged metrics to comb through for a more detailed analysis in the coming weeks. We are not yet 100% certain that the entirety of the test was flawless, so we are still analysing the logs to make sure there were no issues or misconfigurations. If that were the case, we may need to run another test with the required fine-tuning. The large network simulation testing was also the final missing piece needed for the long-anticipated Network technical paper. Hopefully, we will be able to share the results with you all in the coming quarter.
Another big update on the Network front is the implementation of WebRTC, to replace the current WebSocket model. Most of the functionality tests passed successfully, except for those related to storage. The next steps are: fixing remaining issues, then abstracting the implementation so it can be a drop-in replacement for the Network, without causing too much disruption. WebRTC offers some nice properties like NAT transversal, native P2P connection functionality, and is built for large scale data streaming.
To make data transport more secure across P2P Broker nodes, we have been also doing cross-client tests for the next stage end-to-end encryption roll-out. The goal is to have a built-in key-exchange and a key rotation mechanism to enforce data streams permissioning, without relying on a centralized entity to coordinate the process. If you are curious to learn more about symmetric/asymmetric encryption and how it is applied to other sectors like VPN, for example, I found this article to be quite informative for non-technical readers too.
On the Data Unions side, we are making progress in multiple areas. Regarding the DU framework roll-out, our developers have been refactoring some of the DU server codes to make it more stable and scalable for the official launch. More specifically, we made sure that heavier calculations like Merkle tree computation would not block the server from receiving other requests. Additionally, we are trying to implement state-replication across multiple DU servers, so that we can handle a much higher concurrent connections load (hoping in the 10k-100k range). Our DU smart contract is currently being audited by external parties to make sure there won’t be any security bugs. Finally, regarding the progress of projects piloting DU, Swash is moving full steam ahead with their recently released roadmap for 2020, and MyDiem’s backend integration with the DU server is progressing well too.
What about token economics? It is not a secret that many community members have been very interested in the token utility incentive for the Streamr ecosystem. At the current stage, I believe most parties, both internal and external, have reached a common agreement that the strongest layer where we can apply a sustainable, long term token utility model is the Network, as an infrastructure service to all apps built on top of it, including Data Unions.
We are happy to announce that we have engaged the BlockScience team to assist us in the initial analysis of potential token economics models we could implement for Streamr. We will be hosting the first workshop at the end of this month. Of course, this is a complex topic that will impact our entire ecosystem, so we probably won’t come up with a solution right after the first session, but this marks the official start.
If you’re a dev interested in the Streamr stack or have some integration ideas, you can join our community-run dev forum here.
As always, thanks for reading.
Network
- All planned experiments (up to network size of 2048 nodes) were run without blockers, producing 20GB of logged metrics for analysis. So far analysis shows close to expected results, but requires additional inspection
- Working on WebRTC: most tests passing except storage-related ones
- Cross-client tests for end-to-end encryption reaching almost MVP stage
- Engaged BlockScience team for token economics study and hosting initial workshop for end of March
Data Unions
- Smart contract security audit is in progress. The whole process to be completed by mid-March
- Frontend work for DU is pretty much finished and in design review
- Data Union server is being improved to answer requests even while a new Merkle tree computation is in progress (making it non-blocking)
- Data Union server needs to be tested and deployed in watcher/validator mode for load balancing of incoming requests
- Backend permission overhaul work is 90% done. Related frontend work is waiting for backend work to complete.
Core app (Engine, Editor, Marketplace, Website)
- Use case pages are nearly ready, just require a design and SEO review
- Uniswap tests on mainnet were successful, pending on frontend sign-off
- New Marketplace v2 smart contract finalised, which adds the option to set a transaction fee.
