The TezEdge node: A year in review
Introducing version 1.0 of the TezEdge node
Every kind of system benefits from increased diversity. In the case of Tezos, creating new node shells will benefit the ecosystem as a whole, increasing the stability of the network, opening the door for further innovation and allowing us to verify that the protocol is unambiguous.
We want bakers, developers and other members of the Tezos community to be able to select from a range of different node shells — each person has their own individual needs that may be best met by a particular implementation.
Since last year, we have been working on the development of the TezEdge node, a new node shell for Tezos that is written entirely in Rust. In this two-part article, we want to introduce the TezEdge node by first recapping our progress from our past year, and in part 2 we will present our plans for the following year.
If you want to try out the node, scroll to the bottom of this article for a quick start guide.
Over the past 12 months, we’ve accomplished the following:
- Secured interoperability between Rust and OCaml in the TezEdge node
- Implemented multi-CPU support for mempool and operation pre-validations
- Created the TezEdge Explorer, a debugger for P2P, RPC, storage and bootstrapping
- Created the TezEdge Sandbox, a fully offline Tezos chain simulator
- Added a Tezos dissector into Wireshark for easy debugging and tracing
- Developed an eBPF-based firewall for defense against malicious attacks
In addition to this, we also published monthly articles detailing our work on the development of the TezEdge node. In these articles, we updated the community on our progress and explained the inner workings of various components of a Tezos node. As we near the conclusion of phase 2 in the development of the node, let’s take a look back and examine what we’ve accomplished over the past 12 months:
The TezEdge Explorer — revealing the messages of the Tezos P2P network
Before we began developing the node itself, we wanted to create robust developer tools that would allow us to view, access and test the various modules of a Tezos node.
For this purpose, we created the TezEdge Explorer, a tool with which we could view the various layers of the Tezos network. We began with the P2P layer.
The TezEdge P2P Explorer is an in-browser tool for everyone who wants to record and view the contents of the messages that are sent through the Tezos P2P network. Bugs arising from bootstrapping and synchronization can be easily tracked down and examined, saving time for developers who want to fix such issues.
The P2P Explorer can display all incoming and outgoing messages between peers, message types and their contents. You can filter by peer id or message type and there is also a search function. The P2P Explorer was the first step in our development of a wider-encompassing TezEdge Explorer.
Read more about how we created these tools in our article.
The TezEdge Debugger — view all messages from the P2P and RPC layers
We then continued by adding a debugger for messages sent across the P2P and RPC layers, which are then displayed in the TezEdge Explorer’s user interface (UI).
The TezEdge Debugger allows you to debug and simulate any state in the node by accessing the P2P network, RPC endpoints and the storage. The Debugger decrypts and (if necessary) deserializes the messages sent via the P2P and RPC layers, and then provides them in a human-readable format in the TezEdge Explorer.
The TezEdge Debugger is node-agnostic, meaning that it will run on any kind of Tezos node, whether it is the native OCaml version, the Rust-based TezEdge node or other alternatives.
Read the full details in our debugger article.
The TezEdge Explorer — Visualizing the flow of data in a Tezos node
Having created these tools, we now wanted to develop a user-friendly interface through which we can visualize this data.
We wanted to be able to quickly find certain events or items within the data. For example, when the node exchanges messages with other peers via the P2P network, we want to know which peers sent a connection message to our node, as well as the message details in its metadata.
In order to quicken the debugging process, we added filters. To create a high performing filter, we need to utilize our own indexes.
Conveniently for us, RocksDB holds its data sorted by its key, which provides us with a functionality that is necessary for fast data filtering. RocksDB contains one more powerful mechanism called column families, which allows us to group data of the same type into their own “named columns”. This way, we can separate data indexes, allowing us to efficiently look up data with specific properties from the database itself.
Here you can read more about how we visualized the data of the Explorer.
The TezEdge Node — A deep dive into the mempool, part 1
Having developed robust debugging tools, we were now ready to begin working on the components of the node itself, starting with the mempool. The purpose of the mempool, short for memory pool, is to temporarily store and manage the movement of new operations before they are validated and baked into blocks.
One of the most important duties of the mempool is to efficiently manage resources in order to prevent accidental overloading or intentional flooding by adversaries. We wanted to ensure that the mempool is resilient to such threats and to halt the propagation of any would-be errors, we utilized the actor model in the design of the node.
The development of the mempool module was a prerequisite for the creation of various chain configurators, including a blockchain sandbox, a developer tool which allows you to run a node without an internet connection.
You can read the full details in our article.
The TezEdge node — A deep dive into the mempool, part 2
To continue in the development of the mempool, we wanted to create an (initially) internal version of the TezEdge sandbox. We utilized the sandbox as a testing environment for the TezEdge implementation of the mempool. Our tests focused on the safety and stability of the mempool, but we also wanted to increase the number of transactions per second it can process.
A sandbox is an environment which developers can fully control. They can set the parameters, create funds for testing purposes and safely test the node’s various modules (including the mempool).
We ran similar consistency tests on the mempool as from our first article, but now we were running both node implementations (the native OCaml node and the Rust-based TezEdge node) in sandbox mode. To confirm the correctness of our implementation, we demonstrated the injection of the first block and an operation into the TezEdge node and the OCaml node.
Through this article, we demonstrated that operations were being injected into the TezEdge node’s mempool via RPCs and then broadcasted to other nodes (including OCaml nodes) across the Tezos network. We utilized continuous integration (CI) tests as proof of this mechanism.
Read more about how the sandbox and our development of the mempool.
Using Wireshark to intercept, analyze and display network data in Tezos
Next, we wanted to find a way to intercept, decrypt and analyze traffic sent across the Tezos P2P network.
For this purpose we utilized Wireshark, a widely-known developer tool. Wireshark utilizes so-called dissectors to analyze various types of protocols. There is a large range of dissectors readily available for Wireshark, but in our case, we had to develop our own dissector for the Tezos P2P network.
We designed a mechanism with which our dissector could analyze the encrypted communication on a Tezos network.
Here you can read more about how exactly the Tezos dissector for Wireshark works in our article.
The TezEdge Sandbox — a fully offline Tezos chain simulator that reveals all interactions between the protocol and the storage.
We continued developing the TezEdge sandbox into a version that could be used by other blockchain developers. By using the sandbox, developers can now create protocols, smart contracts and apps for Tezos while being in full control of a simulated Tezos network, allowing them to adjust various parameters and chain configurations.
Once the first block is injected into the sandbox, a protocol is activated. Users can then view operations in the mempool, bake blocks and see all of the interactions between the protocol and the storage. All of this can be done without a connection to a Tezos network or even the internet.
Click here to read the full article on how the TezEdge sandbox works.
Improving the security and speed at which the TezEdge node calls the Tezos protocol
Next, we focused on improving the interoperability between the TezEdge node (written in Rust) and the native Tezos node (written in OCaml).
The TezEdge node has to regularly call the native Tezos node. However, whenever two languages interact with each other, the boundaries are very often the source of bugs. Foreign code that interacts with OCaml’s runtime must conform to certain requirements to ensure that the program and OCaml’s GC interact well and safely.
For this purpose, we built ocaml-interop, which enforces these rules statically and provides easy conversion between Rust and OCaml values. It has a strong focus on safety without sacrificing performance. You can read the full details of ocaml-interop here.
The TezEdge node — Using multiple CPU cores to pre-validate operations in Tezos
The node’s mempool must constantly manage new operations, either validating and subsequently adding them into blocks which are then applied to the blockchain, or invalidating them. It is crucial that this validation subsystem is accurate, fast and efficient.
To achieve these goals, we had to scale up our validation system, improve its performance and avoid flooding the mempool. We accomplished that by scaling and parallelizing validations through the use of multiple CPU cores.
The main feature is that we can parallelize the pre-validations thanks to having parallelized access to the OCaml FFI runtime through dedicated OCaml FFI runtime pools. Here you can read more about how we implemented multi-CPU support to improve the pre-validation subsystem.
Integrating an eBPF-based firewall into the TezEdge node with multipass validations
A blockchain node has a large amount of incoming traffic that it must sort through. Adversaries may attempt to flood the node with repeated requests, aiming to overload the server and disrupt its service. One example of such an attack is the Distributed Denial of Service (DDoS).
We addressed this issue by implementing a firewall that prevents bad traffic from entering the kernel. For this purpose, we utilized an eXpress Data Path / Extended Berkeley Packet Filters (XDP/eBPF) module that acts as a layer outside of the node itself. This module allows us to run an application in the kernel that acts as a firewall, filtering incoming messages. This prevents bad traffic from entering the system at the earliest step possible.
For more details on our implementation of an eBPF-based firewall, read our full article here.
A deep dive into the Tezos storage — How the blockchain state is stored in the TezEdge node through Merkle trees and Git-like semantics
A blockchain node must store blockchain data in order to function properly. However, the amount of data can be very large, and is constantly growing as new blocks are added to the head of the chain. At the same time we need a quick and reliable way of verifying and accessing the blockchain data.
We cannot go through the entire data to verify its contents, as this would take too much time. Instead, we had to figure out a smarter and more efficient way of storing the data.
In the TezEdge node, blockchain data is stored in the form of Merkle trees, a type of data structure that utilizes hash functions to create a tree-like system of referencing. Due to their use of hash functions, Merkle trees allow for very fast, reliable and efficient verification of data. For more details on how exactly Merkle trees work and how they are implemented in the TezEdge node’s storage, read our article here.
Implementing garbage collection into the storage of the TezEdge node
As mentioned before, a node needs to store blockchain data in order to function properly. There are two options; storing it on a disk or storing it in memory. While hard disks offer much greater capacity for storing data, their drawback is slower performance as the data must be read and processed from the disk. On the other hand, memory has very limited capacities, but offers better performance as data can be quickly retrieved and used.
Therefore if we want to maximize performance, we need to store data in memory. Since memory has a very limited capacity, it must be efficiently managed. We want to avoid running out of memory, as doing may lead to crashes.
This means we need to efficiently manage the memory and remove entries that are no longer necessary for the node’s operation. We addressed this issue by implementing a garbage collector (GC) in the TezEdge node’s storage module. Old unused entries are regularly garbage collected, meaning that they are deleted and the freed up memory is reallocated for new entries.
We attempted multiple approaches in the development of the GC, with the end result being a so-called ‘mark and move’ system of garbage collection. You can read more about the GC in the TezEdge node’s storage here.
Running the TezEdge node
We invite you to try out the TezEdge node by yourself.
You can either check out a demo of the node on the TezEdge website or you can run it by yourself by following the instructions listed below.
The node should be working as intended, but there is still much to be done in regards to optimization and fine-tuning performance. One thing we will definitely improve is the management of resources such as memory and storage.
Quick start guide
- Download the TezEdge source code
# Open shell and type this code into the command line and then press Enter:git clone https://github.com/simplestaking/tezedgecd tezedge
2. Run docker (compose)
# Open shell and type this code into the command line and then press Enter:docker-compose pulldocker-compose up
3. Open your web browser by entering this address into your browser’s URL bar: http://localhost:8080
All feedback is welcome and we look forward to your comments, suggestions and questions regarding the node.
We thank you for your continued support through the past year and we look forward to the next phase of the TezEdge node’s development. In our next article, we will discuss our plans for the upcoming year and provide a sneak peek into phase 3 of the TezEdge node’s development.
We appreciate your time in reading this article. Feedback is always welcome and feel free to contact me directly by email.
If you want to read more about Tezos and the TezEdge node, please subscribe to our Medium, view our documentation or visit our GitHub.
