Improving the Storage Layer of Dune Network
During the last months, we spent a lot of time preparing the infrastructure for the launch of Dune Network. Now that the launch was successful, we can start focusing on the integration of some big features in the node. One of them is Ironmin, an improved Storage Layer that should decrease disk space usage and improve I/O performances. We hope to be able to release it in the next weeks, after a lot of testing.
The History of Ironmin
Ironmin was not initially designed for Dune Network, but for Tezos, as a replacement for the current storage layer, Irmin, developed as part of the Mirage project at OCamllabs. We developed Ironmin when we were at OCamlPro, between December 2018 and February 2019, as a specific storage for Irontez, a Tezos-node with extensions dedicated to the deployment of private Tezos networks. However, it was easy to make it compatible with Irmin, so that even public Tezos nodes could benefit from it. After the public announcement, the code was published in a free software license and a few bakers started using it for a few months. Ironmin also featured a fully working garbage collector, to save even more space. Users of Ironmin could choose at runtime between using Irmin and Ironmin, and could even switch between them at any time.
As we were interested in the inclusion of Ironmin into the official Tezos node, we started discussions with Nomadic Labs (a French private research lab owned by the Swiss Tezos Foundation) in January 2019. OCamlPro proposed to integrate the code in Tezos and to maintain it afterwards, for just 90 k Euro (around 100 k USD). Unfortunately, we got no reply from Nomadic Labs in the following 6 months. It was finally bought by Origin Labs for the Dune Network.
The Rebirth of Ironmin
The main challenge with the integration of Ironmin in Dune Network is the support for snapshots. Snapshots were added to Tezos in May 2019, just after we stopped developing Ironmin. Snapshots allow users to start a node by just downloading a recent history of the last few cycles, instead of waiting a few days for the node to synchronize with the network. It’s a very useful feature, but unfortunately implemented with strong assumptions that Irmin is being used as the storage layer. Of course, another important challenge is testing, and for that, we benefit from the whole history of Tezos that we can replay as often as necessary to catch bugs.
At the end, our short term roadmap for Ironmin looks like that:
- Integrate Ironmin in a recent version of Tezos, disable snapshots, and test it a lot…
- Add support in Ironmin for snapshots, and test it a lot…
- Check that the garbage collector is still working, and test it a lot…
- Merge into Dune, and test it a lot…
And we are already half way, here is a comparison of Ironmin with Irmin on the last 600,000 blocks of the current Tezos history:
All these results are quite encouraging, and we hope they will remain the same until the release in the next weeks. Of course, it would be very interesting to compare these results with Irmin2, the new version of Irmin expected to land in Tezos in the next 2 to 4 months. Our plan is to have both Ironmin and Irmin2 available in the Dune node, so that node admins can choose the best option for their setup.
