Zilliqa Project Update #23 — Launched Testnet v3 with Public Mining

Published in

Zilliqa — Official Blog

7 min readDec 11, 2018

The last 2 weeks have been very busy for our tech team. We unveiled Testnet v3, codenamed Mao Shan Wang — a major technical milestone for Zilliqa. As far as we know, this is the first fully-fledged testnet in the world to implement network, transaction, and smart contract sharding.

Testnet v3 also opens up Zilliqa’s far more eco-friendly and cost-effective version of mining to the public for testing. Zilliqa’s version of mining only requires GPUs to run at full load for a small fraction of time, as Proof of Work is only used to establish identities. Not only does this mean lower electricity consumption, it also means that miners will be able to dual-mine Zilliqa and Ethereum.

Fun fact: Zilliqa’s testnet has now run about 10% as many transactions as the Ethereum network has to-date.

We are now in feature-freeze for the upcoming Zilliqa mainnet (scheduled to go live on Jan 31, 2019), and Testnet v3 includes the same set of features as the upcoming mainnet, including:

Full-scale implementation of Scilla across the network, whereby new nodes can now also execute contracts. (In the past testnet, Scilla was only implemented on a small set of 20–40 nodes for developer testing.)
New incentive mechanism that rewards nodes (miners) by taking into account their contributions to the consensus protocol, e.g. by counting the number of times they signed in the past few epochs.
New upgrading protocol makes it easy and seamless for nodes to upgrade to the latest version.

You can read up on more details about Testnet v3 here.

As always, please feel free to connect with us in any of our social channels:

Telegram: https://t.me/zilliqachat
Slack: https://invite.zilliqa.com/
Twitter: https://twitter.com/zilliqa
Reddit: https://www.reddit.com/r/zilliqa/
Github: https://github.com/Zilliqa/zilliqa
Gitter: https://gitter.im/Zilliqa/ (Dev-related topics including the Ecosystem Grant)

Previous Events

ETH Singapore (7 December 2018)

We had a great time getting to meet many enthusiastic developers and see many interesting projects during the ETH Singapore hackathon. Yaoqi, our Head of Technology, sat on a panel to discuss blockchain scaling. Edison, our Blockchain Application Developer, also gave a tech talk on smart contract security. Thank you to everyone for coming!

*Yaoqi, Head of Technology, sat on a panel to talk about scaling public blockchains.*

*Edison, our blockchain application developer, presented on what makes smart contract programming different and how to properly ensure security for smart contracts.*

Upcoming Events

Things will be quieting down for the rest of December as activity winds down for the holiday season, and as we double-down on delivering mainnet. We are in the process of building out our January events agenda, and will share more details when they are confirmed!

Tech Updates

Testnet v3 represents the full set of features heading into the mainnet. Two weeks after the launch of Testnet v3, our tech team continues to work on improvements to the code base. Here are some of the more significant ones.

Generalized DataSender Class

We observed from test runs that our assignment algorithm for forwarding messages between the DS committee, shards, and Lookup nodes was rather rudimentary. Additionally, the assignment code was spread out across several functions among the different classes. We have improved on this aspect by creating the DataSender class, which acts as a wrapper for the message forwarding. This new implementation selects nodes to act as message forwarders based on the co-signature of the most recently completed consensus. The fact that these nodes participated in the consensus serves as an indication of their continued liveness and increases the likelihood that the message will be forwarded successfully.

Changes to Transaction Processing, Verification, and Transmission

After observing our testnet perform under stress conditions, we concluded that our practice of buffering transactions with unexpected (i.e., larger) nonce values unnecessarily increased the complexity of handling transactions in a consistent manner across the nodes. This was especially the case when different recovery mechanisms such as view change would be triggered, or when transaction packets from Lookups would be received in an untimely manner. So, instead of maintaining the buffer, we chose to re-populate the current pool of transactions with the ones that would have been buffered previously, thus simplifying the overall workflow of transaction processing.

The verification algorithm was also modified to allow for an adjustable tolerance with respect to the ordering of the transactions proposed by the shard leader. Furthermore, to reduce the likelihood of the leader proposing a block with unexpected transaction ordering, we have modified the transaction packet transmission in such a way that the Lookup nodes send the transactions to the shard leader first. Should the shard backup nodes be missing some of these transactions, we already have code in place whereby the backups can fetch these missing transactions from the leader.

More Restrictive Timestamp Check

Another improvement we have recently made addresses potential security vulnerabilities in the way we verify timestamps across blocks. Previously, any new block proposed by the leader during consensus would be accepted if the timestamp was larger than the previous one in the chain. This time, we have changed the acceptance criterion to be based on the difference between the backup node’s local time and the received timestamp from the leader.

Bounded Message Queues

Our use of boost::queue::push to insert messages to our incoming and outgoing queues allowed the addition of entries beyond the defined size of the queue. This meant resource over-utilization was possible, including the uncontrolled spawning of hundreds of threads. Now we have moved to use boost::queue:bounded_push, which prevents adding new messages beyond the size limit. We have also removed the code that retries message insertion when the limit is reached. In effect, new messages are dropped when the node reaches capacity. We will be testing out this new behavior in the coming weeks, particularly to analyze the resource consumption of nodes in our testnet.

Scilla Interpreter

During the last two weeks, we started working on a versioning system for Scilla. A proper versioning system will allow us to seamlessly make future upgrades to the language. Details on how we handle Scilla versioning is available in the language docs.

We have also been working on documenting scilla-checker — — a tool that works as a compiler frontend, parsing the contract and performing a number of static checks including typechecking. To read more about how scilla-checker works and how to add your own static checks and analyses, jump to the relevant section in the language docs.

A summary of the other tasks that we finished is given below:

Fixed a bunch of crashes when handling large Maps and Lists.
Made JSON output of scilla-checker consistent with the interpreter’s output.
Updated the output format for _accepted (which conveys if a transition accepted money).
Corresponding updates to the C++ codebase to read this new format.
Fixes in the C++ codebase to handle null messages by a transition.
Generalized ADT construction in JSON parsing, so that it can work with custom ADTs too.

Developer Tools and Libraries

We have spent most of our time shoring up our libraries and conducting internal tests on our testnet.

In particular, we’ve invested a significant amount of time into making our Schnorr signature scheme implementation in @zilliqa-js/crypto more robust by incorporating suggestions from external security auditors and improving handling of corner cases in line with the C++ codebase.

Community members who have written their own version of Schnorr signature scheme in Swift or C# are strongly recommended to check the most recent version of our implementation in JS.

Kaya RPC has also been patched to be compatible with the newer JavaScript libraries. In addition, we have also made enhancements to mimic the Zilliqa blockchain’s server behaviour more accurately. For example, a request can be sent to the Zilliqa blockchain but will not be confirmed. This can happen for a few reasons, one being that the payload nonce is incorrect. In that case, instead of returning an error to the app directly, Kaya RPC will return a transaction hash just like Zilliqa blockchain. Developers will have to query the Kaya RPC to check if their transaction has been confirmed via the GetTransaction method. If verbose mode is enabled, developer can also see the reason for rejection and this will help them to debug more easily.

On another note, we’ve started to ramp up on internal tools to simulate real-world throughput levels for complex smart contracts, with corresponding utilities for extracting and analysing the resulting data.