High Performance and Scalability on a Python Blockchain
Lamden makes it easy and fast to build decentralized applications (dApps). Our vision is to provide what developers want in order to create a large ecosystem of dApps with on-chain assets. Demand for dApps is growing rapidly, and the current blockchain technology is too slow and unable to scale in order to meet today’s demand for on-chain transactions. We not only want to provide a solution to the current deficit in blockchain performance, but provide a future-proof platform that can handle levels of even greater demand for on-chain activity. This is why Lamden’s phenomenal achievements in performance and scalability are foundational to our success.
Have you heard that Lamden is written in Python? That is correct, it is.
Have you also heard that dApps and transactions on Lamden must be slow because Python isn’t as fast as some other languages, like C++? If you have, that is NOT correct! Not at all.
In this article, we’re going to explain why this commonly echoed idea is a misconception by answering these five questions:
- Which programming languages can be used with Lamden?
- What are blockchain throughput, scalability, and latency?
- How is Lamden engineered to provide very high throughput AND very low latency?
- What is Lamden’s max transactions per second (TPS) and why are high throughput and low latency both needed for high performance at scale?
- How is Lamden engineered to scale linearly with node hardware upgrades and when will node operators want to upgrade their hardware?
Which programming languages can be used with Lamden?
Any major programming language can be used to write an application that can integrate with the Lamden blockchain. If a developer wants to code in C++ because they need bleeding edge gaming performance, then that’s what they should use. If they want to use JavaScript, that’s fine too! And if they want to use Python, it’s easy and also highly recommended for certain applications such as finance and data science.
When interacting with the Lamden blockchain, every transaction that occurs on-chain is actually a smart contract written in our native language: Contracting. As a subset of Python, Contracting utilizes the Python Virtual Machine so developers spend less time with complex computer science problems and instead, code much more efficiently. Being natively integrated with Python enables rapid prototyping within any available Python integrated development environment (IDE), as well as access to a large and robust set of widely used Python tools and libraries.
What are blockchain throughput, scalability, and latency?
Throughput is commonly referred to as TPS, or transactions per second. This is the rate at which transactions can be processed by the entire network, per unit of time. By analogy, this is similar to the width of a river. If the water always flows at a fixed speed, then to handle more water within its banks, the river must be widened.
Scalability is the ability of a blockchain protocol to meet increasing levels of time-sensitive demand made by the applications that integrate with it. This includes how a decentralized network handles transaction fees as more and more transactions are pushed through the network. However, for the purposes of this article, we’ll only be looking at how scaling and performance metrics are related. A blockchain network’s scalability is sufficient when it can handle increasing demands for transactions or messaging without suffering significant performance losses. In other words, during a flood, the river can become wider to keep all of the water flowing smoothly within its banks.
Latency is the time it takes for a transaction to be finalized on-chain after it has been initiated. Finalization is important because, on a blockchain, once it has finalized it is irreversible. Latency is like the speed at which water flows in a river. Higher speed means less time flowing from point A to point B.
How is Lamden engineered to provide very high throughput and very low latency?
The blockchain itself is written in Python. Let’s dive into some of the key ways in which Lamden has achieved high throughput and low latency. In order to surpass the competition and offer industry-leading performance, Lamden’s blockchain architects have identified and solved two critical performance bottlenecks that any blockchain engineer would need to solve for:
- How well is the cryptographic “intake valve” flowing?
- How fast is the communication protocol?
The encryption algorithm that is used to verify and secure every transaction can be thought of as the intake valve into the system. This is the first bottleneck. The system’s speed is limited by how long the encryption method takes for computation. An example of a modern public key cryptography method called Elliptic Curve Digital Signature Algorithm (ECDSA) uses elliptic curve cryptography, and it’s pretty fast. It’s faster than the older Rivest-Shamir-Adleman (RSA) method, but it’s still not fast enough to reach Lamden-level speeds on a cryptographically secured blockchain.
Lamden uses ED25519, which satisfies our requirements for a high-speed, high-security signature. If you are interested in a deeper dive into the specs of this signature, more info can be found here: ED25519.
The communication protocol is the other major bottleneck to solve in blockchain engineering. Lamden uses an asynchronous messaging library called ZeroMQ (ZMQ). It’s written in C++, calls C++, acts as a message queueing system, and it uses the ED25519 signature. It’s similar to Apache Kafka (originally developed by LinkedIn) because it’s very fast, but unlike Kafka, ZMQ can be used for decentralized protocols.
With these two primary bottlenecks solved, our consensus method and a bespoke technique for reactive block times are the other key components that make Lamden blazing fast. Our consensus method produces immediate finality for all transactions; there’s no waiting for confirmations to determine probabilistic finality. Nodes react to available work by processing work immediately instead of waiting for a preset block size or time. Lamden achieves transaction finality in less than a second, in addition to very high throughput. This means transaction latency can be as low as 400 milliseconds! With this much optimization baked in, the transaction latency is so low that it’s actually constrained by the time it takes the internet to move and route messages between machines across long distances.
For more details on consensus, official documentation on Lamden’s Delegated Proof of Committee — a Byzantine Fault Tolerant consensus algorithm — will soon be released.
What is Lamden’s TPS, and why are high throughput and low latency both needed for high performance at scale?
Maximum throughput and latency are both very important to consider together because high throughput means that the network can still execute the lowest possible latency when there are high levels of network activity. This interrelationship is indicative of how network performance can scale under load, such as when there’s a wildly popular on-chain decentralized exchange (DEX) handling lots of trading activity.
To illustrate the relationship of throughput and latency, we’ll use the analogy of the river again. If the water level has risen to the top of the river banks, then any additional rainwater that the river can’t contain will make the river overflow. Water that overflows on top of the banks must then “pool” and wait to re-join the roaring rapids of the river. In more technical terms, latency must increase if throughput is maxed out. Transactions that can’t be processed immediately must go into a queue and wait to be worked on.
If max TPS equals X, then latency would increase when the demand on the network, or requests per second (RPS), is greater than X. As RPS increases beyond max TPS, the increase in latency isn’t linear because as long as RPS remains higher than TPS, the transaction queue grows larger and larger, and therefore newer requests will take longer to process through the queue. Instead, as RPS grows beyond max TPS, the ratio of RPS to max TPS needs to be squared to reflect the compounding size of the queue.
A simple, theoretical approximation of this scaling problem is represented by the following logic:
If RPS > TPS, then average latency with high load = (average latency without high load)*(RPS/TPS)²
Let’s look at how this logic applies to blockchain performance using hypothetical numbers. If we assume that Blockchain A’s average latency without high load is 5 seconds, and max TPS is 100, then one could estimate that if RPS > TPS, then average latency = (5 sec)*(RPS/100)².
Therefore:
- If RPS is 2x TPS, or 200, then the average latency is about 20 seconds. The demands on the network doubled its max TPS, and the result is that average latency quadrupled to 20 seconds. This longer latency may still be acceptable for some use cases.
- However, if RPS then increases to 5x TPS, or 500, then average latency slows to over two minutes. This is probably not an acceptable latency for most use cases.
Latency will also increase if it stays at very elevated levels. To return to sustainable levels of network activity, RPS needs to eventually drop (the rain slows and waters on the river banks recede) below the network’s max TPS threshold so that available bandwidth can be restored.
Now, let’s look at Lamden’s scalability using real-life estimates for TPS and latency.
Assumptions:
- All nodes are running 4-core CPUs.
- Max TPS is 3,000 per core, or 12,000 total.
- Average latency is 700ms, which is without heavy load and with an average degree of complexity in the smart contract logic.
- If RPS is equal to TPS, or 12,000, then average latency is still 700ms, because max throughput has not been exceeded.
- If RPS increases to 1.5x TPS, or 18,000, then average latency slows to about 1.6 seconds. This is still very fast and acceptable for many use cases.
- If RPS increases to 2x TPS, or 24,000, then average latency is still only about 2.8 seconds. So, even when the network is close to double its max TPS, Lamden’s latency would still be less than 3 seconds. This is still acceptable in many use cases.
- And even if RPS increases to 3x TPS, or 36,000, then avg latency is STILL only about 6–7 seconds using this model.
This type of scalability is only possible when TPS and latency are both exceptionally performant. This is why having both high TPS and low latency is such an important combination for Lamden’s performance at scale.
How is Lamden engineered to scale linearly with node hardware upgrades and when will node operators want to upgrade their hardware?
Although we’ve already covered many details of Lamden’s architecture, there’s yet another key factor in Lamden’s industry-leading scalability: parallel processing technology is built into the protocol. By running Lamden software, each thread of execution in a node’s CPU is able to process transactions in parallel (at the same time).
This means that a node operator can upgrade to more CPU cores, and Lamden’s parallelization technology enables that node’s max TPS to increase linearly with the increase in core count. As more nodes in the network increase their core count, the network’s max TPS also increases. If all node operators on the network double their CPU core count (such as from 4 to 8 cores), then the network’s max TPS doubles. Yes, that’s right! Lamden’s throughput scales linearly with CPU upgrades to more cores.
When a single node has more CPU cores than the majority of other nodes and the network isn’t under heavy load, then there won’t be much benefit to that node operator spending the extra money for better hardware. But as network activity increases and the network activity approaches the max throughput threshold, then nodes will be incentivized to upgrade their hardware.
For example, if all nodes are running 4-core CPUs and network load starts to regularly exceed max TPS, then nodes are incentivized to upgrade to 6 or 8 cores so they can capture rewards for processing the “pooled” transactions. In other words, since more CPU cores can handle more transactions per second, more performant nodes will receive the so-called overflow, which other nodes can’t process fast enough. Then once more than half of the nodes have upgraded in this fashion, even if the network isn’t consistently under heavy load, the rest of the nodes are incentivized to upgrade just to compete with the other nodes for work. And lastly, if a node really starts to fall behind, they can be removed from the network by the on-chain governance voting them out. Therefore, nodes are incentivized in more than one way to upgrade their hardware when network scaling needs to happen.
Summary
The Lamden team has worked tirelessly to make our blockchain architecture better than we ever imagined. With the upcoming release of our mainnet, we will offer a developer-centric platform with the fast and scalable technology needed to power the next phase of blockchain adoption.
To summarize some of the technical accomplishments for our Python-based blockchain:
- Transaction latency is 400–1000ms under normal load.
- Transaction latency is so low that even when the prevailing max throughput is exceeded and latency is forced to increase, the network still performs at a speed that is acceptable for a wide variety of applications, and is still faster than the majority of other blockchains.
- Max TPS is approximately 3,000 per CPU core.
- Total network throughput scales as nodes upgrade their hardware and CPU core count increases.
Mainnet launches September 16, 2020! You can view a mainnet countdown timer at lamden.io.
Lamden is faster, cheaper, easier, and BETTER.
Thanks for reading!
For more information, please visit:
