Coding a P2P blockchain in Rust: Part 1
This is an attempt to learn how to build a new blockchain that is customised for a specific purpose, with open source libraries. Objective is learning and strengthening of concepts. However beyond that, the main incentive for me is that I get to code in Rust, a programming language that I fell in love with, at first sight, earlier in the year. I also noticed recently that several high-profile blockchain platforms are being quietly built in Rust — Facebook Libra, Hyperledger projects (Sawtooth, Aries, Indy and Ursa), Parity (Ethereum client), and Exonum, so there are indeed several (smart!) guys and gals who have seen the value of Rust in blockchain protocol & core modules development.
As an aside, even though I don’t particularly have a great track record of successfully predicting which tech/language/framework will win in future, it doesn’t stop me from making one more prediction. Mark my words: Rust will rule the world of systems programming and high-performance server development within the next 10 years. If you don’t believe me, contact me after 10 years.
Now with the proclamation out of the way, I need to make some disclaimers.
- The code below is experimental , and is in no way production-ready.
- This is not a Rust tutorial. Readers need basic knowledge of Rust (or any other programming language) to follow the tutorial.
- Feature-wise, this is a very rudimentary blockchain. I will cover P2P networking, blockchain data structures and proof-of-work algorithm. Beyond that, there is a lot more that can be done- adding cryptography, account management, wallet, consensus, data stores, privacy features etc . But how much further I will go to build out additional features, I don’t know yet.
- I welcome suggestions from experienced Rustaceans to improve the code- I’m still learning.
Let’s first define the problem we are trying to solve, which has an impact on the design of blockchain. Three of the major groups of blockchain use-cases are :
- Value transfer: All cryptocurrency , stablecoin & asset tokenization use cases fall under this bucket. Here we require assets to be issued on the blockchain, and ownership changes and latest balances to be tracked.
- Asset monitoring and provenance: Mainly used in supply chain scenarios to monitor asset ownership, location and quality tracking.
- Shared data: This includes scenarios where data needs to be shared among a group of participants , either in a private setting (within an organisation) , in a permissioned setting (like a consortium chain where data is shared among a group of participating organisations) or in a public setting (like a public notary or credential service).
We are going to focus on the third scenario for this tutorial. Here are some sample use cases that are applicable to the shared-data scenario:
- A university wishing to issue degree certificates/credentials for its students (public blockchain)
- An insurance industry consortium that wants to share records of their mutual invoices, claims , approvals and payments in a trustless manner, and for regulatory purposes. (consortium blockchain)
- A corporate that wants to maintain their contractual docs, procurement records and list of business process approvals spanning multiple departments: e.g, procurement, credit, finance, IT, business units etc. immutably. (private blockchain)
In all these cases ,we don’t necessarily need to define an asset with financial value, we are only concerned with recording and sharing data among a group of participants in a manner where no single entity/participant owns the data. So, we can adopt a slightly different , light-weight approach here to design this.
Let’s first define what features of blockchain are needed to implement a simple but immutable shared-data blockchain :
- Define the structure of a basic transaction. In our tutorial, a transaction refers to a piece of shared data. For simplicity, we will just use unstructured text (stored as bytes) to represent any generic type of shared data (eg contract, invoice, credential or business approval ).
- Define how a transaction can be initiated . We will submit new transactions through the console/terminal (CLI).
- Define how the transaction will be confirmed . We will generate a new block per transaction, and the transaction is deemed confirmed soon after it is propagated and accepted by the participating nodes in the p2p network. I derived inspiration from Ganache (ethereum dev blockchain) for this model.
- Define how the same copy of transaction records will be stored by all participating nodes . We will use a P2P network to propagate transactions (encapsulated in blocks) to all participating nodes.
- Define how to protect data from being tampered as it flows across the network. Our nodes will use proof-of-work to mine new blocks. Once the new block is received, each node then verifies the proof-of-work to ensure the block is valid and not tampered with in transit.
- Define who can create new transactions and new blocks . Any node can initiate a transaction and create/mine a new block, so it will be a public blockchain by default. For a new block created, all other nodes can verify if the block is valid, and add it to their own local copy of the blockchain. Note: If you want to restrict access to identified participants, we can add node-level permissioning (optionally), and convert it into a permissioned or private blockchain.
Design constraints for this tutorial:
- There will be no support for account management and digitally signing messages.
- Because there are no named assets to be issued and transferred , a transaction does not represent value transfer, but only a piece of text data that is recorded on the blockchain.
- Transactions will be propagated across the network in clear text, and recipient nodes will accept the transactions without questioning (you see, this is the advantage of writing your own blockchain, because you then get to decide the rules).
- There is no consensus algorithm, i.e., the various nodes don’t have to agree on which is the next block to add to chain.
- Crash-fault tolerance is not built-in , so only the nodes that are online can receive the new blocks as they are created.
- This blockchain will not be tolerant to byzantine faults, but will have instant settlement finality and high transaction throughput (as blocks are created instantaneously).
- We will store the blockchain data in a local file on each node. Ideally we would have the world state stored in a KV database like LevelDB, but since we don’t have any assets for which we need to maintain latest balances, we will store only blockchain transaction log on a local file.
To recap and reiterate , we are building a blockchain because we want to store identical copies of shared data across different nodes on a peer-to-peer network, managed by a common protocol, and ensure the data once stored, is immutable. The participants of this network can include multiple individuals, businesses and institutions spanning across geographical and organisational boundaries.
This is a rather modest but important goal, which cannot be achieved with a regular centralised database.
Now that we know the what and why, let’s go to the how.
Let’s start by defining a few data structures:
pub struct Transaction {
pub transaction_id: String,
pub transaction_timestamp: i64,
pub transaction_details: String,
}pub struct Block {
pub block_number: u64,
block_timestamp: i64,
pub block_nonce: u64,
pub transaction_list: Vec<Transaction>,
previous_block_hash: String,
}
We are defining a simple Transaction struct, which does not have sender and receiver account details. There is a transaction_details field, which can store transaction text. We will use this field to store the shared data.
Similarly, the Block struct does not have a separate header and body, just the header fields (block_number, block_timestamp, previous_block_hash) and and a transaction list. This should suffice for now.
Let’s add the following dependencies to Cargo.toml file:
serde = "1.0.98"serde_json = "1.0.40"serde_derive = "1.0.98"crypto-hash = "0.3.3"chrono = "0.4"
Serde-related crates are for serializing and deserializing block data. SHA256 digest comes from crypto-hash crate. Chrono is for block and transaction timestamps.
We will now define the following methods on the Block struct:
- genesis(): To create the genesis block
2. serialize_block(): Serialize block contents to json format for i/o (network, storage) purposes.
3. generate_hash(): Generate SHA256 hash of a given block.
4. is_block_valid(): Function that checks if the blockhash meets the difficulty level set (i.e. if the block has contains the required number of leading zeros).
5. new(): Create a new block, when a new transaction arrives.
6. mine_new_block(): Do proof-of-work calculations to determine the nonce.
impl Block {
pub fn genesis() -> Self {
let transaction = Transaction {
transaction_id: String::from("1"),
transaction_details: String::from("This is dummy transaction as genesis block has no transactions"),
transaction_timestamp: Utc::now().timestamp(),
};
Block {
block_number: 1,
block_timestamp: Utc::now().timestamp(),
block_nonce: 0,
transaction_list: vec![transaction],
previous_block_hash: String::from("0"),
}
}pub fn serialize_block(&self) -> String {
serde_json::to_string(&self).unwrap()
}pub fn generate_hash(block: &Block) -> String {
hex_digest(Algorithm::SHA256, block.serialize_block().as_bytes())
}pub fn is_block_valid(hash: &str, prefix: &str) -> bool {
hash.starts_with(prefix)
}pub fn new(transactions: Vec<Transaction>, previous_block: &Block) -> Block {
Block {
block_number: previous_block.block_number + 1,
block_timestamp: Utc::now().timestamp(),
block_nonce: 0,
transaction_list: transactions,
previous_block_hash: Self::generate_hash(previous_block),
}
}pub fn mine_new_block(block_candidate: &mut Block, prefix: &str) {
while !Self::is_block_valid(&Self::generate_hash(block_candidate), prefix) {
println!("{}", block_candidate.block_nonce);
block_candidate.block_nonce += 1
}
}
}
Let’s now write some unit tests:
#[cfg(test)]
mod tests {
use super::*;#[test]
fn test_genesis_block() {
//create blockchain
let p2p_bc: Vec<Block> = vec![Block::genesis()];
assert_eq!(p2p_bc[0].block_number , 1);
assert_eq!(p2p_bc[0].transaction_list[0].transaction_details, "This is dummy transaction as genesis block has no transactions");
}#[test]
fn test_new_block() {
let mut p2p_bc: Vec<Block> = vec![Block::genesis()];let new_txn = Transaction {
transaction_id: String::from("1"),
transaction_timestamp: 0,
transaction_details: String::from("Testing a new transaction"),
};
let mut new_block = Block::new(vec![new_txn], &p2p_bc[p2p_bc.len() - 1]);Block::mine_new_block(&mut new_block, &PREFIX);
p2p_bc.push(new_block);assert_eq!(p2p_bc.len(),2);
assert_eq!(p2p_bc[1].transaction_list[0].transaction_details,"Testing a new transaction");
}
}
In Rust , unit tests can be included as part of the source (.rs files). Unit test scripts are indicated by #[cfg(test)] attribute and the individual unit tests are listed within the tests module (mod tests). Here we have declared two unit tests one to test genesis block and the second to test creation of new block.
For test 1 (test_genesis_block), we can instantiate a new blockchain with the genesis block as follows:
let p2p_bc: Vec<Block> = vec![Block::genesis()];We then make two checks for this test- one to check if the block number of genesis block is 1, and the second to check whether the transaction details are stored correctly for the genesis block. (Note: In a ‘real’ public blockchain, the genesis block contains block reward for the miner, but here we don’t have any mining rewards, so we are just adding a dummy transaction).
For the second test (test_new_block) we have to create a new block. for which four steps are needed:
- Create a new transaction
- Instantiate a new block passing in the new transaction struct.
- Mine the new block
- Append the new block to the blockchain.
The four steps are shown below:
- Create a new transaction:
let new_txn = Transaction {
transaction_id: String::from("1"),
transaction_timestamp: 0,
transaction_details: String::from("Testing a new transaction"),
};We are hardcoding the transaction id to “1” as we will have only one transaction per block in this tutorial, for simplicity (like how ganache does it).
2. Create a new block as follows:
let mut new_block = Block::new(vec![new_txn], &p2p_bc[p2p_bc.len() - 1]);3. Mine the new block:
Block::mine_new_block(&mut new_block, &PREFIX);
p2p_bc.push(new_block);Note here, for mine_new_block method, we are passing the newly created block (mutable borrow in rust terminology) to the mining method which will directly update the nonce in new block struct once Proof-of-work is concluded. So there is no return value in this method. There is also a second parameter to this method, &PREFIX, which is a constant specifying the difficulty level for proof-of-work. Declare this constant as follows :
pub const PREFIX: &str = "00";4. Append the newly mined block to the blockchain:
p2p_bc.push(new_block);We can now check with assert_eq! macro whether the transaction details are correctly recorded in the newly created block.
Here is the complete code split across two files — blockchain.rs and main.rs
Create a new project using cargo new <project_name>, and under the src directory, populate the blockchain.rs and main.rs files as below:
Blockchain.rs
extern crate crypto_hash;
extern crate serde_json;
extern crate chrono;use crypto_hash::{hex_digest, Algorithm};
use chrono::prelude::*;#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct Transaction {
pub transaction_id: String,
pub transaction_timestamp: i64,
pub transaction_details: String,
}#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct Block {
pub block_number: u64,
block_timestamp: i64,
pub block_nonce: u64,
pub transaction_list: Vec<Transaction>,
previous_block_hash: String,
}pub const PREFIX: &str = "00";impl Block {
pub fn genesis() -> Self {
let transaction = Transaction {
transaction_id: String::from("1"),
transaction_details: String::from("This is dummy transaction as genesis block has no transactions"),
transaction_timestamp: Utc::now().timestamp(),
};
Block {
block_number: 1,
block_timestamp: Utc::now().timestamp(),
block_nonce: 0,
transaction_list: vec![transaction],
previous_block_hash: String::from("0"),
}
}pub fn serialize_block(&self) -> String {
serde_json::to_string(&self).unwrap()
}pub fn generate_hash(block: &Block) -> String {
hex_digest(Algorithm::SHA256, block.serialize_block().as_bytes())
}pub fn is_block_valid(hash: &str, prefix: &str) -> bool {
hash.starts_with(prefix)
}pub fn new(transactions: Vec<Transaction>, previous_block: &Block) -> Block {
Block {
block_number: previous_block.block_number + 1,
block_timestamp: Utc::now().timestamp(),
block_nonce: 0,
transaction_list: transactions,
previous_block_hash: Self::generate_hash(previous_block),
}
}pub fn mine_new_block(block_candidate: &mut Block, prefix: &str) {
while !Self::is_block_valid(&Self::generate_hash(block_candidate), prefix) {
println!("{}", block_candidate.block_nonce);
block_candidate.block_nonce += 1
}
}
}
main.rs
#[macro_use]
extern crate serde_derive;
extern crate chrono;mod blockchain;use blockchain::*;fn main() {
println!("Welcome to P2P Rust Blockchain experiment");}#[cfg(test)]
mod tests {
use super::*;#[test]
fn test_genesis_block() {
//create blockchain
let p2p_bc: Vec<Block> = vec![Block::genesis()];
assert_eq!(p2p_bc[0].block_number , 1);
assert_eq!(p2p_bc[0].transaction_list[0].transaction_details, "This is dummy transaction as genesis block has no transactions");
}#[test]
fn test_new_block() {
let mut p2p_bc: Vec<Block> = vec![Block::genesis()];let new_txn = Transaction {
transaction_id: String::from("1"),
transaction_timestamp: 0,
transaction_details: String::from("Testing a new transaction"),
};
let mut new_block = Block::new(vec![new_txn], &p2p_bc[p2p_bc.len() - 1]);Block::mine_new_block(&mut new_block, &PREFIX);
p2p_bc.push(new_block);assert_eq!(p2p_bc.len(),2);
assert_eq!(p2p_bc[1].transaction_list[0].transaction_details,"Testing a new transaction");
}
}
From the home directory of project, run Cargo test.
You will get the message that 2 tests have passed successfully.
So far , so good. But where is the P2P part? How can additional nodes be spun up and how will they discover and communicate with each other?
We will look at it in Part 2. (if you don’t see the link here , it means I haven’t yet published it).
If you like what you read, show it with claps. If you didn’t like it, feel free to critique it. If you have any suggestions, leave them in comments section.
Happy Rust-ing!
