Write your own blockchain and PoW algorithm using Crystal!

This tutorial was influenced from this excellent blog post about writing a basic blockchain using Go. I’ve ported it over to Crystal. I’ll start off with two explanations: A blockchain is essentially a distributed and decentralized database of ordered transactions that uses cryptography to keep this “ledger” secure and immutable. Crystal is a general-purpose, compiled, object-oriented programming language with static type-checking. Side-note: If you’ve ever written any Ruby, you’re going to love Crystal :)

After this tutorial you should have a better understanding of:

• How a blockchain works on a high level
• How hashing works to maintain and secure the chain
• How to implement a simple proof-of-work (PoW) algorithm and what that means
• How blocks are added to a blockchain
• Crystal and what it can do

At the end you will be able to read your blockchain in your browser and on your terminal, as well as create new blocks through HTTP requests! To keep this post simple, please do keep in mind this is not a real/full implementation of a blockchain, but rather an introduction to the fundamentals :) The network aspect of blockchains won’t be touched on at all in here. There’s a link to the full code at the bottom of this post for reference in case anything gets confusing at any point.

Setup

First lets get Crystal installed. I’m on macOS and went with the Homebrew installation. It isn’t supported on Windows yet (although there is a bit of a workaround). You can read about that and get the installation instructions for macOS and Linux distros here. Once you install the compiler you will have a crystal binary to use. I recommend briefly going over using the compiler and you’ll be good to go!

We’ll create a new app with:

crystal init app crystal-blockchain
cd crystal-blockchain

Next lets add Kemal to our dependencies. Kemal is a simple web framework written in Crystal. We’ll use it for reading and writing to our blockchain through HTTP requests. In your shard.yml file add the following

dependencies:
  kemal:
    github: kemalcr/kemal

After that run

shards install

Shards are the standard format for distributing programs and libraries (in a self-contained format called a “shard”). They are very similar to “gems” in Ruby. You will hear me make some comparisons to Ruby throughout this tutorial, as Crystal’s syntax is heavily influenced by Ruby.

What is a block?

Lets create a bare-bones data model for what our blocks will look like. Inside of crystal-blockchain.cr in our Crystal::Blockchain module we’ll add:

Here we’re defining a global blockchain variable which will hold an array of NamedTuple. With each key, we declare it’s type. I bounced between a few different data types and landed on an array of NamedTuple for it’s ease in transforming into JSON when we return the blockchain from a GET request. Below is a diagram to show the relationship between blocks.

Each block stores the previous block’s hash to preserve the integrity of the chain.

This model ensures order and integrity. Next let’s create another file inside of src/called Block.cr. We’ll make this a module, although it just as easily could have been a class. I felt towards the end that for this implementation a module made sense and worked just fine. So in the new Block.cr file we’ll start with:

If you aren’t familiar with Crystal or Ruby yet, I recommend at least skimming through some information on classes and modules.

A module can extend another module and the above extend self code would be the same as writing extend Block. At it’s core, it is simply a way to avoid having to write self. before every method definition in a module. More on that here - which is a link to a Ruby explanation, but unfortunately there isn’t a whole lot of examples out for Crystal yet.

Now we can add our first crystal method.

We’ll add a method called “create” which will create the block:

NamedTuples are immutable so we’ll use .merge to add a new key - value pair. In this case it returns a new NamedTuple that includes the new pair. As you can see we’re calling self.calculate_hash here which we haven’t defined yet. This method will concatenate the block’s data within a string and hash it.

Let’s take a brief look at the fundamentals of one-way cryptographic hashes. Above we used the SHA-256 algorithm. The concept is that given the same input of any length, it will always return the same output. So for example, if you try hashing “Hello World!” you will always get the same output.

I encourage you to mess around with it. Here is a good online tool for it. The other side of that concept is that it’s easy to verify but extremely difficult to reverse engineer.

This saves a enormous amount of space as well. So if you’re using a hexadecimal representation (as we are doing), you could hash “Hello World!” or the entire Library of Congress and you will always get a 256 bit, 64 character long string back. Try changing just one character and notice that the entire hash changes.

These hashes link the blocks and preserve the integrity of the blockchain. If a bad actor were to try to manipulate the data, the hashes would change and therefore break the chain. This is a fantastic video about hashing and mining and I highly recommend watching it if you aren’t already comfortable with it.

Creating a block and viewing the blockchain

Hop back over into your crystal-blockchain.cr and we’ll create our first block and push it into the blockchain array. Inside our module and underneath our blockchain declaration we’ll define an endpoint to hit to read our blockchain in JSON format and then start our server:

We will need to require in Kemal as well as our Block module so at the top of crystal-blockchain.cr add:

require "kemal"
require "./Block"

Go into your terminal and run crystal src/crystal-blockchain.cr to start the server. Open your browser and visit http://localhost:3000 and you should see your blockchain with the first block created! If you aren’t already using a browser extension for viewing JSON, I really like this one. You can find it in the chrome web-store here. Or if you’re used to living in the terminal, just use curl.

Adding blocks to the blockchain

For blocks to be added to the blockchain they must be “mined.” Were going to focus on a Proof-of-Work (PoW) consensus algorithm in this implementation. This algorithm is used to confirm transactions and produce new blocks on the chain. With PoW, miners compete against each other to complete transactions on the network and get rewarded. This proof proves that a miner did spend time and resources to solve the problem. This process regulates the speed at which blocks are created and a blockchain’s digital asset is released by means of block rewards.

As I mentioned in the beginning, this post doesn’t cover any of the networking between nodes, etc. However I think it’s important to mention that there should always only be one explicit set of blocks in the chain at any given time. In the case of conflicts (two nodes both generate block #10) we choose the chain that has the longest number of blocks. This is another spot I recommend digging deeper into if you aren’t already comfortable with it. This has some pretty good information/descriptions on Bitcoin’s mining process.

The way we’re going to implement PoW is a simpler version of Bitcoin’s. Bitcoin does this by making the network’s participants hash combinations of letters and numbers with the block hash until the result contains a specific number of leading “0”s. We will call the number of leading “0”s the difficulty. The combination of letters and numbers that successfully gave us the correct prefix of “0”s will be the nonce.

Example working with the string “Hello, world!”

Notice the last entry above starts with 4 zeroes. If the difficulty is 4, then this is the solution. The nonce here would be “4250” - this is the data hashed with the block that produced 4 leading zeroes. Again, verifying this is as easy as running “Hello, world!4250” through the SHA-256 algorithm, but getting the solution to begin with is extremely difficult. It’s basically a guess and check game.

Let’s write the code!

First add two new key-value pairs to the end of our NamedTuple declaration - difficulty and nonce.

As you can see we’re using a 32 bit integer for our index and difficulty. In Crystal an integer literal has an optional + or - sign, followed by a sequence of digits and underscores, optionally followed by a suffix. If no suffix is present, the literal's type is the lowest between Int32, Int64 and UInt64 in which the number fits.

So out the box for simplicity’s sake our index and difficulty will default to Int32 (we don’t specify a suffix and they start at “0” and “3”). You could argue that for this project we could use a UInt8 for the difficulty, as we wouldn’t ever have an unsigned integer nor an integer greater than 255. In the same way you could argue that we could use an unsigned integer for our index as well. Although a 32 bit unsigned int would probably be well more than enough as long as we aren’t expecting to have more than 4,294,967,295 blocks :) To keep it simple we won’t declare what kind of integer we want to use when we create a NamedTuple and we’ll let it decide and give us our Int32 which will work just fine.

Lets not forget we need to do something similar inside our create method. We can set the nonce to an empty string for now and set the difficulty to self.difficulty.

Add a hardcoded difficulty to the Block module to work with. In a functional blockchain this would be adjustable to help regulate the time it takes to add a block to the chain.

Lastly, add #{block[:nonce]} to the last position in the plain_text interpolation inside calculate_hash. This ensures we hash the nonce with our block hash.

Next, the generate method inside Block.cr.

Inside generate, start off by calling create which creates our block. Next we start a loop. Inside the loop we pass the current index of the loop iteration (i) through the to_s (to string) method with an argument of 16 (base 16). This gives us our hexadecimal string. We then merge that hex into our block under the nonce key on every iteration.

We’re almost there!

Now within in the loop we will run the hash of the block with the new nonce and the difficulty through our is_hash_valid? method. Lets define that method on our Block module as well.

This will check against our difficulty that we have the correct number of “0”s at the beginning of the hash.

So now within the loop in generate we run that check and if it comes back true then we’ve found the solution! If not we increment i and do it all again. The commented out sleep call is there for adding some time between loops. You can toggle it on and adjust the time to see the iterations in your console coming through one by one.

Last but not least: POSTing a new block

Jump back into crystal-blockchain.cr and we’ll add our route handler for adding a block.

Here we will handle POST requests to /new-block and strip the content coming from the data key. We create a new block, passing in the last block on the chain and our new data.

Before we add the new block into the chain we will do a check to make sure the block hasn’t been tampered with. Add our new method back in Block.cr.

In here we’ll check for three things:

• Is the new block’s index an increment of one higher from the last block in the chain?
• Is the new block’s previous hash equal to the last block’s hash?
• If we run calculate_hash over the new block, is it equal to the hash stored inside the new block?

So long as it passes these conditions we have successfully created a new block!

See it in action

Let’s start our server and visit http://localhost:3000. You should see your initial block.

JSON representation of the blockchain in browser

Cool, looks like it’s running as it’s supposed to. Now let’s add some blocks. I like to use Postman a lot of the time for testing out API calls.

Here I’m making a POST request to http://localhost:3000/new-block with a JSON body of { "data": "Some new data for the second block" }. If all goes well you should see the newly created block come back inside the response in Postman. I’ve also added some logs so that you can follow in your terminal. It will print out all the unsuccessful attempts it made at mining the new block until it finds the solution. It will then print the successful nonce and the new block.

This is where you can go back and add a delay in the iterations of mining if you want to see them happening a little more slowly in your terminal.

Finally if you go back and view the blockchain in your browser you will see the added blocks.

Notice how the hashes after the initial block all start with three “0”s? You can adjust the difficulty and see those change. Be careful when increasing the difficulty too high though, as you may find it can take a loooong time to find a solution. Also you can now visually see the links between each blocks previous_hash and the previous block’s hash.

Here’s a link to the code in full:

bradford-hamilton/crystal-blockchain

Wooo!

Congrats! You just built your own blockchain with a PoW algorithm in Crystal. Hopefully this has helped build on a foundation for you. Some good next steps would be to dig much deeper into these concepts, learn about the networking portion (P2P broadcasting between nodes, etc), and learn about other consensus algorithms.

What I really want to say is that I highly recommend learning about Ethereum ;) Only joking, you should gravitate to whatever you find the most interesting. I will say that within the blockchain space, I focus the majority of my time on Ethereum. It would be another great next step for you to take after understanding some of the foundations of blockchain.

Follow me on Twitter for content about smart contracts, the Ethereum Virtual Machine, and Dapp development.

Bradford Lamson-Scribner (@LamsonScribner) | Twitter