Blockchains, Smart Contracts, & Scale

To briefly disclaim everything I’m about to write, I’m not a professional in anything regarding blockchains and smart contracts, nor have I studied them at length. I’m only relatively adept in understanding how a computer works, how you can run things, like a blockchain or smart contract system on a computer or set of computers, and the fundamentals of scaling computer systems. With that being said, these are some of my spark-notes on blockchains and smart contracts.

Smart contracts are rather poorly named in the sense that they’re thought of as applications, for drafting up and executing negotiations, and are most likely stored in and accessed through blockchain technologies.

To give you a high-level overview of what blockchain technology is and how it works; it emerged in 2008 out of cryptocurrency, specifically Bitcoin that had a some papers published about it — and that’s really where it all started.

So let’s define what blockchain technology is; blockchain technology is a distributed, decentralized system in which all parties have an entire record of the system’s activities and can view the transactions that took place in that system.

So what does a decentralized and distributed network look like?

In a classic centralized network, you have a central node to which all other nodes in the network connect to.

In a distributed system you might have clusters of nodes with a few links between them. But, if a certain node is knocked offline, you lose the part of the data that was stored on that system.

For a distributed, decentralized solution, in the case of blockchain, all parties have a complete record of the database and system, combined together to model one single database whereby each individual node has a complete copy of all transactions.

You can think of the blockchain system as a distributed ledger or like a transaction based database.

That’s what a blockchain fundamentally is, it stores a ledger of transactions.

Blockchains are also immutable, which is to say; the blockchain can be updated, and new sets of transactions can be added to it — but — once something’s added, that block cannot and should not be changed or mutated any further.

The core technologies that underpin blockchain are hash codes and hash puzzles, Merkle trees, and public-private key cryptography.

In the common parlance of bitcoin you have what is known as a bitcoin wallet, which is actually your public key in terms of your private-public key cryptography. So essentially, in the case of blockchain, you have your wallet (i.e. your public key) and your private key which is how you identify and verify yourself when you make or recieve transactions. An interesting thing about the way bitcoin has implemented on the blockchain technology is that it’s all or nothing, so the total wealth of my current wallet has to be spent in a transaction, so if I’ve got five bitcoins using the Bitcoin technology and I only want to send one to you then I send one coin to you and four back to myself.

So I had to do a form of circular transaction and I basically said that, along with the one bitcoin I’ve sent to you, that I don’t necessarily know you, I don’t know who you are, there’s no reason for us to have met or to even know your name — what I’m really saying is the wallet that is labeled by this public key, the person who owns the corresponding private key to that public key — I’m authorizing them to take ownership of this one Bitcoin and that’s how public private key Encryption is used by means of legitimizing/ authoring transactions — and since I said yes — I have said you can spend this amount I wish transfer the amount to you, and you know that the transaction was initiated by me and not somebody else on the network poking at me.

Other important topics to consider regarding blockchain are hash codes and merkle trees.

For a given input, let’s say I make ABC my input and I put that through hashing algorithms and I get the output, `05BC`.

I only have to change `ABC` marginally, I could just add `D` to it and it will completely change the corresponding hashed code.

You only have to change the input in a small way to get a drastically different hashcode coming out — which will be of the same uniform length, a particular one despite the fact that the input which of a different length — you can think of it like labeling it with a “unique” — I’d use ‘unique’ carefully here because there are hash collisions on occasion, whereby different inputs resolve to the same hash — but you’re aiming toward uniqueness and the idea is you can think of it as like a fixed length signature for a given input.

Now the transactions we talked about earlier are grouped together, and you can think of transactions when they are first published to the network, as existing out there in the internet somewhere and then the job of what you might have heard of as “miners” in blockchain technology is to gather a bunch of these two transactions together form them into a block and begin a process of doing that is what involves hashing and checking that a block it remains immutable and hasn’t been changed.

For anyone who takes on the role of a miner, their job is to grab some of these transactions down from the ether put them into a block. A block has several things in it; a set of transactions for example if A said something to B, B said something to C and so on and so forth …

A block also has the hash value of any previous block that is in the network, and additionally it has something known as a Nonce value, which is just a randomly guessed value, to impact the signature hash of this block.

So you hash the combination of the record of transactions, the hash value of the previous block and this Nonce value to generate a hash code that uniquely labels this block. Now the reason behind the Nonce value is to create a hash puzzle for this final hash code that labels this block.

So what a miner does, is they group transactions together and they puts in random guesses for that Nonce value until this hash value that labels this block satisfies the criteria of whatever the hash puzzle is.

Part of the mathematics of the blockchain network is a way of certifying that a block is legitimate and that it’s agreed upon, and it’s a way of managing the distributed nature of the network because anyone can take any collection of transactions that you like that are sort of out in the open that not yet formed into a block and try and create a block and there’s this sort of mathematical race of a sort of compute power to solve this puzzle.

The first person who solves this puzzle or the first group of people that solve it — they’re the ones that generate a block that goes onto the chain. So it’s a way of managing the fact that everybody is free to try and create blocks to add to the chain.

So what happens then when, say, two people both happen to solve a block at the same moment at precisely the same moment — it is rare, but they happen and in those instances you can get forks in block chains.

The people in the network will see one block added, then go off and try and build blocks on top of that block whereas the people that see a different block being added go off and try and continue the chain in another direction — this structure eventually evolves into what looks like a Merkle tree (or hash tree).

So how is it that everyone has this single agreed upon ledger?

There’s essentially a sort of a democracy throughout the network, that the longest chain is the one that everyone usually agrees that they should build on. Furthermore, the mathematics are such but you’re unlikely to sustain forks for a long period of time — as you know people are looking all the time to update their current ledger to the longest chain.

So as an example, to keep it simple, say we have five people in our network and we have a forked chain.

Imagine we have two competing chains, it only takes three of those out of five to have seen the longer of the two chains for there to be a consesus made by majority vote which would reject any chain that is shorter than the agreed upon chain.

So in the shorter chains, that’s where you’re out of luck — they go back into the ether! They’re not yet codified or incorporated into the blockchain and so the process is not legitimate.

So while block creation is not instantaneous, it’s not too long to wait and at that point there’s a very slim chance that your transaction is going to go back into the ether and then you know to be incorporated in a block later on.

If you are in the unfortunate circumstance where you were past that shorter fork of the blockchain and your transaction hadn’t been accepted And it’s been turned back into the ether then yeah you’re just waiting for another miner to come along and grab that series of transactions you and try and build another block.

So could this mining process go on forever or is there a limit to how many blocks can be created?

At the moment Bitcoins are sort of being introduced to show you through the mining process in that you’re being financially rewarded in Bitcoin, I think at this point in time the creation of a block merits six coins in the wallet of the miner or as is commonly the case now with people grouping computer out together the group of people that are made after sort of mining pool.

At some point after all of the bitcoins have been introduced to the network, you’ll no longer gain financially from mining, so in that regard how you will then gain is through transaction fees — sort of like how PayPal or Venmo take their cuts of consumer transactions.

From Bitcoin’s original conception they’ll only ever intended and designed to be 21 million bitcoins in existence, so theres like some sort of radioactive half-life to controlling the release of those bitcoins and how that’s reflected is through the level of rewards for mining.

So every and every 210,000 blocks that are produced the rewards for mining and creating a block is halved and so you’re introducing half the amount of bitcoins to the network as you were previously and so on and so forth — until eventually all 21 million bitcoins have been introduced to the network.

Now reviewing a little bit about blockchain … a blockchain is a technology, a distributed ledger, and means of validating and seeing a series of transactions such that no party in the network needs to directly trust, or know anybody else.

One of the first applications of this was cryptocurrencies, most notably Bitcoin. Now people have started talking about using this blockchain technology not just for storing sort of transactions of bitcoins or cryptocurrency, but as storing smart contracts and running smart contracts.

But let’s say just because we’re using an application that allows me to pay an Uber driver, surely I’m still going to need Uber’s ridesharing application to get me in touch with whoever’s offering the service (the driver)?

This question introduces something called distributed applications, where in fact anyone could offer Uber’s service; you would have your smart contract that is essentially your business logic for negotiating the transfer of funds for a provided service to, taking the Uber case, the driver and the client.

Then you have your front end — it could be a web page, but equally it could be a desktop or mobile application — like Limewire and Napster something running locally on your machine and anyone is free to create their own front end to a given smart contract application and distribute it in any way they like so it doesn’t have to be running from one particular website or run by one particular organization it could even be put on something called the interplanetary file system or IPFS … which sounds dope.

IPFS is a peer-to-peer network file system, such that it couldn’t actually be taken down or moderated by any one individual in the way a standard website can.

IPFS has the potential at least to completely remove the middleman, the escrow account, because your smart contract ends up performing that function. An escrow is like a third sort of impartial mediating party — it’s the function of Uber. Uber takes your money, from which Uber takes their cut and then pays the driver.

If you have smart contract taking the place of Uber there’s no middle man taking the cut.

It’s really is a technology that represents revolution rather than evolution and It’s something that’s going to allow you to move away from ownership of data or ownership of a service to designing the best interface for that service.

Powerful and robust services can be run on blockchain technology and used by anyone or viewed by anyone so the best written service that performs the job is the one that’s going to be used most!

What determines whether or not an Uber type business will make money is whether you’ve got the most user-friendly, prettiest app to interface with the smart contracts.

So there is absolutely nothing that can be done to stop an open source community, an individual — like a graphic design student from producing the most amazing graphical design interface to replace Uber and distribute their app for free because they can and have the time — assuming they have the smart contract properly setup in a blockchain.

So the drivers would have to say “Right — I roam with this kind of contact,” or they log themselves into three different types of contracts, and then you just find them however you find them?

That’s the other thing there for to consider — the use of different contracts and different business models.

So you could have a smart contract that models a Taxi-like service, like Uber or Lyft, for weekdays and a different one for weekends that offer some benefit to the driver for weekends. They can absolutely make use of both contracts. In fact, you can have the same interface to both contracts. So in terms of the client finding them — it’s seamless and the smart contract writer don’t have to know or care about this.

In terms of the driver, they’re potentially optimizing the way they’re getting paid depending on which contract they’re using. Your smart contract could model a luggage allowance or something like it. Here you end up getting a minivan if you’re taking a whole bunch of people and a bunch of luggage.

Would there be one blockchain that the different contracts all stack their data on or would there be a blockchain for each of the contracts?

There are different blockchain outfits out there at the moment, each with their own ability to express smart contracts — so in fact Bitcoin has been expanded to allow some kind of smart contract capabilities, but it’s not what it was designed for.

The platform that’s really sort of making headway in a big name in this area of smart contracts is something called Ethereum.

Ethereum is a blockchain that is designed not primarily to store cryptocurrency, and you know tokens in transactions, but actually to store a global computer state and a series of computational operations.

Therein Ethereum could be thought of as a very large, very slow global computer stored on the blockchain.

But how does the money work then?

Although the Ethereum blockchain was not primarily a cryptocurrency — you still do pay for things in it with cryptocurrency.

So your Ethereum blockchain stores a global computer state and a series of computation operations, but you py for performing those operations, and you pay in the Ethereum currency which not Bitcoin but Ether, and in fact there’s a translation between the the Ether which is the cryptocurrency and the computational operations which are charged by a guessed rate and there’s a guess to the ether exchange rate.

An example of a distributed application that exists for the Ethereum network is the application, Mist.

You could download this as your frontend, i.e. your means of interfacing with the Ethereum network and the browser which allows you to keep tabs on other distributed applications that are out there and potentially view them.

A service for converting one crypto currency to another called Shapeshift and you can search for it through Mist.

So this Mist application can run locally on your machine as an interface, and it can negotiate when to send transactions when you’re either converting Ether to Bitcoin or vice versa or performing the functions of a given application that you’ve loaded into the app.

So Mist is essentially your gateway to interfacing with smart contracts, which are hosted on the Ethereum network — so then whatever function you call — like a function by sending a transaction is executed eventually on all the nodes in the network in parallel, which is unfortunately a very slow operation.

So your computation, whatever it might be — let’s say adding two numbers together — will be performed by every single node, then agreed upon, and then the result of that aggreement is then added global state of the Ethereum virtual machine global computer is updated.

Now, let’s dive into some issues of scalability with an example of a blockchain service, Bitcoin.

In scalability, two key metrics that we are looking at here are the throughput of the system, that is, what is the maximum rate at which transactions can be handled by the system in terms of writing them to the blockchain, and then the other metric is that of latency, that is from the point when a client submits a transaction to the system, how long it takes until the transaction is written to the blockchain.

Now a lot of people have started to become interested in blockchains because these can be applied to other scenarios which are beyond it’s immediate application to digital currency.

So if you look at the blockchain, one of the key properties of the blockchain that distinguishes it from regular traditional databases is that of public verifiability and public verifiability is enabled by two other properties called integrity and transparency. Two things that are very hard to come by now-a-days.

Now by transparency, I mean that anyone can verify how the state of data has been updated and by integrity we mean that anybody can be sure that the data that they are retrieving is not corrupt and it is correct.

Let’s not forget that systems like Google or Facebook also deal with usually a much greater magnitude of users, and the advantage that these systems have over systems like blockchain is that they are centralized systems that are under the control of a single entity or a consortium of entities while in a blockchain the network is between a fully decentralized set of nodes.

So what does a blockchain really look like?

Well you can think of a blockchain as a linked list of hash pointers.

So a hash pointer is simply a hash of some data.

Now regarding a linked list of hash pointers, in the beginning we need to start from somewhere, so we have sort of a dummy or a padding bit of data called a genesis block and we take the hash of this first block which is included in the header of the next piece of data.

So there is a hash pointer which points to the previous piece of data and new data.

Then imagine that you have another piece of data and the process is replicated — and this is, in essence, the simplest form of blockchain.

Now the hash pointer serves two purposes; the first purpose is that it uniquely identifies the previous block of data, and due to the properties of hashing functions, also confirms that the data has not been corrupted.

So anybody can independently take hash of this data, match it with the hash included in the next piece of data, compare two blocks and if they match then they can be sure that the data has not been corrupted.

There are two kinds of operations that can be performed on a blockchain; one is a read operation and one is a write operation.

Now as you might have guessed, read operations are easy because they simply involve retrieving some data off the blockchain.

However, write operations are more challenging because they can potentially hurt the properties of transparency and integrity that we expect from the blockchain.

So a blockchain has two basic functional components, one is to validate transactions and the other is to write to the blockchain.

Let’s look into the validation of transactions.

What is a transaction?

You can think of a transaction is an operation that changes the state of data that lives on the blockchain.

So let’s look at a very simple example in the context of Bitcoin … A transaction might look something like “Pay ten Bitcoins to Bob”, and this transaction has been generated by Alice.

Now, Alice has two things to prove to the blockchain network, 1) Alice first has to prove that she owns these ten Bitcoins so she is able to pay them to Bob.

Which means that as input, she also has to provide the id of the block in which she became the owner of these ten Bitcoins.

The other thing that Alice has to prove is that it’s actually her generating this transaction and not somebody else who is pretending to be to be her.

For this purpose she takes advantage of a cryptographic primitive called a digital signature.

So she basically signs this whole thing with her private key she then submits this transaction to the network.

Next, any node that receives this transaction, if it wants to validate this transaction, it has to perform two checks first to check that Alice actually received ten Bitcoins at some point in the history of the blockchain.

So if you remember the structure of the blockchain was something like this …

From the “head” of the blockchain, the transaction will traverse the blockchain and at some point it will reach the block where Alice came into the ownership of these ten Bitcoins. So the first check has been passed. The other check that it has to do ip traverse back in the blockchain and confirm that Alice actually possesses these ten Bitcoins and that up until now Alice has not tried to pay these ten Bitcoins to anybody else because then that would be considered a double spend. Imagine if Alice has already paid out these ten Bitcoins to Charlie, then she’s not the current owner of these ten Bitcoins even though she was at some point in the past.

So that’s another thing that the node has to confirm; that from the point when these ten Bitcoins came into the ownership of Alice she did not at any point transfer them to anybody else.

If the transaction passes both these checks then the next step is to write it to the blockchain! Dope.

Now this would be straightforward in a centralized system where there are all trusted nodes and they’ll just simply perform the right operation. However this is significantly more complex in a fully decentralized network like that in the blockchain, because all the untrusted or semi trusted nodes have to reach an agreement among themselves — whether they are going to write this piece of data or transaction to the blockchain or not.

This is where the issue of consensus comes into the picture.

Consensus simply means to reach agreement about a value among multiple nodes within a network.

Now in the case of Bitcoin this consensus is done by getting all the nodes to solve a hash puzzle or provide some proof of work.

This is also called mining, and for this reason potential Bitcoin leaders are also called miners.

A miner can also be thought of as a leader because when the miner has successfully solved the puzzle it can write to the blockchain.

There are two main issues in Bitcoin (however, note that even though I mentioned these issues in the context of Bitcoin these are shared by all blockchain systems that work, along the same principles as Bitcoin); the first issue is that of forks now Bitcoin is decentralized system and it’s a peer-to-peer network where different nodes communicate transactions and other information to each other using a gossip protocol.

Now the network has its own limitations, for example; propagation delays whereby transactions could simply get lost on its way, or a node could be simply bad. This could mean that the nodes in the network could potentially have different views of the system.

What might happen is that a minor builds on top of this block, while another minor who has not yet heard about this block, builds on top of some other block and then the blockchain has effectively been forked and some nodes will continue to extend one part of the blockchain while others would continue to mine on top or extend the other part of the blockchain.

This is really bad for blockchain and it defies the fundamental philosophy of the blockchain which wants to provide a consistent view of the world to its users. So all systems need to have some mechanism to resolve such conflicts, in the case of Bitcoin there is this implicit rule that nodes who find themselves in a fork situation should follow the longest fork.

So a gossip protocol is a more naive kind of protocol where for example if there are five people in the room and they want to communicate some information to each other — then one person tell some information to as many people they know in the room and all those people will also communicate the same information to everybody else in the room. So basically everybody is talking to everybody else.

The other issue in Bitcoin is that of performance now there are two key metrics that are used to assess the performance of such systems, one is that of throughput and the other is that of latency (remember these?).

By throughput, we mean what is the maximum rate at which the system can write transactions to the blockchain, and by latency we mean that after the point at which a node hears about a new transaction how long does it take to actually write it to the blockchain.

Now in Bitcoin the throughput of the system is basically a function of two properties of the system — block size and the other is the inter-block interval.

But what is a block?

We’ve been talking about transactions so far, a block is simply an extension of that idea.

You package a number of transactions into a block.

So what gets written to the blockchain is not a single transaction but rather a block of transactions. The size of this block in Bitcoin is currently capped at one megabyte.

The inter-block interval sort of relates to a difficulty of the hashing puzzle, because that has to do with how fast miners can solve the puzzle.

Currently this is set at about 10 minutes and the idea is that the inter-block interval should be large enough that the majority of the nodes in the network have heard about a new block that has been added to the blockchain, before another block is added to it to avoid forks.

With the current block size of 1 megabyte and an inter block interval of 10 minutes the throughput of Bitcoin is at is capped at about 7 transactions per second, and a client that submits a transaction to Bitcoin, has to wait on average for about six more blocks to be added to the blockchain to be sure that it’s block is actually permanently added to the blockchain. Scary right!

Now this is very slow just to give you an idea main stream payment processing systems like Visa which can process transactions up to at up to 24000 transactions per second with a latency of just a few seconds.

Now one might argue that by re-parameterizing the inter-block interval value and the size of the block, maybe these performance metrics of Bitcoin can be improved, but it turns out that recent experiments and forks of Bitcoin that the best we can do right now is 27 transactions per second and a latency of about 12 seconds.

People have now started to look at options to redesign blockchains to address these issues of forking and low performance.

Now note that the issue of consensus is not new it has been studied for well over two decades in classical distributed systems and there are well known and well tested protocols that enable agreement among a group of nodes that can tolerate two kinds of errors.

There are crash failures and there are byzantine failures the protocols that are well known for crash failures are for example raft, paxos, and so forth.

By crash failures, we mean that a node simply becomes silent for example because it crashed and is not able to actively participate in the consensus protocol. Byzantine failures are worse in this case the nodes might behave arbitrarily or inconsistently — for example sometimes it might exhibit the behavior of crash nodes by becoming silent or sometimes it might lie about the value on which consensus is required.

The most popular protocol for tolerating byzantine failures is called the practical byzantine fault tolerant protocol or PBFT.

So what do these classical consensus protocols have to do with Bitcoin or blockchains in general?

Recall that in Bitcoin the way to reach consensus, was that a miner solves a puzzle — so basically it receives the transaction it does mining, solves a puzzle and then gets to write the transaction to the blockchain. What some researchers are currently looking at is what if we replace a leader with a group of leaders so instead of having one leader who is a miner, we have a group of leaders who are not really mining, well the criteria for entering this group of leaders could include mining, but once they enter this committee or group, they are not really mining.

So what this group does is when it receives a transaction it basically runs a PBFT protocol among the nodes and then they reach an agreement whether or not they are going to write this transaction to the blockchain.

So if they decide to write it to the blockchain then it’s a collective decision of all the nodes in this group and it is unlikely for a majority of these nodes to be corrupt and do wrong things with the blockchain.

So how would we know there’s not some massive conspiracy going on here?

This is one of the research challenges in this area right now because in classical systems the entity that controls the network already knows the nodes so this is not an issue in the first place. But with decentralized networks, identity is a huge issue! Who do you allow to join these elite group of leaders? One way is to create a cost for entering the group by getting nodes to do work — whether by mining, but more recently, this can also be a proof of stake and the trade-off here really is that we want the nodes who want to enter the committee to show some sort of commitment that they’re not going to misbehave either by making sure that they have used their resources, for example by mining, or they have sufficient stakes in the system that they wouldn’t want to misbehave for their own benefit or cost.

So this might be good way of preventing/ limiting the amount of folks that happen — so would this it make blockchain any faster?

Well it does to a certain extent because we no longer have to worry about an inter-block interval to make sure that forks do not happen because once the committee has reached a decision that we are going to write this block to the blockchain. Then that’s it — it would be written to the blockchain for sure.

However there is a limit to how much performance gains we can achieve to the system because this committee is still the bottleneck really.

Every node is handling every single transaction, so it doesn’t really scale.

In a scalable system when you add more nodes to the network it should ideally improve the throughput of the system.

However, the problem with these single committee systems is that because of the way PBFT works, and because of messaging complexity and the amount of complexity involved in reaching an agreement; the more nodes you add to this network, the transaction throughput decreases (slows down).

Obviously this is an issue.

It’s really an open research area right now, and a classical way to improve the scalability of a system at least in the context of traditional databases is that of sharding.

By sharding we mean that you split the state or the data among multiple committees or groups within the network.

So basically, instead of having a single committee you have multiple committees and then you make these committees responsible for managing transactions that have to do with data stored on clusters of nodes.

Essentially you have split the transactions among multiple committees.

So not every node is handling every single transaction anymore but this brings its own unique challenges because this is an application to a setting for which blockchain wasn’t really written for in the first place.

So for example, the idea is also that each one of these committees have their own blockchain, for example; if there is a transaction that takes as an input one piece of data from one blockchain, and one piece of data from another blockchain … etc … which committee is going to handle this transaction now!?

Blockchains might not end up as something quite like the internet, however in one form or the other they are probably here to stay but there’s still a long path between where we are right now, up until the point where they are as widespread as say, for example, traditional datastores.