Terraforming Bitcoin Testnets — Part 3/3

At bloXroute Labs, we are developing a Blockchain Distribution Network (BDN, akin to Akamai for blockchains) in order to reduce propagation times and therefore reduce orphan and uncle block (for ETH) rates in any blockchain network, allowing for throughput increases.

In Part 1, we covered building a basic network of a full node and miner. In Part 2, we talked about Bitcoin Docker images, and simple scripts for managing and interacting with networks. In Part 3, we’ll cover how Bitcoin peering works and how we automated the specification of peering topologies.

Standard Bitcoin Peering

In normal usage, Bitcoin nodes connect to some number of peers, and then discover additional peers through a gossip protocol where each peer tells your node about its peers.

In our tests, it was imperative to achieve as high a throughput as possible before introducing our block distribution network so that we could credibly claim that we are adding throughput beyond what’s already possible.

In order to do that, we experimented with a variety of network topologies ranging from nodes that were directly connected to every other node in the network, to ones where we specified very specific paths from one node to another.

The Peering Topology File

Since we controlled how many miners and clients (non-mining nodes) were spun up with every network, we decided on a file format that allowed us to specify which nodes connected. Here’s an example of a peering topology file:

client1=miner1
client2=miner2
miner1=miner2

This topology tells two non-mining nodes (client1 and client2) to connect to one mining node each (client1 to miner1, client2 to miner2), and it tells the two mining nodes to connect to each other (miner1 to miner 2).

This will form a complete path that looks like client1<=>miner1<=>miner2<=>client2.

The only issue here is that Bitcoin normally wants to aggressively seek out new nodes to improve connections, so just by connecting to miner1, which knew about miner2, we would get a connection from client1 to miner2, which we did not want.

To combat this, we employed Bitcoin’s RPC commands for banning and adding nodes.

The Peering Script

Let’s walk through our boot script for peering.

First, we spin up a network using terraform (Part 1 of this blog series). In the terraform code, we inject two variables: total_miners and total_clients which tell us how many nodes are in our system, and add a bitcoin_peering parameter which points to a path in s3 where we’ve uploaded our file.

Then, in the boot script of each bitcoin node, we download the peering file, which we have read from an AWS instance tag:

if [[ ! -z "$TAG_bitcoin_peering" ]]; then
  result=$(/usr/local/bin/aws s3 cp $TAG_bitcoin_peering .)
  peering_file=$(echo $result | awk '{print $NF}')
fi

Since the file contains a record for each node, we need to find the record for our node:

my_peer_line=$(cat $peering_file | grep "^${TAG_node_name}=")

Now, get the list of peers we should connect to from that.

csv_peerlist=$(echo $my_peer_line | cut -d'=' -f2)
newline_peerlist=$(echo $csv_peerlist | tr "," "\n")

Next, let’s loop through every node in our network and explicitly ban it unless it’s in our peer list. Afterwards we’ll add the ones we want to peer with.

There’s a function we use called dnsLookup because we’re instantiating these networks and are sometimes waiting for nodes to come online. The DNS lookup allows us to wait for a node’s DNS record so we can ban it properly by ip.

After banning all the nodes that aren’t in our peer list, we can now peer with the ones that should be connected:

Let’s Recap

To achieve experimental Bitcoin networks where we controlled the topology, we:

1. Dynamically spun up the network (Part 1)
2. Uploaded a peering topology file to S3
3. Injected the total count of clients and miners into the EC2 instance tags for each instance that’s coming up, as well as the reference to the peering topology.
4. On boot of each instance, we read those tags to get the peering file and total counts.
5. Looped through every node that wasn’t in the peer list and banned it using the RPC commandsetban
6. Looped through every node that was in the peer list and added it using the RPC command addnode

Now we have a Bitcoin network of arbitrary size, arbitrary topology, fully automated for experimental use.

Next, we created a parallel test runner that would allow us to spin up dozens of such networks in parallel, but that’s a blog post for a different day.

For more information on bloXroute Labs follow our project on Twitter @bloXrouteLabs or visit us at bloxroute.com. Curious about how Bitcoin, Ethereum, and other blockchains work on a deep network level? Come work for bloXroute and find out.