libp2p-pubsub Peer Discovery with Kademlia DHT

libp2p gossipsub

(λx.x)eranga

Published in

Effectz.AI

5 min readMay 15, 2022

Background

In my previous post I have discuss about building peer-to-peer publisher/subscriber(pubsub) system with libp2p-pubsub. The implementation done with the gossipsub of go-libp2p-pubsub library. Main concern I have had with that application is, it cannot discover the peers behind the NAT. That means it cannot connect to the peers behind the NAT. For an example, if I run one pubsub peer in my local machine and other one in AWS cloud. The gossipsub messages does not deliver between these peers. The reason is the multicast DNS(mDNS) peer discovery algorithm I have used. To overcome this peer discovery issue we can use Distributed Hash Table(DHT)-based approach. In this post I’m goanna discuss about libp2p-pubsub peer discovery with DHT. All the source codes which related to this post available in gitlab. Please clone the repo and continue the post.

libp2p Peer Discovery

After creating a host, how are hosts going to discover each other? There are two options available within libp2p: multicast DNS (mDNS) and a Distributed Hash Table (DHT). mDNS sends a multicast UDP message on port 5353, announcing its presence. This is for example used by Apple Bonjour or by printers. It works on local networks, but of course it doesn’t work over the internet. A DHT can be used to discover peers as well. When a peer joins a DHT, it can use the key-value store to announce it presence and to find other peers in the network. The key used for announcing its presence is called the rendezvous-point.

There are two major differences between using mDNS or a DHT for discovering peers. The first one I mentioned already, mDNS doesn’t work over the internet, where a DHT does. The second difference is that a DHT requires bootstrapping nodes. Other nodes can join the network by connecting to a bootstrapping node, and then discover the rest of the network.

Scenario

I have used libp2p-pubsub library to build the Librumchain which is highly scalable, lightweight blockchain storage. In Librumchain there is a core consensus group which runs on cloud-based nodes with Proof-of-Authority consensus. The edge nodes designed to run on Raspberry-Pi-based lightweight nodes. The core consensus group generates blocks and store them on IPFS-Cluster(IPFS-Cluster has been used as the block storage). The generated block information(e.g IPFS hash of the block) will be published to Raspberry-Pi edge nodes via libp2p gossipsub. Following figure described the architecture of the Librumchain with Core consensus group, IPFS-Cluster and Raspberry-Pi edge nodes.

In my previous post, I have developed this application with libp2p gossipsub using mDNS peer discovery method. It only worked in local network and did not work with the peers in public internet. So I have incorporated DHT peer discovery method to solve that issue. In the pubsub application, the publisher peer take messages from command line and push them to a topic called librum. Then the subscribers of that librum topic receive the data via peer-to-peer gossipsub. The peers discover other peers in the network using DHT method. So it can work over the internet. This is a simplified version of Librumchain’s pubsub system.

Kademlia DHT

libp2p DHT uses Kademlia DHT. In following program I have created Kademlia DHT registered bootstrap peers on DHT. One main thing to notice here is on line 17–18. It added an option to instruct the peer that, in case no bootstrap peers are provided, it should go into server mode. In server mode, it acts as a bootstrapping node, allowing other peers to join it. Sometimes you may need to enable UPnP in the configuration of the router to make this work. In case you want to join the global Kademlia DHT of libp2p, you can use the bootstrap peers in dht.DefaultBootstrapPeers.

Peer Discovery

Then I have create another program to discover the peers in the network using previously created DHT and rendezvous peer discovery protocol. Rendezvous protocol is intended to provide a lightweight mechanism for generalized peer discovery. It can be used for bootstrap purposes, real time peer discovery, application specific routing, and so on. Any node implementing the rendezvous protocol can act as a rendezvous point, allowing the discovery of relevant peers in a decentralized fashion.

In here, the FindPeers() function provides all the peers that have been discovered at the rendezvous-point. Since the node itself is also part of the discovered peers, it needs to be filtered out. For all the other peers, check if they are connected already, if not, then Dial them to create a connection.

gossipsub Publisher

There are two types of publisher peers in the network, bootstrapping peers(public peers) and normal peers(peers behind the NAT). As I mentioned above, the DHT bootstrap functions added an option to instruct the peer to acts as a bootstrapping node, where allowing other peers to join it. To keep things simple I have created these two publishers in separate programs(you could easily create it on single program). One publisher created empty bootstrap list, so it act as a bootstrapping peer. Other created with the bootstrapping peer’s IPFS address. Following are the golang implementation of publishers. I have added comments to describe function of each and every line in the code. Publisher take messages via listening to the command line and publish to the topic.

gossipsub Subscriber

Similar to the publishers, I have created two types of subscriber peers to keep the things simple, bootstrapping peers(public peers) and normal peers(peers behind the NAT). One subscriber created empty bootstrap list, so it act as a bootstrapping peer. Other created with the bootstrapping peer’s IPFS address. Following are the golang implementation of subscribers. I have added comments to describe function of each and every line in the code.

Run Application

Following is the way to build and run the application. I have run bootstrapping publisher and subscriber on AWS cloud with public IP. Normal publisher and subscriber run on local machine. The publishing data will be available for the subscribers in real-time. The gossipsub communication happens successfully through the internet using DHT peer discovery method.

Following is the output. First terminal tab is the bootstrap subscriber peer which runs on AWS machine. Second terminal tab runs local subscriber peer. Third terminal tab runs local publisher peer. When publishing messages from the local publisher, they will available in both local subscriber and AWS subscribers.