libp2p-pubsub with Golang

libp2p gossipsub

(λx.x)eranga

Published in

Effectz.AI

4 min readMay 7, 2022

Background

In this post I’m gonna discuss about building peer-to-peer publisher/subscriber(pubsub) system with libp2p-pubsub. The implementation done with go-libp2p-pubsub library. All the source codes which related to this post available in gitlab. Please clone the repo and continue the post.

libp2p

libp2p is a network framework that allows you to write decentralized peer-to-peer applications. It evolved out of IPFS and became an independent project. There are certainly a ton of applications being built using libp2p. IPFS, Parity, Ethereum, Filecoin, Polkadot are some popular examples. libp2p provides flexible solutions for essential peer-to-peer elements like transport, security, peer routing, and content discovery. It consists of a catalogue of modules from which p2p network developers can select and reuse just the protocols they need, while making it easy to upgrade and interoperate between applications. libp2p has implementations in Go, JavaScript, Rust, Python, and C++. The specification of libp2p can be found in its specs repo.

libp2p-pubsub

pubsub is a messaging system that is used to enhance asynchronous communication between senders and receivers. In this system, a node can publish content to a topic while another node can subscribe to each topic of interest. In peer-to-peer pubsub system, peers participate by delivering messages to the groups of interested receivers without reliance on centralized infrastructure.

The libp2p-pubsub has created a number of peer-to-peer pubsub implementations(e.g gossipsub, floodsub, fpisub) that have enabled real time peer-to-peer pubsub application development(read more about these different pubsub implementation from here). In this post I have used libp2p gossipsub to develop the peer-to-peer pubsub application. gossipsub is named after the fact that peers gossip to each other about which messages they have seen and use this information to maintain a message delivery network.

Scenario

I have used libp2p-pubsub library to build the Librumchain which is highly scalable, lightweight blockchain storage. In Librumchain there is a core consensus group which runs on cloud-based nodes with Proof-of-Authority consensus. The edge nodes designed to run on Raspberry-Pi-based lightweight nodes. The core consensus group generates blocks and store them on IPFS-Cluster(IPFS-Cluster has been used as the block storage). The generated block information(e.g IPFS hash of the block) will be published to Raspberry-Pi edge nodes via libp2p gossipsub. Following figure described the architecture of the Librumchain with Core consensus group, IPFS-Cluster and Raspberry-Pi edge nodes.

In this post I have discussed about developing simple pubsub application with libp2p gossipsub. In the pubsub application, the publisher peer take messages from command line and push them to a topic called librum. Then the subscribers of that librum topic receive the data via peer-to-peer gossipsub. This is a simplified version of Librumchain’s pubsub system.

gossipsub Publisher

Following is the golang implementation of publisher. I have added comments to describe function of each and every line in the code. It creates libp2p host, gossipsub router, setup node discovery with mDNS(DiscoveryServiceTag used to discover the peers in the topic network) and create publisher to the librum topic. Publisher take messages via listening to the command line and publish to the topic.

gossipsub Subscriber

Following is the golang implementation of subscriber. I have added comments to describe function of each and every line in the code. It creates libp2p host, gossipsub router, setup node discovery with mDNS(DiscoveryServiceTag used to discover the peers in the topic network) and create subscriber to the librum topic.

Run Application

Following is the way to build and run the application. I have run two subscribers and one publisher. The publishing data will be available for the subscribers in real-time.