Making Real-Time Trading App with Golang, Kafka & Websockets: Introduction & Setup.

As an engineer, we constantly wonder how different systems work behind the hood. Since I invest a bit of my earnings on a monthly basis, I have had a fair share of experiences with various trading apps. Trading View, Binance, and Zerodha Kite are some of them.

We will be dividing this blog into 4 Parts:
1. Introduction & Setup
2. Setting up Kafka with Golang.
3. Consumer Service & Websockets.
4. Frontend.

High-Level System Design.

Before we proceed with the code part, let’s have a look at high-level what are we developing, and the tools we will be using.

This article will guide you through the basics of making such a platform with Golang, Kafka, and Websockets. First things first, let’s justify the components we are using.

1. Golang: Golang directly compiles machine code and is simpler to write which makes it a good choice to make systems which required to run on low latency. Sure there can be a debate about choosing other languages, disadvantages of Go, and so on, but for now Go will get the work done.
2. Kafka: A Question arises that since we are using web sockets why not use it between producer and consumer?. 2 major reasons. The first web socket is a bidirectional protocol used for facilitating data exchange. That’s it, it only does that, it never stores data. There can be multiple services that might do a timely analysis of financial data. Distributed architecture is another reason for Kafka. Stock market platforms are usually popular across various countries, which makes Kafka a solid choice over Websocket.
3. Websocket: Now why do we need websocket? I just stated that Kafka is distributed. But here is the issue, it is by default designed to work in a multi-host environment. In Kafka, no one partition can be subscribed by more than one consumer. This means that if we open our app from 2 different tabs, both of them won’t receive the same message but instead will receive messages one by one. Also, not many client-side libraries are made for Kafka due to this reason, especially for the web. In mobile apps, it won’t be an issue since we only have one instance at a time running for a particular app. Using WebSocket makes it a good choice to enable it by default for all the platforms.
4. Web App: We will be using Flutter to create the app with syncfusion charts. I won’t be giving in-depth attention to the front-end app. The purpose is to give a gist of how the overall flow is connected from data source to client.

Setup

Note: From here I will be assuming that as a reader you will be familiar with the basics of the topics we are discussing as well as docker installed on your pc.

Let's get started. We will be using Binance’s open-source web socket connection to feed our data source. The tickers we will be subscribing too:

btcusdt,ethusdt,busdusdt,bnbusdt,ltcusdt,xrpusdt,maticusdt

Now that we have decided on our data source, it’s time to set up Kafka.
I will be using docker images from bitnami.

version: '3.7'
services:
  zookeeper:
    env_file:
    - ./.env
    image: bitnami/zookeeper
    expose:
    - "2181"
    ports:
    - "2181:2181"
  
  kafka:
    image: bitnami/kafka
    env_file:
    - ./.env
    depends_on:
    - zookeeper
    ports:
    - '9092:9092'

We have 2 take 2 services to begin with. Zookeeper and Kafka. For those who are unaware with zookeeper, it’s a centralized cluster management system developed by Apache. Zookeeper is usually used when distributed systems come into play, In the context of Kafka it answers some of the questions

1. For a given topic & partition which broker should handle the publish/subscribe?
2. How many nodes/server instances are available in the cluster?
3. Available topics, data retention settings, etc.

While it is just a gist of it you can read more about it here.

Why are we using Zookeeper, since for testing and development we don’t need multiple nodes?
Kafka can run without a zookeeper, but it is not recommended by Apache to do so in production, so it’s best we start using it right away in similar fashion.

Here is the .env file

KAFKA_HOST=kafka
KAFKA_PORT=9092

#Zookeeper
ALLOW_ANONYMOUS_LOGIN=yes
ZOO_PORT_NUMBER=2181


#Kafka
KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
ALLOW_PLAINTEXT_LISTENER=yes
KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092

TICKERS="btcusdt,ethusdt,busdusdt,bnbusdt,ltcusdt,xrpusdt,maticusdt"

After this do:

sudo docker-compose up -d

This should be enough to set up Kafka & Zookeeper in your local environment.

What’s Next?

We will be starting with the implementation part of producer service in Golang followed by consumer service and frontend app in flutter.