How to Build a Trading Bot: Step by Step

Day 3: Trading System Lambda Architecture

Today, we are going to move on to our core step: implementing a Lambda Architecture to the trading system.

I will use Kafka as the data streaming layer, probably use Mongodb or Cassandra as the batch layer, some in memory databases or spark as the speed layer. I will ignore the serving layer for now. So let’s begin today’s journey from Kafka.

git hub: https://github.com/bingyangli/kafka-sparkstreaming-cassandra

  1. Terraform+Ansible to manage Kafka + zookeeper dockers.

First of all, you have to install the terrafor and ansible. After the installation, fill out the information in terraform.tfvars.example, and change the name to terraform.tfvars. Then:

git clone https://github.com/bingyangli/kafka-sparkstreaming-cassandra.git
cd kafka-sparkstreaming-cassandra
cd terraform
terraform init
terraform plan
terraform apply

After this, copy your kafka ips to host file under the playbooks folder, then run to get kafka node started:

ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook -i hosts start-network.yml