We wanted to build a service which can store price data for multiple financial products (like BTC future quoted in USD) and can answer time based price aggregation queries within milliseconds.
We evaluated a bunch of time series databases and decided to finally go ahead with InfluxDB.
- SQL like query language
- Supports auto rollups using Continuous Query
- Comes with clustering support (Enterprise version)
- Easy StatsD integration
Since InfluxDB was going to do most of the major heavy lifting, we decided to go ahead with an express server. Nodejs was also an obvious choice to ingest trades data from our Kafka cluster.
Our trading engine publishes all trades on a Kafka topic. We have a service written in Nodejs which reads data from Kafka topic and writes it to InfluxDB. We define Continuous Queries (CQ) in InfluxDB to create aggregates for certain pre-defined resolutions. So for example, we create a CQ which runs every 5 mins and calculates price aggregates for the last 5 minutes. We choose the following resolutions — 1m, 5m, 15m, 1hr, 6hr, 1day.
Nodejs Service — Kafka to InfluxDB
For a detailed understanding on how to setup InfluxDB, we suggest the reader to go through official InfluxDB docs.