Building a historical price engine using InfluxDB

We wanted to build a service which can store price data for multiple financial products (like BTC future quoted in USD) and can answer time based price aggregation queries within milliseconds.

Choosing InfluxDB

We evaluated a bunch of time series databases and decided to finally go ahead with InfluxDB.

Pros

1. SQL like query language
2. Supports auto rollups using Continuous Query
3. Comes with clustering support (Enterprise version)
4. Easy StatsD integration

Since InfluxDB was going to do most of the major heavy lifting, we decided to go ahead with an express server. Nodejs was also an obvious choice to ingest trades data from our Kafka cluster.

Setup

Our trading engine publishes all trades on a Kafka topic. We have a service written in Nodejs which reads data from Kafka topic and writes it to InfluxDB. We define Continuous Queries (CQ) in InfluxDB to create aggregates for certain pre-defined resolutions. So for example, we create a CQ which runs every 5 mins and calculates price aggregates for the last 5 minutes. We choose the following resolutions — 1m, 5m, 15m, 1hr, 6hr, 1day.

Nodejs Service — Kafka to InfluxDB

Service to ingest messages from Kafka and insert into InfluxDB

Influx Schema

OHLC data in computed periodically for different resolutions (1m, 5m, 15m …)

For a detailed understanding on how to setup InfluxDB, we suggest the reader to go through official InfluxDB docs.