How Prodio leveraged NodeJs async/await to modify Zerodha APIs — Part 1

Published in

Prodio DesignWorks

5 min readSep 17, 2020

We are building something that helps to manage risk, buckets your capital into meaningful allocations, automates your trade, and monitors your trading actions & decisions for retrospection and future trading decisions.

One of our core and challenging requirement is to process tick data which we receive from Zerodha which publishes tick data on websockets. Zerodha has provided a nice NodeJs SDK to save our coding efforts and simplified the whole process.

Our problem that Zerodha SDK doesn’t solve

Zerodha SDK provides an event “ticks”, which can trigger a callback function passed to it on every received tick. By definition, we can receive multiple of ticks in a single second. But Zerodha SDK calls that callback function in a synchronous way which could block further execution of thread if we’ve to execute algorithms inside that callback function. This leads to missing ticks, if the post tick arrival processing — whether running some formulae or tracking for strategy — runs longer as number of instruments and number of formulae rise , the next tick will be missed.

e.g. One feature which required immediate solution was threshold tracking. In this feature, whenever price of an instrument crosses certain threshold of profit defined by subscriber in a limited time period (eg. if price of an instrument increases by 5% in 10 days), a notification should be sent to the subscriber.

So — we needed to convert zerodha node sdk into async mode, so we could set our processes running after the tick is received, while our main thread continues to listen for next tick. That allows us almost any amount of time for processing compared to earlier 500 ms that’s generally there between Zerodha ticks.

//Sample code for logging frequency of tick data from zerodha using zerodha SDKconst KiteTicker = require("kiteconnect").KiteTicker;
const ticker = new KiteTicker({ 
    api_key: "api_key", 
    access_token: "access_token" 
});ticker.connect();
ticker.on("ticks", onTicks);
ticker.on("connect", subscribe);
var start = process.hrtime();function onTicks(tick) {
  elapsedTime("TICK RECEIVED IN:"); // first log may give incorrect result
  console.log("TICK DATA RECEIVED FOR INSTRUMENTS: ", tick.length);
}function subscribe() {
 const items = [738561, ........];
 ticker.subscribe(items);
 ticker.setMode(ticker.modeFull, items);
}function elapsedTime(note){
   const precision = 3;
   const elapsed = process.hrtime(start)[1] / 1000000;
   console.log(note + elapsed.toFixed(precision) + " ms");
   start = process.hrtime(); // reset the timer
}//Tick data logsTICK RECEIVED IN: 545.586 ms
TICK DATA RECEIVED FOR INSTRUMENTS: 60TICK RECEIVED IN: 600.785 ms
TICK DATA RECEIVED FOR INSTRUMENTS: 75TICK RECEIVED IN: 614.124 ms
TICK DATA RECEIVED FOR INSTRUMENTS: 60TICK RECEIVED IN: 712.715 ms
TICK DATA RECEIVED FOR INSTRUMENTS: 43TICK RECEIVED IN: 510.640 ms
TICK DATA RECEIVED FOR INSTRUMENTS: 29TICK RECEIVED IN: 582.587 ms
TICK DATA RECEIVED FOR INSTRUMENTS: 36TICK RECEIVED IN: 552.164 ms
TICK DATA RECEIVED FOR INSTRUMENTS: 52

What were our options:
1. Multithreaded programming language like golang, python.
2. Some way to make NodeJs more performant

Analysis:
1. Adding one more programming language in tech stack was not a viable option for us. Also, these programming languages had their own learning language which means developers had to spent more time in learning new language rather than solving important problems, implementing crucial features and contributing to the product.
So, we rejected this option.

2. We’d to figure some way to execute all algorithms parallely without blocking NodeJs event loop. So, we decided to solve this problem leveraging NodeJs’s async await feature. We wrapped the onTicks function with async op, this async keyword tells NodeJs to execute this operation or algorithm asynchronously without waiting or blocking next operation.

ticker.on("connect", subscribe);
ticker.on("ticks", 
    async (tick) => ( await onTicks(tick))
);async function onTicks(tick) {
  // logic
}

In this onTicks function, we could split multiple operations like persisting ticks in db, executing algorithms, etc.

ticker.on("connect", subscribe);
ticker.on("ticks", 
    async (tick) => ( await onTicks(tick))
);async function onTicks(ticks) {
  const ps = [saveTick(ticks), splitAndProcess(ticks)];
  await Promise.all(ps);
  return 'success';
}async function saveTick(tick) {
 //send to sqs or REST API call
}async function thresholdTrackingAlgo(tick) {
 // algo logic
}async function splitAndProcess(ticks) {
  const executions = ticks.map(
    tick => thresholdTrackingAlgo(tick)
  );
  
  const results = await Promise.all(executions);
  return results;
}

Problem 2: Latency in network requests. We didn’t want to waste our precious time in network requests (for fetching tracked instruments current price, last traded price in previous tick, traded volume, volume traded in last 5 mins, etc.)

Options:
1. Blazing fast cache service like redis
2. In-memory caching

Analysis:
1. Redis comes with it’s own learning curve, which being a new technology for us could’ve taken more time for implementing the essential feature. New technology ( or service, tool, etc.) means more infrastructure burden, configuration, and management. Also, syncing tracking list & data between db and Redis.
Communicating with redis via network request would’ve introduced more delay, even though in multiple of milliseconds was a waste of available execution time.
So, we rejected this option for our use case.

2. Major advantage of in-memory caching was it brought down delay in the range of nanoseconds. It boosted I/O operations tremendously. There are already in-memory caching solution available on npm. We choose “node-cache” for it’s simple API’s.

Check this out : https://gist.github.com/jboner/2841832

Conclusion:

We decided to use NodeJs async/await feature along with in-memory caching which executes multiple and parallel processes like persisting ticks, executing strategy on ticks, stop loss monitoring, etc.

On the next article i.e Part 2, I will be adding a bit more to this by providing some results and analytics to show the execution time we saved before next tick arrives and some logging results.This part will also include more advanced use of async/await and running NodeJs in cluster mode across multiple CPU cores.
Note: If you need any clarification catch me as @pawan on prodio slack channel.

How Prodio leveraged NodeJs async/await to modify Zerodha APIs — Part 1

Conclusion:

References:

Written by Pawan Wagh