Benchmarking billions of Cryptocurrency Market Data
Hey folks, our beta is here!
For several months now we embarked on the project of building a small crypto market information processing infrastructure and what we have learned so far is that it has been anything but smooth. Our goal is to observe the crypto market closely in order to find the underlying logic of the price formation in such a fragmented market. We have patiently built and tested several types of solutions and experienced a lot of failures but we finally managed to have a stable and resilient service. Here is the testimony of these few months of work and what we’ve learned from it in substance.
What is a cryptocurrency index or a price data provider?
From the end user's point of view, it’s a website or a platform that lists prices and various data on the capitalization of cryptocurrencies. The most common examples are CoinMarketCap, WorldCoinIndex and CryptoCompare. These websites were primarily used for the instant aggregated price information and ranking associated with market cap calculation — i.e. the current price multiplied by the outstanding supply. In order to provide this information, these services must be able to connect and process data issued by exchanges. Here is the story of our journey :
Trying to connect!
Our specifications were simple: connect, stay connected, receive and process data as fast as possible. To meet those requirements and start as soon as possible, we made the decision to rely on CCXT, a library widely known for the incredible number of exchanges it supports and the strong activity of developers around it. Unfortunately, several factors led us to develop a full in-house solution: high-frequency updates, the growing number of parameters and the still young WebSocket support which required to manage the REST rate limits of each exchange. We really enjoyed using CCXT, this library is quite impressive, perfectly documented and well suited for trader needs but not for us.
Fighting to process!
We had no chance to build a resilient service capable of handling live order books and trades from multiple exchanges using a simple big fat monolithic pattern. We moved to a microservice architecture, a software design pattern which breaks the entire system into smaller, independent, and functional units, each of them is capable to run and scale independently. We were lucky enough to be helped by @obusco who shown us how to start (thanks to him :D) with Node.js, Moleculer, and NATS. The results were great until we hooked up the storage service. The raw data stream was simply too large for any basic DB instance (more than 100GB of data per day for up to 6 exchanges). Despite having a microservice architecture, our storage was as a single master entity that couldn’t fit all our needs.
Here we are :
From now on, we’ve reconsidered the way we store data by bringing it as close as possible to the consuming processes. We provide all the metrics on a 24-hour average (volume aggregates and weighted price averages) compared to the 24 hours before and based on strict timestamped trades. Today we deliver a new way to see the “big picture” crypto market price data. With just a few lines of code, anybody can connect to a cryptocurrency API and execute his own trades, that’s a big difference with traditional finance. But if we want our sector to grow, we will need risk management tools suited for those assets. That’s what we are building: a consolidated reference data service and we hope it to be helpful to you.
If you guys want to try, go to koinju.io and please help us to improve it by answering our very quick survey about you experience on Koinju.
We hope to achieve significant improvement by February, so stay tuned!
