Paradoxes in Crypto Quant Models: The Retraining Dilemma
Outlier events make the retraining of predictive models in crypto a non-trivial challenge.
In a recent article published in CoinDesk, I outlined some of the key challenges of quant strategies for crypto assets. Creating predictive models and quant strategies for crypto assets is a fascinating challenges and one that present very novel difficulties compared to traditional capital markets. As we have been building more machine learning(ML)-based predictive models at IntoTheBlock, we have encountered several hurtles that fall outside traditional machine learning and quant methodologies. One of those challenges is what I referred to in the article as the “retraining dilemma”.
ML-based predictive models for financial assets such as cryptocurrencies are fundamentally based in supervised learning methods. The essence of supervised learning states that models are trained in a given labeled dataset such as “Bitcoin trades in Coinbase” or “Ethereum orders in Binance” and they learned patterns or features in that dataset with the goal to forecast a specific target value such as price or volatility. Training is not a one-time event and should be performed regularly. Figuring out the correct retraining frequency is an important challenge for any prediction model. Model retraining is one of those aspects that looks very differently in predictive models for crypto assets compared to traditional capital markets.
Suppose that we have a machine learning model that attempts to predict the price of Microsoft’s stock(MSFT). The model has been trained in MSFT historical order book at the NASDAQ for the last decade. Given that the performance of MSFT has been relatively stable compared to market conditions, the rules for retraining our predictive model are based on two fundamental criteria:
1) Frequency-Based: The model could be retrained every few months to capture the knowledge of the latest trading activity.
2) Model-Drift Based: The model could be retrained when its performance starts degrading. This is known in machine learning as model drifting and its illustrated in the following figure:
Those two steps seem relatively straightforward. Either we retrained a model regularly or when its performance degrades. That methodology works incredibly well on predictive models for traditional asset classes but regularly failed when applied to crypto. This is the essence of what we like to call the retraining dilemma.
The Retraining Dilemma
The main reason why regular or drift-based retraining methods works in regular asset classes is because their stability an efficiency. Going back to our MSFT stock example, a predictive model might encounter some outlier events but they should be few and far between and certainly unlikely to have an impact in the long term. Crypto is the exact opposite.
The essence of the retraining dilemma can be illustrated by expanding on our sample scenario and assume that we are training a similar predictive model to our MSFT example but this time to predict the price of Bitcoin based on Coinbase order book records. Just this year, we will encounter outlier events such as the March crash, consistent weeks of no volatility in June-July, a crazy spike in volatility at the end of July followed by another sharp drop in early September. Given the young history of most crypto assets, the sequence of many of these outlier events is relatively new and will affect the performance of most predictive models. At that point, we need to decide whether retraining our predictive model is the appropriate solution.
In the face of an outlier event, the decision of retraining a predictive model is far from trivial. If we proceed to retrain, we are assuming several risks such as overfitting the model for the outlier records or even degrading its performance. Furthermore, there is no way to validate if the model learned anything new given that the outlier even just happened. If we decide to not retrain the model then we are risking to see further performance degradations. Now imagine that our predictive model is faced with these outlier events regularly and you get an idea of the magnitude of the challenge. That’s the essence of the retraining dilemma.
There are no silver bullets to deal with the challenges of the retraining dilemma in crypto asset predictions. At IntoTheBlock, we regularly faced those challenges and have explored different creative solutions which will be the subject of a future post 😊