The Backstage of Backtesting
Written by BAM.money
In our previous article, “Digital Data: Gold mine or Garbage Dump”, we introduce the potential that digital data holds for traders. But how can traders utilize this information?
From a signal standpoint and given all the traps, traders must focus on nowcasting instead of the usual forecasting models. Through nowcasting, traders can get:
- Direct measurements that always hold true because they do not rely on a statistical lead-lag relationship
- Short-range predictions are statistically more reliable than long-range ones, which also implies that most published discoveries or signals in finance are false after a while
As a smart trader, you know the importance of backtesting your trading signals to develop or refine your trading system. The idea is that your trading strategies that may have worked well in the past may stop doing so in the future. If you are smart but less experienced with what machine learning may have brought to the market, you may not have paid attention to how backtesting must change to truly contribute to alpha creation and trading in general. There are two related issues here: (a) machine learning has the power to democratize access to data relationships but on the other hand, (b) financial data has a very low signal to noise ratio.
We define the basic framework in two ways:
- We will identify at least two statistical sets, then run the result on one and compare it to the other.
- We will rely on rational observation: if the two sets are very similar, the results are likely to be very similar. When this happens, the backtesting is not informative.
From this, we now know that we should differentiate the data sets. As we focus on market realities (thanks to the no-arbitrage principle), we should also note that there are no parallel universes in financial markets. From here, we can conclude that financial market data is limited.
5 Common Traps When Backtesting
The most important trap you’ll find when backtesting has to do with economic reasoning. Here are some of the other ones that I’ve found through the years:
1: Using the same historical data or finding combinations that yield statistically significant results (i.e., creating false positives or data snooping)
2: Ignoring rational economic theory — or expecting the same results to hold in the future even if not all conditions are the same.
3: Knowing that the larger the dataset, the more behaviors you can expect, but ignoring research that indicates that Sharpe ratios quickly deteriorate when applying strategies backed by the same historical data.
4: When more and more traders are equipped with the same signals, their trades become very crowded and their signals less potent.
5: Not having guardrails in place to protect you and your trading system. Traders must now keep backtesting and adjusting old signals as often as possible.
Can This Age Of Big Data Predict the Future?
The answer is no. With machine learning, it’s tempting to think that traders can perform miracles and predict the future with all these zettabytes of data and their computing resources. Unfortunately, while you can certainly learn a lot through signals, there is a limit to what you can gain from the data you process. More data does not always mean better forecasts.
In addition, there are no accurate out-of-sample data sets in the financial world. In fact, the only true out-of-sample is live trading. This is why you must backtest dynamically and continuously. Combine this with cross-validation, and you keep track of various combinations of variables.
Are We Ignoring Economic Reasoning?
Economics, unlike many other fields (e.g., physics, medicine), don’t have the luxury of being able to carry out extensive out-of-sample tests. When you do, it’s critical to consider individual behaviors (humanity’s preferences, needs, and attitudes) that change over time. You can develop tests covering different economic regimes and see the direct impact of changes in individual and collective behaviors and their impact on models and data sets.
Read our next article “How data is transforming financial markets” for more insights.
