The Journey of a Quantitative Trading Strategy Development (Part 5)
A no-code overview
This article is the part 5 of a series of articles on quant trading. My goal is to present the steps involved in a quant trading project, in a simplified and non-programmatic way. In part 1 we defined our hypothesis, a market benchmark to which we can compare our trading results, and structured our data to use it in the following stages. Part 2 presented the initial backtests and some hidden costs in quant trading. We analyzed a few portfolio metrics to understand more details of our strategy in Part 3. Parametric analysis and overfitting were the topics of Part 4. The scripts and frameworks developed to this series are on my Github.
- Part 1 — Hypothesis and Benchmark
- Part 2 — Backtesting and Hidden Costs
- Part 3 — Key Portfolio Metrics
- Part 4 — Parametric Analysis and Overfitting
- Part 5 — Forward Testing and Statistical Significance
Disclaimer: This article was developed for informational purposes only, it is not a trading recommendation.
Disclaimer 2: All charts and tables presented in this article are developed by myself.
We finished part 4 showing an out-of-sample testing of our strategy and discussing its results. The data we used to develop our model was from 2018 to 2021 and our testing data was from 2021 to 2022. But there is a way to make better use of our data instead of this.
Forward Test
A common technique used when developing trading models is the walk-forward testing, or just Forward Test, which consists of training the model with part of the available data and testing it with a small sample at the end, repeating the procedure to simulate a real live trading situation. Results obtained with such test are much more reliable and less prone to look-ahead bias. Also they make much easier for us to identify overfitting in our strategy.
I performed a forward testing procedure in our study and obtained a log similar to this one. As you can see the iterative training and testing steps walk forward until all data is tested.
The overall results of the forward testing are good in terms of outperformance of the benchmark, but as we can see there is room for improvement of the other metrics. As we can see the strategy had a 54% Drawdown during the study, which is tough for any trader to support.
Ok, our testing framework is definitely improving. We have developed a tool that backtests our hypothesis, compares it with a benchmark, takes into account hidden costs, calculates relevant performance metrics, analyzes the sensitivity to our model parameters, and performs forward tests. There is one additional feature that I want our framework to have, which is to test the confidence in the alpha that our strategy is generating.
Statistical Significance
As you must have realized until now, quantitative trading relies a lot on scientific methods to make profits. Rule number 1 of scientific methods is testing hypothesis to check for strong evidences, which in most cases are supported by statistics. Statistical significance is a strong evidence that the relationship of two or more variables is not explained by chance alone. In practical terms, we test what is the probability that the alpha that we obtained with our strategy was a pure coincidence, and if that probability is sufficiently low, then we have strong indications of a cause and effect situation. This is very useful in our quantitative trading field, as we want to find trading strategies that capture actual market behaviors and not random noise. So how we test for statistical significance when backtesting a strategy?
Drunk Trader
To explain this analysis I personally like to call it the drunk trader. Imagine that a drunk trader took over the desk and operated random trades for the entire backtest period, instead of following the algorithm recommendation. In a more scientific term, we randomized trades through the backtest period. We repeat this procedure a few million times to check for all possible drunk trader situations and plot their metrics in a distribution to compare with the metrics of our actual strategy. To make sure that the trading costs don’t impact considerably on the results, let’s fix the number of random trades to the same number of our strategy’s trades.
The image below shows the normalized returns for 100 iterations during the testing period:
As we can see, different drunk traders obtained very different returns in the study. It is an interesting image, but does not give us much information. Let’s run 100,000 iterations and analyze those returns in a Histogram. As we could expect, the data distribution is skewed, and most of the drunk trader results are near 0. The red line is the 95th percentile of the distribution and the green line is the result we obtained with our strategy:
Our strategy performed better than 95% of the drunk traders, what means that there is less than 5% probability that our strategy obtained its results out of random chance. Do you remember our initial hypothesis from Part 1? The statistical significance that we see from our test is a very good indication that indeed the crypto market is strongly moved by price trends.
Rewriting the past
Another possible way to analyze statistical significance is taking the historical price changes, “scrambling” them and testing our strategy in this new sequence of price movements. This scrambling is actually called bootstrapping in statistics and allow us to generate random scenarios to check how our strategy behaves. We repeat this analysis a few million times and analyze the distribution of the metrics that we are studying. The difference from the previous study to this one is that in the drunk trader analysis we are testing random trades in the actual historical price, and in this one we are testing the actual strategy that we developed in random price movements that never existed. The output of both studies is quite similar, we have scenarios of random performances like the one below.
Just as in the previous analysis, the most valuable information is the histogram of random performances. From that we can take a significance level, such as 95% (red line) and compare it with the performance of our strategy in the actual historical price (green line).
The result shows us that there is less than 5% of probability that we obtained our performance results from random chance. This is a strong indication of an edge.
Conclusion
Thank you very much for reading this series. In these 5 articles I tried to give a brief overview of the development of a quantitative trading backtest framework and its challenges. The truth is that there is an entire universe of concepts, theories and tools in that field, and working with it requires constant learning and testing. There is no magic recipe to make money with trading, even great strategies can fail for many reasons, but I believe that applying rigorous scientific methods and keeping our biases out of decisions is better than trusting our guts. Feel free to contact me on my LinkedIn for any questions or suggestions of new articles.
