The Journey of a Quantitative Trading Strategy Development (Part 4)
A no-code overview
This article is the part 4 of a series of articles on quant trading. My goal is to present the steps involved in a quant trading project, in a simplified and non-programmatic way. In part 1 we defined our hypothesis, a market benchmark to which we can compare our trading results, and structured our data to use it in the following stages. Part 2 presented the initial backtests and some hidden costs in quant trading. We analyzed a few portfolio metrics to understand more details of our strategy in Part 3. The scripts and frameworks developed to this series are on my Github.
- Part 1 — Hypothesis and Benchmark
- Part 2 — Backtesting and Hidden Costs
- Part 3 — Key Portfolio Metrics
- Part 4 — Parametric Analysis and Overfitting
- Part 5 — Forward Testing and Statistical Significance
Disclaimer: This article was developed for informational purposes only, it is not a trading recommendation.
Disclaimer 2: All charts and tables presented in this article are developed by myself.
Let’s review the results we obtained with our backtest in the end of part 3.
In my point of view, even though there were returns above the benchmark our algorithm lacks consistency, and trades too frequently. I believe that there are ways to improve this, so what can we do?
Parametric Analysis
Parametric Analysis is a process that consists in testing variations of a model’s parameters, to understand their effects on the model’s attributes. In practical terms, we want to analyze the behavior of our portfolio metrics by changing our strategy parameters. In many cases a change in parameters needs to be done to make sure that the strategy is feasible in a live trading situation, considering real life constraints such as liquidity and computational power. Let’s see an example of our backtest performance with three different parameter sets:
- Combination 1: SMA_S: 15, SMA_M: 50, SMA_L: 200
- Combination 2: SMA_S: 30, SMA_M: 100, SMA_L: 200
- Combination 3: SMA_S: 30, SMA_M: 200, SMA_L: 500
As we can see, changing parameters has a considerable effect on the results of a strategy. In order to understand the magnitude of this effect we will define the parameters that we want to study, define a range of values that our moving averages can reach, and then run a brute force algorithm to backtest our strategy using all combinations of parameters, storing the results to compare them later. Let’s analyze combinations of moving averages using the following values:
- SMA_S: 10, 20, 30
- SMA_M: 50, 100, 150
- SMA_L: 200, 400, 600, 800
Those values sum up 36 combinations, and the results table looks like this (truncated to 20 lines):
As we can see the analysis returned a huge variety of performance results. So we have to find the one with the best results and use it for our trading strategy, to maximize our profits, right? Well, not really. Let’s talk about an important topic in model development.
Overfitting
Overfitting is defined by Investopedia as:
“Overfitting is a modeling error in statistics that occurs when a function is too closely aligned to a limited set of data points. As a result, the model is useful in reference only to its initial data set, and not to any other data sets.”
When designing a model, we want it to be useful with new data, so we have to avoid overfitting as much as possible, and the best way to do that is to perform out-of-sample testing. In a nutshell, out-of-sample testing is checking the quality of a model when dealing with data that was not used in its development. Overfitted models will usually show a high in-sample performance and poor out-of-sample performance.
So let’s check for overfitting in our study. We will perform a parametric analysis of our strategy using data from 2018 until 2021, and check the parameters that provided the top 5 results in terms of Multiple.
Now let’s do an out-of-sample test and check how the best parameter combination (20, 150, 600) performed during one year between 2021 and 2022.
During that window our strategy actually lost money. By running the parametric analysis in this window we can see that the parameter combination that we chose was not even between the top 5. Those results are a strong indication of overfitting.
OK, now we know how to detect overfitting, but how to avoid it? Well, there is no magic rule for this, but these good practices will help you:
- Try to develop models with as few parameters as possible. The more parameters, the more overfitting you’ll have.
- Parameter tuning is not about choosing the combination with best performance. It is about understanding areas where your hypothesis seems to hold true. Look for robustness, not for performance.
- Test your strategy using other assets. Generalist models will have consistent performance through different assets.
- Be skeptical of outstanding results. Developing profitable trading strategies takes time and practice.
- Paper trade your strategies before you actually put money on them.
Parameter analysis is an important part of a quantitative trading strategy development, but it’s important to avoid overfitting at all costs. Out-of-sample testing is a very useful procedure to check for overfitting but also to simulate more realistic results, we will discuss more about it in part 5! Feel free to contact me on LinkedIn for any questions or suggetions.
