# Is Your Hedge Fund Testing Trading Strategies With Your Money At Stake?

Developing trading strategies is a tedious and time consuming task with a low success rate. Significance of results often cannot be inferred from historical tests and validation samples due to data snooping, especially when many trials are involved. As a result, one way of testing strategies is with real money. This is what your hedge fund may be doing.

Let us start with a simple question: How is trading strategy significance determined? One practical answer is via the use of Calmar ratio (annualized return divided by maximum drawdown); if the strategy delivers sufficiently high risk-adjusted returns when compared to the performance of a benchmark, then it is a significant result.

For example, Calmar for SPY since inception based on total return is about 0.16 (annualized return is about 8.9% and maximum drawdown is close to 55.2%.) If a hedge fund delivers a higher Calmar ratio for a period of about 3–5 years, then this is significant performance. Obviously, a Calmar ratio of 0.4, could be achieved by a strategy with lower drawdown and annualized return, for example 6% annualized return and 15% maximum drawdown. But some investors look only at the absolute return and ignore the drawdown. They should answer this question: Would they have stayed invested in the market after a 55% drawdown to realize the longer-term returns or would have sold everything in panic for a large loss? This is where an investor should be honest.

**A time consuming process**

A fundamental problem of trading strategy developers is that it takes time and money to test a strategy live. This is simply not practical and as a result a theoretical measure of significance is required to serve as a filter of candidate strategies.

Consider the theoretical answer to significance: a strategy is usually considered significant when it represents a good forecast of future returns. But this is a very abstract notion of significance and it has to be quantified. One approach to achieve this is via the use statistics. **Statistics allow us to make inferences about population parameters from samples**. This is the main goal of this art and science. Actually it is more of an art than a science. How does this apply to testing the significance of trading strategies?

A simple test is to try to find evidence against the null hypothesis that the returns of the strategy come from a population with zero mean. But all we have is a sample of historical data. What complicates this task is the possibility that the sample may not representative of the population because the test did not include a wide range of market conditions. Then, at another complication level: the sample of returns may be the result of data snooping bias. This bias arises from the dangerous practice of reusing historical data to test alternative hypotheses with a large number of predictors, also known as features or attributes in machine learning.

Machine learning has gained popularity but its effectiveness depends on the quality of features used and its results may be plagued by data-snooping bias because it involves testing many alternative hypotheses with different combinations of features and different classifiers. It is exceptionally hard to test the null hypothesis with results derived from machine learning. Several methods have been proposed for adjusting statistics for multiple trials but risk of false rejections and false positives is still too high.

Machine learning can be useful and successful in identifying the best alternatives among a group of good options but if the options are not good (economically significant) the results are usually random and over-fitted to noise. A degree in Computer Science is not sufficient to develop significant trading strategies. In-depth understanding of markets and trading is also required. Fore example, when volatility returns to the markets, some quant funds may face large losses if their strategies fail to adjust to abrupt changes in price action because they were developed with data from a period of low volatility.

The conclusion is that determining theoretical significance is a difficult task and the probability of a false positive (Type-I) error is always high not only due to the fact that the signal to noise ratio in the markets is low but primarily due to multiple trials and data snooping bias.

As a result, it is possible that your hedge fund is testing its strategies live with your money. Here is a list of questions you could ask its management:

- Is the hedge fund using machine learning?
- Is there a claim of a quantitative approach to strategy development?
- Is there any reference to backtesting, historical testing or simulations?
- Has the fund been in operation for more than 5 years?
- Is the fund operated by experienced traders or only by tech enthusiasts?

Note that not all funds that use backtesting and machine learning are in the process of testing random strategies with your money but it is always good to keep that possibility in mind. I hope this brief article was useful but professional help will come from your financial adviser. There are good and well-informed financial advisers that can guide investors in this area and help them understand the risks of investing in quantitative hedge funds.

If you have any questions or comments, happy to connect on Twitter: @mikeharrisNY

**Disclaimer:** No part of this article constitutes a trade recommendation. The past performance of any trading system or methodology is not necessarily indicative of future results. Read the full **disclaimer here****.**

**About the author:** Michael Harris is a trader and best selling author. He is also the developer of the first commercial software for identifying parameter-less patterns in price action 17 years ago. In the last seven years he has worked on the development of DLPAL, a software program that can be used to identify short-term anomalies in market data for use with fixed and machine learning models. Click here for more.