The grim reality of retail day traders. Part I: Equity.
1. Introduction. Becoming a retail trader has never been easier. There is a plethora of brokers and platforms, opening an account takes between five and thirty minutes and could be done from your couch. Many people aspire to be traders in the search for financial independence, an increased wage, or simply an appeal for the attractive lifestyle. However, in reality between 62–89% of all retail trading accounts are losing money, varying from platform to platform. Every broker, operating within the EU, is obliged by law to disclose this information, so you could just browse the ones you are interested in and read the corresponding percentages. Moreover, if you average over all brokers, the base rate of failure is about 75%. We illustrate the potential profitability of the remaining 11–38% of trading accounts with the following example. Imagine you open a trading account with 10000$ in your favorite platform on 1 January 2019. If you close the year with 10001$ in your account, you are considered profitable. On the other hand, the S&P500 index alone has risen more than 27% in 2019 and more than 57% since the beginning of 2016. We leave to the reader the estimation of what proportion of the profitable retail trading accounts have made more than 25% in 2019, because the corresponding traders/investors could have simply bought an instrument that tracks the S&P500 index and spend their time doing something else.
The question that interests everybody is: why is that? Why are so few accounts actually profitable and why is almost no retail trader consistently profitable. The standard explanation is that most people are just not committed enough, they are not running the extra mile and doing their homework. In other words, the common explanation of this staggering statistics is that people are not skilled enough in the holy grail of retail trading — technical analysis (TA). A theory that requires no prior education, takes months to learn, years to master and explains it all. Inspired by the terrible numbers above and by the work in [1], we performed a series of experiments in an attempt to answer the opposite question: how profitable can you be if you do master TA and trade solely based on that knowledge? In the sections to follow, we explain the model we used and provide a detailed summary of the results. Brace yourselves!!!
2. Smoothing: the heart of TA. If you ask a financial professional to describe in one word what TA is, you would probably hear “magic” or “alchemy”. If one looks closer and more seriously at the tools of TA, she or he will find out that, in essence, it consist of smoothing and analyzing local extrema. In this section, we explain what these two things are, how are they the backbone of TA, and construct our model, which is based on [1].
Let XYZ be your favorite stock. Imagine the daily price chart of XYZ (here we focus on daily prices, but the intervals are not essential to our model), which consists of equidistant discrete time points and a stock price for each of them. A trader would often look at the candlestick chart, which contains a little bit more information but for our explanations, the series of opening/closing prices would suffice. Inspecting closer this chart, we see that it consists of a general direction, in which it moves, plus sporadic jumps up and down from it, which are usually called noise and considered as randomness added to the trend. The first step of TA is to learn how to filter out the noise in order to “purify” the trend curve. It is important that we smooth out the right amount of noise, which means that if we leave a lot of noise, our patterns would be affected by randomness, and if we smooth out too much, we will lose valuable information. This process of extracting the trend of a time series by removing the noise is called smoothing. It is very well studied in statistics and there is a myriad of smoothing techniques that work well in different situations. In fact, the different types of moving averages, that are widely used in TA, are probably the simplest examples of smoothing. They vary from simple averaging to an average of the values withing a chosen window, where we also assign weights to these values, i.e. we consider some of them as more important than others. Typically, we would weigh the values that are close to the point, where we want to estimate the trend curve, as being more important than the values that are far. The smoothing method that we use is called non-parametric kernel regression and is a far-reaching generalization of the moving average. To illustrate our model, we pick an actual stock, that we will still call XYZ, and fit a kernel regression to smooth out the noise. How smooth the fitted trend curve is, or equivalently, how much of the fluctuations of the price we consider as noise to be filtered, is controlled by a parameter called the bandwidth. Larger bandwidth indicates a smoother curve. Since in TA there is no agreed upon rule for what fluctuations are considered noise and the general approach is rather…subjective, we perform all our simulations for seven different models, starting from a bandwidth of 0.6, which overfits the price data quite a bit, all the way to a bandwidth of 1.8, which is fairly smooth and removes a large portion of the fluctuations. In Figure 1 below, for a short time window, we illustrate how we fit our models and also display the daily trading volumes.
You can see a clear HS pattern in the original price data, in blue, which is only reflected in some of the fitted smoothing curves. For instance, when the bandwidth is 1.8, the trend curve smooths out most of the local movement and does not consider them as forming an actual pattern. On the other side of the spectrum, the model with bandwidth of 0.6 also does not consider this pattern, because it overfits the noise of the movement and finds several more locally emerging patterns that overshadow this one. This is a good illustration of how models with different bandwidths capture patterns, emerging at different scales. In the rest of the article we generally consider each model separately and analyse the results on every scale.
Every technical analysis of a stock starts with smoothing out the noise by fitting a trend curve to the price series or candlestick chart. Usually, a very simple model is used, such as a simple or exponential moving average, or determining the trend is left to the intuition of the analyst. Moreover, “the right amount” of noise is treated very subjectively with the belief that it can be learned through experience. In any case, this is nothing but a simple smoothing in the sense that we introduced above. The next step of the analysis, whether it would be detecting emerging technical patterns, support and resistance lines or anything else, boils down to a simple analysis of the local minima and local maxima of the trend curve. On Figure 2 we show how, by analyzing the local extrema of the fitted curve for a bandwidth of 1, detects the emerging HS pattern.
In fact, a HS pattern is simply a sequence of five consecutive local extrema, which starts with a local maximum, where the “bullish trend is starting to reverse”, such that the difference between the first and the third local maxima (the shoulders) is not too large. For the formal details, as well as the definition of several other technical patterns, including broadening tops, triangle tops, rectangle tops, and double tops, we refer the reader to [1]. The authors of this paper provide exact definitions, that are in accordance with the theory of TA and also approved by professional traders. Following these formal definitions and using snippets of code from [2], we implemented algorithms for automatic detection of the HS and IHS patterns. It is important to mention that, in accordance with [1], we only count patterns that emerge in the course of a window of at most thirty six time points (days), which rules out possible patterns due to trends over very long periods of time. It is also important to notice, that people often require an increase in the daily volume of the asset at the closing of the pattern, just as it happens on the chart above. For flexibility, we consider both cases in our analysis, i.e. without taking the daily volume into consideration and conditioning on an increase in daily volume in the second half of the pattern.
Before we continue further, we would like to mention two additional things. First, finding support and resistance lines is no different than finding emerging TA patterns. It is simply a procedure of locating “the right” sequences of local maxima/minima of the smoothed trend curve, so that there is a line, tangent to all of them. Second, we implemented a slightly simplified version of the algorithm in [1]. Namely, in the article, instead of fitting one trend curve to the whole price series, the authors use a small sliding window and fit a separate kernel regression for each iteration. What they achieve with this complication is the elimination of a look-ahead bias. More precisely, they allow only for patterns that emerge within the sliding window, which implies that in fitting the smoothed trend curve that detected the pattern, no information was used about prices that occur after the pattern was completed. In our model, we always fit a single smoothed curve to the whole price series, which means that when detecting and trading the pattern, our algorithm implicitly uses information about the prices that occurred after the completion of the pattern. This means that our algorithm should have a slight advantage to an actual trader that trades on HS and IHS patterns, and that it is expected to perform slightly better on average.
3. Trading strategy. In our simulations, we use the following classical conservative strategy to trade HS and IHS. We illustrate it for the HS pattern we discussed above. Once the pattern completes, we open our position with a short sale of one share of the stock and set the stop-loss line at the level of the head, i.e. the highest peak. We set our buy line by reflecting the stop-loss line along the entry level. In other words, we measure the distance from the entry level to the stop-loss and we draw the sell line at the same distance below the entry level. Then, we monitor the closing price of XYZ and once it crosses one of the two lines, we buy one share of stock at the opening price on the next trading day and close our position. Our profit is the difference in the prices. We illustrate the strategy on Figure 3. Dots with the same color represent the opening and closing of the position.
In order to ensure more flexibility, we consider several versions of this strategy, depending on when does the trader become certain that the pattern is complete. We consider five different cases, in which the trader opens her/his position from one day before to three days after the pattern completes. We illustrate the trading strategy and the different entrance days on the figure below. In fact, in this particular case, things worked out very well in the favor of the trader.
The first row represents the pure difference between the prices, while the second is adjusted for
transaction costs. While many brokers do not charge a commission on trades, the trader has to always pay a bid-ask spread. We were not able to find reliable statistics on the average bid-ask spread on equity along different popular brokerage platforms. In our model we chose a universal bid-ask spread of 0.5%, which is rather on the lower side of the spectrum, to put it mildly. In fact, it is quite unrealistically low, but it illustrates well the difference between the adjusted and non-adjusted profits.
4. Simulation results and statistics. Now that we have laid out the details of our model, we present the results of the different simulations we performed. We focused on the stocks that are currently listed on the S&P500 index. Due to missing data, we had to remove some of the symbols, thus ending up with a pool of 460 stocks. We divided this pool into five bins, according to their average liquidity in the period 1 January 2016–31 December 2019. For each of the groups, we picked at random a portfolio of ten stocks, and added one more portfolio, consisting of ten popular companies. We also simulated a random trading strategy, which reflects the average number of trades and days between opening and closing the position, and chooses the trades uniformly at random.
Popular stocks. We start by analyzing the portfolio of popular stocks. A summary of the results is illustrated on Figure 4.
What you are seeing is four situations, depending on whether we adjust for the bid-ask spread and condition on an increase in volume. Following our model, for each of these situations, we have computed thirty five different series of capital changes, all starting with 10000$. For each of the four situations, we have picked the best and worst performing capital series and plotted them, along with the S&P500 and an average of one hundred random portfolios. On the right hand side, the S&P500 is adjusted by 1% per year, and the random portfolio is adjusted by the same 0.5% bid-ask spread for every random trade. The first thing to observe is that the performance is terrible. Not only the returns are mediocre at best, but also the volatility is fairly large. Our conclusion is that picking up a biased set of popular stocks, that we read about in the newspaper, is not the way to go.
All of S&P 500. As a second step, for each of the five average liquidity buckets, we chose ten stocks at random and computed the corresponding capital series, always starting with 10000$. The results were not any better than for the popular stocks, so we decided that we probably got unlucky with the picks. In an attempt to rectify our bad luck, we fitted our model to the whole buckets and computed a capital series for all four situations. On Figure 5 we plot histograms of the total returns of all the capital series. On the x coordinate is the total return, and on the y coordinate, the count.
It is important to notice that there are some strong correlations between some of the capital series, so that these histograms should not be considered as empirical probability distributions of total returns, but rather as an illustration of the total performance.
Next, we examined the average monthly returns of all the capital series and plotted them against their volatility, measured as the standard deviation of the monthly returns. The results are presented in Figure 6.
We found these results uninspiring, so we decided to look into the distribution of percentage returns for all the trades. We computed the adjusted trades for each of the 460 stocks. The results are displayed in Figure 7 and present, for every model, a histogram representing the empirical distribution of percentage returns for single trades. The model is denoted by a pair (bandwidth, days waited).
It is interesting to notice the two bumps in every distribution. This separation is caused by our trading strategy, since we always wait for the price to cross either our buy/sell or our stop-loss lines, which rules out the possibility of no return.
Difference between buckets, HS and IHS. We expected a difference between trading stocks that have very different average liquidity, but we were wrong. For each bucket, we computed an average of the adjusted capital series and observed that they are very much alike, which is illustrated on Figure 8.
There is an emerging pattern of capital series behaving the same way no matter what the underlying stocks are. In particular, there does not seem to be much of a difference when we condition on an increase in volumes in the second half of the technical pattern. In a final attempt to find some variation, we computed HS and IHS separately and compared the results. From each of the five buckets we picked ten stocks at random and computed the capital series resulting from trading only one of the patterns, both non-adjusted and adjusted. We then averaged over all the different models for every representative portfolio. The minuscule difference between HS and IHS is summarized in the table at the end of the section, where we present, for each portfolio, the final sum of the corresponding average of capital series.
4. Conclusion. It should not come as a surprise that technical patterns have little, if any, predictive power. The same is true for TA as a whole and while some of the techniques might be useful in understanding how markets reacted to events in the past, they do not provide any useful information about the future, on any scale. We will briefly name a few of the more obvious reasons for that.
- TA is outdated: The theory was formulated in the 1950s, when computers were not yet commercially available. At that time, trades were made by people and we know that even the more trained of us, often suffer from moments of irrational behavior, especially when the stakes are big. The bedrock of TA is that trading should be based on the “psychology of the market” and, allegedly it offers a toolbox for identifying predictable irrational patterns in other traders’ behavior. The potential usefulness of this is questionable, because trading the stock of a company, while completely disregarding how the company is doing, has never been even close to a good idea, but, on the other hand the idea has some appeal to it. However, that was seventy years ago and financial markets, along with the whole world, have changed tremendously. According to [4], more that 80% of all trades on the forex markets are performed by algorithms. The figures are not lower in the equity markets. If you deduct from the remaining less than 20%, all the trades, made by institutional human traders, you will be left with probably less then 5%, if not less than 1% of trades, made by retail traders. To summarize, more than 80% of all trades are executed by heartless financial killing machines, that do not suffer from a broken heart or from sugar lows, and almost all the rest is executed by trained professionals that also have very sophisticated statistical models at their disposal. From that perspective, this whole analysis of “the psychology of the market” is completely obsolete and as good as trading at random. You cannot use arguments, based on “crowd behavior”, if there is no crowd.
- TA is too simplistic: Price series of securities fall into a broader class of objects called time series. These are of interest both to mathematicians and statisticians and a great deal of efforts has been made in understanding and modeling them. Financial time series are particularly nasty to study. Without going into any details, we can say that, essentially, the reason is that financial markets are very complicated objects. In that regard, despite tremendous efforts, they currently seem to be as unreachable to understand as the climate of the Earth or the human brain. On the other hand, we have the tools of TA, which include fitting moving averages, counting local extrema and drawing straight lines to predict the movement of the price of a security, while intentionally ignoring the myriad of other parameters that affect it.
- While being a peculiar idea to use TA in an era when an intuitive definition of a deep neural network is “five lines of code in PyTorch”, it does appeal to a large group of aspiring retail traders. The reason for that is the way TA is presented: it requires learning but no prior knowledge in mathematics, statistics or finance; it is full of jargon and it gives the impression that, because analysts speak an arcane language, they seem competent and knowledgeable; it is implicitly very subjective, so every losing trade is due to a mistake in the analysis. To understand more about the psychology of this appeal, we recommend the outstanding book “Thinking, fast and slow” by Daniel Kahneman, a Nobel prize winner in economics.
- Markets are not completely inefficient: Even if the above arguments were completely wrong, there is no strategy that could still be profitable after seventy years of use. We do not want to discuss whether markets are efficient or not, but it is certainly true that they do incorporate information over time. From that perspective, whatever information technical patterns could bring to a trader in the 1950s, has been implemented into the market and does not bring any advantage any longer.
If you look closely at some of the figures we presented, you will see that there are a few capital series that do perform fairly well, but we intentionally omitted saying which ones they are. The reason is that we have tested more than 1200 capital series and it is expected that, by chance, some of them will perform well. In fact, in hindsight you can always identify very profitable moves and strategies, especially if you look at anecdotes. For example, in [3] the authors discovered that a portfolio that goes long on stocks that have “S” as the third letter in their symbol and short on stocks that have “U” as the third letter in their symbols, has been insanely profitable in the period 1962–2014. Does that mean people should run to the stock store and buy that portfolio, probably not a genius idea. The point is that in a set of thousands of portfolios/capital series, be sheer chance, some will do really well, some will do really poorly.
You are probably thinking that these results do not concern you, because you are a forex trader and not particularly interested in stocks. Well, there are objective reasons why being a retail forex trader is even more disadvantageous that being a retail equity trader and in a follow-up article we will analyze the performance of TA there.
References:
[1] A. W. Lo, H. Mamaysky, and J. Wang, “Foundations of Technical Analysis: Computational Algorithms, Statistical Inference, and Empirical Implementation”, The Journal of Finance, Vol. lv. №4, August 2000.
[2] An empirical evaluation of technical analysis, quantopian post.
[3] R. D. Arnott, C. R. Harvey, and H. Markowitz, “A Backtesting Protocol in the Era of Machine Learning”, 2018, SSRN.
[4] A. Bigiotti, A. Navarra, “Optimizing Automated Trading Systems”, Advances in Intelligent Systems and Computing, Springer International Publishing, 2018,
