Why you should not try to predict stock markets

I have met a surprising number of people who have tried to build a trading algorithm at some point. Here is why you should not waste your time with that.

First of all, I have to admit: I do it, too. For a long time stock markets have been endlessly fascinating to me, and I have build models to try to predict stock returns. There is a whole industry based on investing, there are quantitative hedge funds and they all seem to make giant amounts of money. So what could be more fun than playing with machine learning models and fantasizing about becoming rich?

The problem is, if you are actually expecting to make money then you are wasting your time.

The efficient market hypothesis

Economists rarely agree on anything, but they do agree that short term movements in stock prices are very close to being unpredictable. The reason for that is not only that they have tried and failed at predicting it. There is also a theoretical argument called the efficient market hypothesis (EMH).

Imagine there is a stock that is worth 100 Dollars today and you know that it will rise to 110 tomorrow. You are 100 percent sure. There is no risk. What would you do? You buy the stock, as long as it is still cheaper than 110. By doing so, you increase the demand for that stock today. There are only a few people in the market who are willing to sell for 100 and you want to buy as much as possible, so you offer a bit more. That means your demand increases the price of the stock TODAY, because you know that the price will rise TOMORROW. How much would the stock price rise? Well that depends how deep your pockets are. But if you have an unlimited amount of money to invest, you would keep buying even if the price ist at 109 already, because you can still make a profit. That goes on until the price reaches 110. In the end the price movement that you knew would happen tomorrow already happened today because of you own doing.

You might be thinking to yourself now that this is an interesting logic — but not realistic at all. You do not have enough money to influence the market like that and there is always a risk that your prediction is not correct. So let’s address that. Economists like these simple stories, but only if they generalize well. Of course, you are probably not rich enough to influence the price of Apple. But institutional investors are. If they have hundreds of millions to invest, they do move the price with their decisions. So the argument still works, if it is not about you but about professional investors. And what if there are many of your kind? 10,000 small investors with the same information together can have the same impact on the market as a single big player. So the assumption that you are big enough to move the market is actually not that restrictive.

What about risk? In reality you will never be sure about the exact price tomorrow. So let’s explore what happens if the price does not move from 100 to 110, but with equal probability to 108 or 112? Obviously, you would buy at least until the price is at 108. What happens after that depends on your attitude towards risk. For simplicity let’s assume you do not care about risk. You only care about the money you are expecting to make. In that case you would buy the stock until the price is equal to the expected value of the stock tomorrow. That is E[X] = 0.5 * 108 + 0.5 * 110 = 109. If the price is already at 108.5, you still have a 50 percent chance to make 1.5 versus a 50 percent chance to lose 0.5. Still sounds like a good deal, right? So you keep buying until the price is at 109. This is the best estimate of the price tomorrow that we can make today.

So you can see that not only the direction of potential price movements in the future is integrated into the stock price, but also the uncertainty about this price movement. This is how financial markets incorporate information. As a result of this, the stock price always reflects the available information in real time. This is a remarkable process. If all actors follow their own interests and try to make money in the stock market, they make sure that the price reflects all information about the value of the company as accurately as possible.

Of course this is theory. The efficient market hypothesis states that a stock price always reflects all available information about the value of the company. The term efficiency is used because this would mean that the market integrates all information as well as possible. This is the definition of Fama (1970). Another definition that lends itself better to test it empirically was proposed by Malkiel (1992). It says when a market is efficient with respect to an information set, then the price will not change if this information set is revealed to all market participants. The term information set in this definition leaves room for strict and less strict forms of efficiency. Remember that the most crucial assumption of the theory is that the people who have the information have enough money to move the price with their demand (or supply if they are short selling).

The weakest form of the EMH is when the price includes all information that can be derived from past values of the price itself. This would mean that technical analysis can not work. Everyone has access to historical price data freely on the internet and everyone can draw trends and resistances, calculate moving averages, or fit simple statistical models such as autoregressions and so on. Therefore it is easy for the market to be efficient with regard to past price data, because all these people together can move the price.

The intermediate form of the EMH refers to all publicly available information. This of course includes past price data, but also weather data, other stocks, exchange rates and interest rates, the latest tweets of Donald Trump, and so on. As far as I am aware researchers generally agree that intermediate market efficiency holds — at least approximately (there are some deviations that are explained by behavioral biases). So you can also not expect to make profits by building a model that utilizes this other publicly available information.

What about insider information? This is the third and final form of market efficiency. If markets were fully efficient with regard to insider information, that would mean that prices would not move if the information was revealed to the public. I think we all agree that this is not the case. For this to happen there would have to be a lot of insider trading going on — and I mean really a lot. So insider information is your chance. Unfortunately it is not legal.

But how do professional investors make money?

So how do these professional investors make money? Well, most of them do not. Do not get me wrong. Most of them return profits, but anyone can make profits by just investing in the market. If investors or fund managers add value, then they should outperform the market, by either having a higher return with the same risk, or the same return with lower risk. If you wanna learn more about this, here is a link to an episode of the Freakonomics podcast that is dedicated to this topic.

But of course the are some investors who make money. So how do they do it? Well, there is a lot of information that is hard and costly to acquire. These investors make money by incorporating information that is harder to access. They may, for example, buy satellite images from supermarket parking lots to try to judge how sales are going. They may also be the first ones to use text based information to predict stock movements based on social media, etc. But the nature of these things is hard and costly, so that you are unlikely to succeed in it alone out of your bedroom.

Transaction costs play against you.

Let’s assume you do actually find a statistical signal that can be used to predict the stock market. For all the reasons discussed above, you will never exactly know how a stock is going to move tomorrow. You will find a slight tendency in stock returns. To exploit such a slight tendency you need to do a lot of trades. These will sometimes win money and sometimes lose money. If your slight edge is true, the average will be slightly positive.

This is where transaction costs come in. If you find an actual way to predict something, you will not be able to trade on it successfully. Your transaction costs as a private investor are too high. If you are trading with retail brokers and your private money, you will be paying flat fees or spreads that eat up you small profit. You are not a hedge fund. Institutional investors who trade large volumes have other access to the market and much lower transaction costs. That means they do not only have a tower full of math- and finance nerds, they also have an edge when it comes to exploiting these little statistical patterns that you might find.

If you think you are successful, you are probably just being rewarded for taking on market risk.

Now, maybe you have build a trading algorithm and it is generating positive returns. This still does not mean that you are doing better economically than you would have if you just invested you money in the stock market and waited. If you are investing, then you get higher returns if you are willing to carry more undiversifiable risk. That is why long term stock market returns tend to be higher than interest rates on government bonds. So to judge if your algorithm really performs better than a buy-and-hold strategy in the market portfolio you have to analyze your risk adjusted returns.

Describing how to do that is another post on its own, but I promise you, if you made it through all the hoops before, this last one is probably doing you in. If you are making positive returns you are most likely carrying some form of market risk for which you are being compensated. You are not outperforming the market.

So trying to predict stock returns to make money may not be a good investment of your time. But it is a lot of fun and stock market data is a great place to hone and develop you skills. Dreaming of becoming rich may keep you engaged for a littler longer and carry you over some hurdles. So have at it. Enjoy yourself.

Originally published at http://firstdifferences.com on April 10, 2020.