How predictable is the stock market?

Wall Street, economic theory, and the ongoing quest for finding the signal in the noise

Samuel Flender
Jun 16 · 8 min read
Photo credit: Rowan Heuvel, Unsplash

Our bias is to think that we are better at prediction than we really are. — Nate Silver

There is a legend that German businessman Nathan Mayer Rothschild used his extensive network of carrier pigeons to be the first to learn about the victory of England over France in Waterloo in 1815, and used this information to profit from the British stock market by taking long positions before the news broke to everyone else. If the legend is true, this would arguably be the first insider trade in the history of financial markets.

Today, insider trades are illegal, and news about businesses or countries spreads almost instantly around the globe. What strategies do professional investors follow in order to predict the direction of the market? Is there any predictability at all?

Technical analysis — finding patterns in noise

There are broadly two different schools of thought on how to explain and predict changes in stock prices, called technical and fundamental analysis. Technical analysts, sometimes also called chartists, look for patterns in past stock data that ‘tell’ whether the price is about to go up or down. They invent fancy labels for these patterns, such as pennants, wedges, flags, triangles, and, one of my favorites, the ‘head-and-shoulders pattern’. According to analysts, a ‘head-and-shoulders’ formation is a strong indication that the stock price is about to drop like a stone:

A so-called head-and-shoulders pattern

But does technical analysis really work? As Quora user Ellery Davies explains,

Hell No!

To be clear, it is always possible to find patterns in even completely random noise, and therefore technical analysts will always find interesting patterns in past stock market data. But perhaps the most convincing argument of why technical analysis cannot work is its self-defeating property: if the price of Apple would go up every Friday, for instance, investors would start buying Apple on Thursdays, hoping to time the market. This would increase the price before Friday, and eliminate the very pattern investors were expecting to profit from. To put it more bluntly, if technical analysis works so well, why doesn’t everyone use it and why isn’t everyone rich?

Still, technical analysis is applied to this day, but only by a minority of analysts (around 10%, according to an estimate from economist Burton Malkiel). The majority of security analysts today perform fundamental analysis instead.

Fundamental analysis — finding value

The idea behind fundamental analysis is that the price of a stock is determined by fundamental indicators describing the health of the company. In A Random Walk Down Wall Street, Burton Malkiel reasons that the most important fundamental indicators of a company are its expected growth rate, its expected dividend payout, and the volatility (fluctuation in the price history). Malkiel’s advice in a nutshell: buy stocks with high growth and low stock price per company earnings (P/E) ratio. Try to avoid stocks with high P/E ratios, even if they have a track record of high growth, because high growth is not always sustainable.

Fundamental analysis is no silver bullet though — one of the most difficult challenges in fundamental analysis is estimating future growth. Even for a company with a long track record of high growth, the outlook can change instantly due to random events, or the entry of a new technology into the market (creative destruction). This is part of the reason why there is no consensus on whether fundamental analysis works or not.

That being said, one of the most successful investors of all times made his fortune using careful fundamental analysis: Warren Buffet.

Photo credit: Chris Li, Unsplash

The efficient market hypothesis

Why is it that only so few people can beat the market? Why are there not more investors as successful as Warren Buffet around today?

The reason that the market is so hard to beat is that the market is extremely efficient in absorbing all possible information almost instantly. This principle is also known in the academic world as the efficient market hypothesis (EMH), and it comes in three different flavors (think of these like spiciness levels):

  • The weak form of the EMH states that no one can predict stock prices based on the price history alone.
  • The semi-strong form of the EMH states that no one can predict stock prices even with all the available public information and fundamental data about the company.
  • The strong form of the EMH states that absolutely nothing that is known or knowable about a company, not even insider information, will help you predict stock prices.

To be clear, the EMH is just that, a hypothesis, and academics as well as Wall Street professionals keep debating to this day about to what extend it holds in the real world. But note the link between the EMH and the common investing strategies outlined above:

If the weak form of the EMH is true, that means that technical analysis cannot work. If the semi-strong form of the EMH is true, that means that fundamental analysis cannot work, either.

But what about the strong form of the EMH? Quoting Burton Malkiel, this is “obviously and overstatement”. Even though it is illegal today, people can and did profit from insider information, such as Nathan Rothschild with his carrier pigeons (allegedly).

So what to make of this? Well, if the semi-strong form of the EMH is true, then this means that investors are best off with buying broad index funds that track the performance of the entire market, as opposed to attempting to beat the market with their own stock picks. In fact, this is what investment gurus such as Warren Buffet or Peter Lynch recommend for most people. If it is not true, however, you can possibly gain an edge by picking your own stocks, using methods such as outlined by Burton Malkiel.

Photo credit: Sawyer Bengston, Unsplash

Momentum in the market

In the very long term (on scales longer than decades), there is a consistent upward trend in the stock market. This long-term momentum is ultimately caused by innovation: companies are simply becoming more efficient at producing things, develop new processes and technologies, and find new and better ways to market their products to consumers. It is easy to take advantage of this long-term trend by simply buying and holding a broad selection of stocks or an index fund.

But what about short-term momentum? This does exist, too. As Nate Silver points out in The Signal and the Noise, the Dow Jones industrial average exhibited short-term momentum at least in the decade from 1966–1975: if the market was up on one day, it was more likely to go up the next day, too, a pattern that is very unlikely to appear by chance alone (Silver estimates a 1-in-7 quintillion chance). But can such a signal be used as a trading strategy?

Yes and no. Silver showed that a ‘manic momentum’ strategy, exploiting the short-term momentum, outperforms a simple buy-and-hold strategy if you neglect the cost of commissions and capital gains tax. If you take these additional costs into account however, the simple buy-and-hold strategy outperforms the momentum strategy by a far margin. Any strategy involving frequent trades really has the most benefit for the brokers, happy to charge the extra commission fees.

Photo credit: David Rangel, Unsplash

Can machines learn to trade?

Finally, there is a trend in the industry to apply sophisticated machine learning and time series models, such as neural networks or ARIMA, to stock market data. However, it is important to note that this is essentially technical analysis — the only difference being that the manual part, the detection of patterns, is automated by the algorithm. So, can you expect such models to work? Not if you believe the weak form of the EMH!

That being said, there is no reason to limit machine learning models to time series data alone. Algorithmic traders also use fundamental company data as well as sentiment analysis (for instance from Twitter) as features in their models. However, it is still extremely hard to develop a profitable algorithmic strategy, and the EMH explains why: the market is just too efficient and absorbs all possible news almost instantly — as soon as you detect a sentiment change from Twitter feeds, for instance, it might already be too late to take advantage of it!

Perhaps history can provide a lesson here. Burton Malkiel recalls how so-called beta investing came into fashion in the early 1970s. Beta is a measure of systematic risk of a security, i.e. risk that cannot be ‘diversified away’. Financial theorists predicted that stocks with higher beta would give higher returns in the long run: more risk, more reward. However, in a study published in 1992, economists Eugene Fama and Kenneth French showed that over the period of 1963–1990, there was essentially no relationship whatsoever between beta and stock return.

Photo credit: NASA, Unsplash

Conclusion — beating the market is hard

Whether technical or fundamental analysis, momentum investing, beta investing, machine learning or sentiment analysis, investors are essentially looking for a modern-day equivalent of Rothschild’s pigeons.

The truth is that there is very little signal and a lot of noise in the stock market. This is because the market is so incredibly efficient in absorbing (random) news almost instantly. This is especially true in today’s connected world where information travels with the speed of light.

It is very hard to consistently beat the market, and there are only very few people in the world who have a track record of doing so. For this reason, investment gurus such as Warren Buffet or Peter Lynch recommend broad index funds as the best investment choice for most investors. After all, probably the best prediction about the stock market we can make is that the entire market will probably go up in the very long term.

The Startup

Medium's largest active publication, followed by +481K people. Follow to join our community.

Samuel Flender

Written by

I write mainly about Science, Technology, Economics and Psychology. My background is in Physics. Feel free to reach out!

The Startup

Medium's largest active publication, followed by +481K people. Follow to join our community.