High-frequency trading — algorithms now rule the world

Why the controversy over high-frequency trading ignores the most important issue

7 min readApr 8, 2014

Michael Lewis’s new book Flash Boys has kicked off a firestorm of debate over high-frequency algorithmic trading (HFT). Is it fair that these firms can pay to “co-locate” their computers within the building of an exchange, gaining precious milliseconds advantage over competitors? Are the HFTs simply “front running,” as Lewis claims? That is, making money by learning how other investors intend to trade and then quickly stepping in front to intercept the profits?

My opinion is that this is a lot of strife over very little. I’m sure there are HFT firms with questionable practices, but we’re talking finance here. What part of the market doesn’t have its questionable practices? To be honest, I’m not even sure what “front running” is supposed to mean. If I hear some good news about IBM and buy some stock because I think it’s going up — meaning I think lots of other people will soon be buying it also because of that good news— then I’m trying to gain a profit by jumping in front of those other people. My purchase drives the price of the stock up for them. So, by Lewis’s definition, I’m front running. All of Wall St., indeed all of global finance, is one enormous competition of front running.

People only complain about it when they are the ones being taken advantage of, and this seems to be the case here. It’s the big banks, the big hedge funds, the institutional investors who, through Lewis’s book, complain they are being beaten to profits by the HFTs. (My take would be that they’re just being out-competed.)

Anyway, there’s a more important point — that Lewis’s book totally ignores the most important thing about HFT, which is how it has changed the financial system overall. It’s changed from a system driven by people making decisions to one controlled by algorithms and fast machines, with humans effectively taken out of the loop. This transition, as some important research shows, has in turn made the system much more prone to extreme events — flash crashes, like the famous event of 6 May 2010— happening on very short timescales. More people should know about this work, as it shows that we’ve moved into a machine-dominated phase of finance, with very little understanding of the consequences.

That markets now do some very strange things is apparent in the basic data. Physicist Neil Johnson and colleagues looked at the data on market fluctuations over the period 2006-2011, and found (looking at many stocks on multiple exchanges) that there were about 18,500 specific episodes in which markets, in less than 1.5 seconds, either 1. ticked down at least 10 times in a row, dropping by more than 0.8% or 2. ticked up at least 10 times in a row, rising by more than 0.8%. The figure below shows two typical events, a crash and a spike (upward), both lasting only 25 ms.

These very brief and momentary downward crashes or upward spikes — the authors refer to them as “fractures” or “Black Swan events” — are about equally likely. And they become more likely as one goes to shorter time intervals:

… our data set shows a far greater tendency for these financial fractures to occur, within a given duration time-window, as we move to smaller timescales, e.g. 100-200ms has approximately ten times more than 900-1000ms.

The study also found something much more significant. The authors studied the distribution of these events by size, and considered if this distribution changes when looking at events taking place on different timescales. The data suggests that it does. For times above about 0.8 seconds or so, the distribution closely fits a power law, in agreement with countless other studies of market returns on times of one second or longer. For times shorter than about 0.8 seconds, the distribution begins to depart from the power law form. (It’s NOT that it becomes more Gaussian, but it does become something else that is not a power law.) The conclusion is that something significant happens in the market when we reach times going below 1 second — roughly the timescale of human action.
Now for the punchline —why this transition is happening. What’s so special about one second? Why is this sharp and distinctive boundary located at that period of time, rather than at, say, one minute or a 10th of a second?

Well, it is more than a little suspicious, the researchers point out, that one second happens to be right around the speed limit for fast human decision making. Experiments with chess grandmasters, for example, show they can assess a complex chess situation and identify a threat of checkmate in about two thirds of a second. Other people operate at comparable speeds in their own areas of expertise. When it comes to making conscious decisions, one second is about the limit. Coincidence? Or are the markets at this timescale showing the signs of an emerging all-machine phase of trading over which human decisions have little influence or control?

Further evidence for the latter interpretation comes from simple models of markets as ecologies of interacting strategies. These models reproduce many of the realistic qualities — or “stylized facts” — of real markets, and can help us anticipate how markets might do surprising things. In particular, they can give hints about how seemingly innocuous, gradual changes might push markets across a threshold and into a regime of dramatically different behavior. Mathematical studies of these models show that one of the most fundamental factors influencing their basic dynamics is how “crowded” the market is — crowded in an intellectual and strategic, rather than physical sense. If the participants in a market use a wide and diverse range of trading strategies, then the market is uncrowded. In this case, the typical behavior is akin to that in a world with few predators and relatively plentiful prey. A healthy diversity of participants earns profits in different ways — thinking and acting on different timescales, taking different views on the future and so on.

Real markets, the models suggest, look a lot like this uncrowded phase, with highly irregular market fluctuations and fat tails. In contrast, if a market becomes overcrowded — that is, if many traders chase few opportunities and use very similar strategies to do so — then the continuity of the market tends to break down. In this regime, the market becomes prone to what might be called “glitches” or “fractures,” sudden moves up or down much like those now observed in the sub-one-second trading regime.

There are good reasons, Johnson and colleagues argue, to think that high frequency markets have indeed entered such a crowded phase. After all, high frequency algorithms by their nature compete on speed and have to act extremely quickly. They have to be relatively simple and can’t waste time analyzing too much information about the past. Given these constraints on the range of possible strategies, and given the number of traders operating within them, overcrowding is quite likely — as are the fractured, troubled market dynamics arising from it.

So, as trading moves to inhumanely short timescales, we shouldn’t be surprised, but should actually expect to see increasingly frequent Black Swan events in microscopic timescales. They may well be the natural consequence of machine trading that is becoming uncoupled from the strong influence of conscious human decision making.

We’re moving, as Johnson and colleagues put it, “from a mixed phase of humans and machines, in which humans have time to assess information and act, to an ultrafast all-machine phase in which machines dictate price changes.” We’re crossing a boundary into a trading twilight zone, and doing so without much thought or awareness of the potential dangers.

So, I find myself agreeing with Felix Salmon that Michael Lewis has in effect ignored the most important issue concerning high-frequency trading — its role in making the entire global financial system highly unstable. As Salmon puts it:

…By far the biggest risk posed by the HFT industry, for instance, is the risk of the kind of event we saw during the flash crash, only much, much worse. The stock market is an insanely complex system, which can fail in unpredictable and catastrophic ways; the HFT industry only serves to make it much more brittle and perilous than it already was. But in Lewis’ book-length treatment of HFT, he barely mentions this risk: I found just one en passant mention of “the instability introduced into the system when its primary goal is no longer stability but speed,” on Page 265, but no elaboration of that idea.

Of course, this IS a good way to sell lots of books. The human mind craves narrative stories of scandal and cheating and nefarious subterfuge. “You’re being ripped off, and they’re doing it!” will always sell more than “the system itself has become unstable, in a way that no one really intended.” But that instability is the bigger problem.

Follow The Physics of Finance on Twitter: @Mark_Buchanan

Other recent articles:

Bio-finance? Why high-frequency trading makes stocks flock like birds

Plus a bonus: enjoy an “awesome” fish video

Taxis 2.0: Streamlining city transport with graph theory

Big city taxi systems could be 40% more efficient with device-enabled taxi sharing

Arrogant physicists — do they think economics is easy?

No. But their ideas can help improve economics, and here’s why

Random oblivious transfer: is quantum secured finance almost a reality?

Why quantum cryptography could soon play a role in our financial lives

The fragile “wisdom” of crowds

Why prediction markets aren’t quite as amazing as economists once thought