“The Man Who Solved the Market”: And the solution was… HMMs and regression
I just finished reading Gregory Zuckerman’s fantastic biography of Jim Simons: “The man who solved the market.” It covers some of the many unique characters at RenTech up to the present, and has the most to say about their 1980s–2000s trading strategies. The best answers I was able to get from the book on what their solution to the market was:
- Markov chains, HMM models
- Linear factor models, followed by high dimensional kernel regression methods
- Big monolithic models and lots of data
- A well orchestrated engineering effort
The history in the book can be summed up in the following chart, which I made by grep-ing the ebook for years:
The steeper the line the more the author mentioned that time period. The majority of the history is about the bond, commodity, currency team (started as Axcom with Ax’s Markov models, taken over by Berlekamp, improved by Laufer, and eventually taken over by Brown & Mercer). Within that steep part is the development of the stock team (which uses Frey’s factor models). The flatter bits like after 2010: not much is written about the fund. And the earlier years with Baum before Ax are dominated by traditional trading.
The book is highly engaging, and contains a lot more than what I highlight here. A lot of the strategies mentioned in the book can be divided into trending and mean reverting strategies: when is the price rise/dip just the beginning, and when is the price about to bounce back to what it should be according to historical factors. But I wouldn’t say that it seems like there is one big idea behind the fund. Its success appears to be a result of a huge software engineering endeavor that processes a lot of data, looks at a lot of correlations, and makes a lot of trades. In this regime, you only need to be right slightly more than half the time to make money, as the book’s characters say.
Early traditional days
One of the first employees of Jim Simons (JS) was Leonard Baum, co-inventor of the well known Baum-Welch algorithm for estimating the transition matrix in HMMs with EM. In an early paper “A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains” that introduces this idea you can find JS as a reference for the never published “Probabilistic models for stock market behavior” which is cited as an application. Interestingly, Baum preferred trading traditionally with JS and not much was done quantitatively.
Quantitative Bonds, commodities, and currencies team
James Ax, a number theorist and strong believer in Markov Chains for financial markets was JS’s first quant, and headed the effort under the name Axcom in CA. By ’86 Axcom traded 21 different futures contracts. René Carmona then joined them to try to incorporate SDEs, his field of expertise. When that failed, Carmona suggested changing their existing linear regression approaches to be nonlinear, high dimensional kernel regression methods, and having the model directly suggest buy/sell orders. These improved results on trending models.
Elwyn Berlekamp (who had worked with Kelly) took over Axcom in ’89 and brings in Henry Laufer in ‘92, they worked on mean reversion strategies and looking at correlations between time periods. They start using data in 5 minute bars, used pairs trading, and had an online learner, constantly looking for trading signals (one auspicious one they called “Henry’s signal”). By ’97 “more than half of the trading signals Simons’s team was discovering were nonintuitive” and they ignored them. But they figured out the right day and time to make their trades. Later Simons says “we’re the best at estimating the cost of a trade.” In ’98 they are the majority of the firm, with stock trading only 10% of the profits.
Quantitative Stock team
This team starts to make progress in ’93 when Nick Patterson contacts Brown and Mercer at IBM’s speech group (which uses algorithms like Baum-Welch in their HMMs for speech-to-text). Brown and Mercer take over Robert Frey’s factor stock trading fund named Kepler (later Nova). Frey (previously statistical arbitrage at Morgan Stanley) was identifying various independent variables for factor trading models. Brown & Mercer retain Frey’s model, and elaborate it to cover real-life technicalities he was ignoring. They make an adaptive single trading system for their whole portfolio, and self-correcting for when the trades it suggests are unexecutable. The system repeated on loop several times an hour. It was a well engineered product, and usually bet on mean reversion strategies. By ’03 their profits are 2x Laufer’s team, and they work on a model to replace the futures team’s. Alexey Kononenko rises through the ranks of this team. Two members Belopolsky & Volfbeyn bolt to Israel Englander’s Millennium management, allegedly with (millions of lines of) code and ideas, and have great success there (“some of the most successful traders Englander had encountered”). By 2010 the system is huge, executing thousands of simultaneous trades throughout the day, with lots of factors and interrelations.
The models and the engineering effort
Both HMMs and factor models are famous by now, and the author Gregory Zuckerman has good evidence that the secretive fund used these techniques. RenTech did a lot of hiring from the IBM speech team. Speech recognition relied heavily on HMMs until 2010, when Deep Learning started to overtake the field. Frey, who left to teach at Stony Brook, has a tutorial on HMMs for finance online. Carmona has a book “Statistical Analysis of Financial Data in R” which includes chapters on using kernel regression for financial data.
A big benefit of hiring from IBM’s speech team was getting people experienced not just in HMMs but also in large software engineering projects. Mercer and Brown showed that Frey’s factor model idea was fundamentally good, and once the implementation carefully incorporated all the elaborate characteristics of real markets the strategy went from dud to stud.
HMMs, factor models, non-linear kernel regression, correlations between various time periods, these are all separately well known pieces. But the book says many times that a single model was handing out trading instructions. My first guess for how to put these different elements together would be that everything else was generating features for the HMM.
Conclusion
Gregory Zuckerman’s exciting book contains much more than the trading strategies of RenTech. The stories of each of the characters and the dramas behind the fund are page-turners. As the subtitle of the book says, Jim Simons certainly did launch the quant revolution. And RenTech is having its best years right now! $63B of its $104B trading profits are since 2010, and with just 300 employees. Surely it has updated its techniques, and probably cashed in on the deep learning revolution raging since (at least) 2014!