The Data Science Behind the Man Who Solved the Market
My holiday reading this year was Gregory Zuckerman’s The Man Who Solved the Market, which I finished in one long sitting on Christmas Eve. It tells a fascinating story of the legendary Jim Simons and his secretive hedge-fund firm, Renaissance Technologies.
Without a doubt, Simons has an extremely successful career. An acclaimed mathematician, he co-developed the Chern–Simons theory with Shiing-Shen Chern, the father of modern differential geometry. Simons started a side project on the mathematical analysis of stock trading strategies when he worked at the Institute for Defense Analyses (IDA) as a Cold War codebreaker. After the IDA fired him for publicly speaking out against the Vietnam War, Simons joined the faculty at Stony Brook University, where he recruited top talents from across the country and built a world-class math department. In 1982, he founded Renaissance Technologies LLC. Between 1988 and 2018, Renaissance’s flagship Medallion fund delivered annual returns of 39% (66% before fees), trumping peers including Warren Buffett’s Berkshire Hathaway, whose annual returns were about 16% over the same period of time. Zuckerman calculated Simons’s net worth to be $23 billion, which makes him the 21st-richest man in the United States. Simons poured his wealth into philanthropic projects that ranged from understanding and curing autism to discovering the origins of the universe and life.
One recurring theme in the book is the reflections on the impact of Renaissance other than making Simons and his employees super-rich. When Simons left his flourishing academia career for the hedge fund business, his colleagues thought “he had been corrupted and had sold his soul to the devil.” But Simons understood at an early age that money is power, and he hungered for the independence and influence that wealth can grant.
Ironically, it was Robert Mercer, a former IBM scientist and a co-CEO of Renaissance, who best understood and mastered the power that wealth can bring. In the 2016 presidential election, Mercer was the largest financial backer of Donald Trump and the one who brought in Steve Bannon and Kellyanne Conway to the Trump campaign. It is no exaggeration to say that Mercer, using the wealth Renaissance Technologies created, sent Donald Trump to the White House.
“Mercer was a scientist who demanded robust arguments and definitive proof at the office, but relied on flimsy data when it came to his personal views.”
Zuckerman sounded puzzled in the book. “Mercer was a scientist who demanded robust arguments and definitive proof at the office, but relied on flimsy data when it came to his personal views.” Colleagues were stunned that a scientist like Mercer could be so dismissive of the threat of global warming and so indulging in conspiracy theories such as President Clinton’s involvement in a secret drug-running scheme with the CIA. Thus, it is quite disturbing if the only impact of Renaissance is wealth creation, and then the wealth is used to effectuate and amplify personal biases.
Reflections on morality or social responsibility are not my only interest — When I started reading the book, I was looking forward to a more technical aspect of “solving the market.” Unfortunately, due to the very secretive nature of Renaissance, Zuckerman wasn’t able to shed much light on the details of its investment methods.
Nevertheless, I would like to indulge myself on a technical point: We are in an era where every business wants to, or will be forced to, become a data-driven business. Simons and Renaissance mastered data-driven methods to solve the financial market. What are the practices other businesses can borrow and lessons other businesses can learn in order to disrupt and innovate in their own markets?
The book did unveil some interesting tidbits about Renaissance’s money-making machine. Zooming in on Simons and his top lieutenants, Zuckerman blends stories of their personal struggles with the technical triumphs they had in three areas: system, model, and data. It seems the key components of Renaissance’s trading system are the key components of any data-driven methods for any business.
Systems
The work on computer systems at Renaissance had a humble start. Its early employees being mathematicians, the system-building work fell on a nineteen-year-old Greg Hullender who was on the verge of getting kicked out of the Caltech in 1978 (he eventually went back to school after spending two years working for Simons). In the years that followed, Renaissance’s computer systems were upgraded many times as more computer scientists and engineers came on board. But descriptions of such systems in the book are sketchy to say the best.
It would be fascinating if we were granted an opportunity to look into Renaissance’s trading system because, very likely, it has encountered and overcome many challenges faced by today’s machine learning platforms.
First, at scale is where all the hard problems reside. Renaissance’s trading system discovers patterns each of which involves a group of financial assets. Considering that there are millions of equities, indexes, futures, options, commodities, etc., the system must have a way to manage the combinatorial explosion. In addition, the system must support high-frequency trading, where latencies are often measured in milliseconds or even microseconds.
Second, a machine learning system must be able to manage large scale simulation, backtesting, and parameter tuning over different versions of data. Nothing can be more true, especially for a financial trading system, but unfortunately, the book does not provide any clue on these topics.
Third, Zuckerman wrote about Simons’ struggle to refrain from manually overriding the system when his fund went through rocky times. In fact, it is hard, but ultimately necessary, for an end-to-end system to incorporate real-time human feedback.
Finally, a big system needs to provide a debugging mechanism. Zuckerman told the story that an early version of the trading system scored sizable gains when it managed tiny amounts of money, but when the trades got bigger, profits evaporated. It frustrated the team for a very long time and Simons almost called off the project until a courageous young programmer, living in the office and poring over code day and night, discovered a bug that involves a constant value on a single line of code.
Models
The secret ingredient to Simons’ success must be a bunch of super-smart statistical and machine learning models. Or is it?
Renaissance boasts of having “the best physics and math department in the world.” Simons himself was a world-renowned mathematician in the field of geometry and topology before he founded Renaissance. The book told the story that, during Renaissance’s early days, its mathematicians were modeling the financial market using Stochastic Differential Equations (SDEs). Still, it is not clear if SDEs or other advanced mathematics tricks were indeed the secret ingredients of Renaissance’s investment methods.
In fact, Leonard E. Baum, Simons’ first investing partner and a mathematician known for the Baum–Welch algorithm, pretty much abandoned mathematical models and was in favor of his own intuition and instinct in trading currencies on the market. “Why do I need to develop those models?” Baum asked his daughter Stefi. “It’s so much easier making millions in the market than finding mathematical proof.”
“Why do I need to develop those models?” Baum asked his daughter Stefi. “It’s so much easier making millions in the market than finding mathematical proof.”
Renaissance didn’t take off until after Simons recruited two computer scientists, Robert Mercer and Peter Brown, who had been working on speech recognition and machine translation at IBM Research. A key tool in their computational linguistics arsenal was probably the hidden Markov model, which is quite simple and applicable for modeling time series data such as stocks, bonds, and other securities.
A hidden Markov model, is that all? Zuckerman wasn’t able to reveal anything more concrete. Afterall, even the latest book on the topic of financial machine learning won’t illuminate the reader beyond some discussions of mundane ensemble methods such as bagging and boosting in this secretive trade.
But, unlike the 1970s when Simons started mathematical analysis of trading, today we can easily download volumes of financial data from the web, construct machine learning models using Python, and venture into the business of predicting the market. However, chances are we don’t find any obvious patterns. Everything seems 50/50, and the market trends look exactly like a random walk. What did Simons see that we don’t? This is where Zuckerman becomes poetic: “It’s a bit like how bees see a broad spectrum of colors in flowers, a rainbow that humans are oblivious to when staring at the same flora. Renaissance doesn’t see all the market’s hues, but they see enough of them to make a lot of money, thanks in part to the firm’s reliance on ample amounts of leverage.”
“It’s a bit like how bees see a broad spectrum of colors in flowers, a rainbow that humans are oblivious to when staring at the same flora. Renaissance doesn’t see all the market’s hues, but they see enough of them to make a lot of money, thanks in part to the firm’s reliance on ample amounts of leverage.”
It is hard to decode this poetic riddle, but the common wisdom is to explore and discover relationships among groups of securities, instead of predicting the rise and fall of a particular stock. When anomalies arise in such relationships, the system wagers on the reversions to their historic norms. The challenge, of course, is that the possibilities of such relationships are endless. Thus, instead of “enumerating” all the possibilities, Renaissance’s secret lies in using statistics to reduce the search space of such relationships.
Data
Renaissance’s models are effective. While very little has been disclosed about them, we might be able to understand what they are up to by looking at the data that is fed into the models.
Zuckerman told a great story of Sandor Straus, a data guru who, in my opinion, was the ultimate enabler of Renaissance’s powerful money-making machine. Straus was obsessive about two things. First, he took painstaking efforts in data cleaning. “No one had told Straus to worry so much about the prices, but he had transformed into a data purist, foraging and cleaning data the rest of the world cared little about.” Second, at a time when investors including Renaissance only relied on stock opening and closing data, Straus dived into more granular data: the tick data featuring intraday volume and pricing information for various futures. Later on, Straus engaged in enriching the data. For instance, to deal with gaps in the historical data, he used computer models to make educated guesses as to what was missing.
Straus’s efforts paid off. The early models involved searching for repeating price patterns among securities across a large swath of time. If the data is not clean, algorithms would either miss authentic patterns or pick up spurious ones. Later, when computation power became available, the granular price data would generate thousands of statistically significant observations to help reveal previously undetected pricing patterns.
A fundamental difficulty is that the pricing data alone does not contain sufficient signals to make good predictions. Simons has always been obsessive about leveraging more data in decision making. When he was frustrated with the progress of unearthing profitable patterns, he even considered the possible influence of sunspots and lunar phases on trading. In 1979, his team studied if a severe weather pattern would affect the supply of wheat. Later on, Straus worked with a weather forecasting company to predict coffee prices using Brazilian weather history data.
Every business that cares about machine learning needs its Sandor Straus. Cleaning and enriching data to make it more useful is the secret ingredient to every successful AI strategy.
Every business that cares about machine learning needs its Sandor Straus. Cleaning and enriching data to make it more useful is the secret ingredient to every successful AI strategy. There are many examples, and here is a most recent one: The NFL is going through a data revolution. Previously, a typical game produced about 160 rows of data, each representing a single play. Now, a new player-tracking technology (Next Gen Stats) collects granular data about movements on the field by making 10 observations per player per second. Instead of 160 rows per game, there are around 600,000 rows, an increase of 374,900%. Using the rich sets of data, machine learning models are being developed to identify smarter ways to play and better ways to identify players.
The NFL example reinforces the common wisdom of “the more data, the better the models.” For financial markets, what is the most critical data? As mentioned before, Renaissance’s trading system detects anomalies in relationships among groups of securities. The anomalies arise because humans are prone to fear, greed, and outright panic. Renaissance’s researchers realized what they were really modeling was human behavior. “Humans are most predictable in times of high stress — they act instinctively and panic. Our entire premise was that human actors will react the way humans did in the past . . . we learned to take advantage.”
“Humans are most predictable in times of high stress — they act instinctively and panic. We learned to take advantage.”
Renaissance does not need to directly model human emotions since they are already reflected in the price moves. In this sense, algorithmic trading is not too much different from the online ads business, where a lot of money can be made by getting fractionally more accurate at predicting clicks.
But not every business is like algorithmic trading or online ads, where a huge amount of online transactions occur every second, and algorithms only need to be right most of the time (Renaissance only profits on barely more than 50% of its trades). In many businesses, transactions occur in the physical world — they are not online and they don’t happen millions of times every second. As an example, in the commercial real estate business, a salesperson and his client make a deal after many rounds of private meetings. Clearly, it is difficult to obtain large volumes of accurate data to train models.
Thus, even in our age of surveillance where “a thousand satellites bloom, a trillion sensors sense,” we are often not able to obtain the data that is most critical for making predictions. This, however, could be a blessing that may not last very long.
