Machine Learning and the Stock Market
30 years ago, if you wanted to invest a couple thousand dollars you would’ve gone to a money, hedge-fund, or any other money manager.
Fast forward to now, and you have robo-apps, as they are called, and online investment “managers” that can automate everything that you invest, often leaving the user with just one decision to make, their risk profile.
Machine Learning is changing the landscape of the investment and money market in ways in which we could’ve never predicted. But before we get to that, let’s dive into machine learning and what it actually is.
Machine Learning is a subfield of artificial intelligence where the underlying mechanism lies in its ability to understand the structure of data and align that data into models that can then be understood by people to make predictions or decisions.
Machine Learning has evolved over time in the way that it processes data and the way that it learns that data. There is supervised learning and unsupervised learning. These methods can best be provided by the following examples of machines built to master the game of chess.
In 1997, an IBM Supercomputer called Deep Blue, had been “programmed using rules written by human players”(The Economist, “March of the Machines”, October 5th, 2019). This was supervised learning where the rules had been programmed into the computer by humans and the machine was able to play in a human style, but better and much faster than any human could.
Jump to 2017, and Google unveils “AlphaZero”, a computer that had been given the rules of chess and was able to teach itself how to play. This was unsupervised learning, where the learning algorithm was able to find patterns and models using input data and facilitate learning. It took four hours of training to beat the reigning supercomputer, Stockfish, which was at the time the best machine programmed by human tactics.
Now on to the stock market.
A quant fund is an investment fund that selects securities by utilizing advanced quantitative analysis. Managers build customized and specialized models using programming software to determine the best investments for fund portfolios.
Quant funds can also be divided into two groups — one that uses human strategies, and one that creates the strategies themselves. Before, investors would be able to test the models against historical data and make a judgement, whereas now, we start with the data and look for a pattern or hypothesis to make a prediction to aid in large decision making.
It is important to note the difference between correlation and regression in terms of the stock market. Correlation would be the measure of association between different variables that affect the market that are independent of each other. Regression models however, differ in that they quantify a dependent variable such as black debt, to independent variables that vary in the stock prices.
These machines have over time decreased the necessity of human beings in the decision making process. One fear that has arised is that these algorithms to predict market behavior may prompt more sudden shocks frequently in the market to create what we call “flash crises”. In 2014, “bond prices rallied sharply by more than 5% again in a matter of minutes”( The Economist, October 5th 2019, “March of the machines”). Many of these factors are due to the liquidity provided by high volume traders in the stock market that exacerbate the trading volatility.
However, on the other hand, it can be argued that the market is in better hands with algorithms and models, rather than humans, dictating the market. Computer after all, lack human emotion, which leads to an array of problems in personal investment.
There is a genuine fear however, what would happen if these quant funds actually successfully achieve what they were set out to do?
Stock markets are integral to economies of state, and dictate which competitors and which companies get the cash on hand that they need. It is coming to a point where these machine learning algorithms are able to pinpoint patterns and models that we humans have yet to be able to understand, if even ever. In that case, any edge gained by any such company that would have this technology, would give them an imperceptible advantage over anyone else. It would be inevitable though that these edges would eventually fall into the hands of every other company and would normalize the market.
One aspect we haven’t yet dived into is deep learning and its massive advantages it would give to the owners. Deep learning attempts to “imitate how the human brain can process light and sound stimuli into sound and hearing”. It uses non-linear processing to extract massive amounts of data, layer by layer, that serves as inputs to the successive layer.
Deep learning is able to absorb the most data and has been historically successfully been able to beat humans in cognitive tasks, especially those requiring financial analysis and concerning financial decision making.
In this context, deep learning would be able to absorb massive amounts of historical stock market data including previous crashes, crises, bull/bearish markets, and be able to aggregate that data into successive categories of variables that are then interpreted in the context of an algorithmic model. The concern here however, is that sometimes in the indexes of the stock market, human leverage or control is required to understand the complexities of what actually drives the stock market, rather than arbitrary variables contrived by computers to actually mean something.
Nonetheless, over time, this trend will only increase and we will see the market dominance over time by quant funds. In the image in the beginning of the article, you can tell that now, almost 40% of institutional shares of trading is being held by quant funds compared to less than 20% less than a decade ago.
To explain this, we have to understand that finance is highly nonlinear and the even the stock price data over time can seem completely random. Implementing traditional algorithmic models in a live trading system gives no guarantee of being able to incorporate new data that is added to the system. This is combated by neural networks, which by designed to be more effective in finding the relationships between data and using it to predict new data.
There are 5 stages to the workflow of a neural network stock market algorithm.
- ) Data Acquisition
- ) Data Preprocessing
- ) Development and Implementation of a Model
- ) Backtest Model
- ) Optimization
One simple form of neural network model is the Multilayer Perceptron (MLP), where simple data is fed as input into a model, and using particular weights on each of those data, the values are given to the hidden layers to produce an output.
Another form or neural network model, is the Long Term Short Model (LSTM), or in other words Recurrent Neural Networks (RNN). These have the ability to store certain information about that data for later use and are able to give the network the capability to understand and analyze the complexity of the structure of relationships between stock price data and historical stock price data and how they intertwine. As the number of layers increases, the learning ability or rate of the network is compounded several times over, and is able to decrease the gradient of anomalies in the market.
To implement these models, keras, an oft-used open-source neural network library written in Python, is used as it uses the idea of successively adding layers to the network rather than defining the entire network at once. This helps to combat the market’s notorious ability to flip at times and is able to accelerate the quick alteration of data given by this.
Despite all the worries and concerns, it’s inevitable that the market will be increasingly governed and run by quant funds/organizations that the only thing we can do at this point is to adapt and help to develop these fund algorithms according to human tactics and behavior models that can be seamlessly incorporated into quant models.
