Reinforcement Learning V.S Supervised Learning in Financial Markets
My opinion on why Reinforcement Learning is superior to Supervised Learning when it comes to Financial Markets.
We all agree that financial markets are at the heart of our modern economy and no doubt that they provide an important avenue for the sale and purchases of assets such as bonds, stocks, foreign exchange, and derivatives. However, to make profits from such markets, investors should study the market’s complicated and highly volatile environment. Let’s consider the stock market as an example, to profit from such a market you need to take into account all the possible political, economical and even environmental (like a pandemic) factors that affect the market movement and that makes trading a very difficult task for a human to do, that’s why in the past decade many efforts have been made to automatically generate successful deals in trading financial assets by designing adaptive systems that take advantage of markets while reducing the risk.
What is the problem with supervised models?
The majority of these adaptive systems were relying on Supervised Learning, which in essence train a predictive model on historical data to forecast the trend direction, but regardless of their popularity these supervised methods suffered from different limitations which led to sub-optimal results.
The main reason for supervised model limitations in financial markets is that trading financial assets is not only a process of predicting the future price as most supervised models do, but it also involves many other aspects that should be considered, such as the risk involved, where the supervised model seeks minimization of prediction error (maximization of return ) regardless of the risk, which it’s not in the interest of the investor, also exogenous constraints (e.g., lack of liquidity and transaction costs) are not being considered at all in most cases. Besides, financial markets data is extremely noisy, hence employing an algorithm with enormous learning capacity such as Neural Networks in such an environment will mostly lead to overfitting.
The aforementioned drawbacks of Supervised Learning can be tackled by using Reinforcement Learning.
So what is Reinforcement Learning (RL)?
In short, RL is an area of Machine Learning, concerned with how software agents ought to take actions in an environment. In the RL set-up, an autonomous agent, controlled by a Machine Learning algorithm, observes a state of its environment at a timestep. The agent interacts with the environment by taking an action in a current state, this action will cause a transition of both the agent and the environment to a new state. Reinforcement Learning will directly learn a trading strategy that integrates forecasting the price and the subsequent portfolio construction in one step to optimize the objective of the investor.
MDP, the secret ingredient.
The financial data is described as highly time-dependent (a function of time) data, making it a perfect fit for Markov Decision Processes (MDP), which is the core process of solving RL problems.
MDP captures the entire past data and defines the entire history of the problem in just the current state of the agent and that’s extremely crucial when it comes to model financial market data. Let’s assume a company with a prosperous history has recently released the annual report with heavy losses, a Supervised Learning algorithm would predict a good performance in the next year based on the good history of the company ignoring the current released report. Reinforcement Learning, on the other hand, will give more weight to the current event (The annual report) and the agent decision will be heavily impacted, no matter the history or goodwill of the company, this annual report will impact the transition from one state to another.
In the end, it would be nice to see more attention being devoted to RL in finance — in particular as the findings can be valuable for RL research in general.