Reinforcement Learning in Finance
Reinforcement Learning has immense applications in the field of finance. The recent availability of large datasets has made the application of Machine learning and Deep learning possible in finance. Many problems in finance also fall into the sequential decision making paradigm. This means we can apply Reinforcement learning in one form or the other. We will look at some of the papers applying RL to various problems in finance. This list is by no means exhaustive as the field is continuously evolving but this should be a good starting point for anybody looking to get started. Some of the areas in which it can be applied are as follows :
- Market Making
- Portfolio Management
- Options Pricing
- Trading
- Optimal Execution
Market Making
Market makers are traders ( individual or institutions ) who provide liquidity by placing buy & sell orders in the LOB. They earn a profit equal to the bid-ask spread while doing so. Market makers provide the market with liquidity and depth while profiting from the difference in the bid-ask spread. Market makers are compensated for the risk of holding assets because they may see a decline in the value of a security after it has been purchased from a seller and before it’s sold to a buyer.
Few papers discussing RL in Market Making:
- Robust Market Making via Adversarial Reinforcement Learning
- An Electronic Market-Maker
- Optimal Market Making by Reinforcement Learning
- Reinforcement Learning for Market Making in a Multi-agent Dealer Market
- Optimizing Market Making using Multi-Agent Reinforcement Learning
- Market Making via Reinforcement Learning
Trading
Trading is buy and selling securities or financial instruments with the objective of earning a profit. Trading is one of the fields where Machine learning has immense potential. Traders usually look at price information and estimate if a stock is going to go up or down. Trading can be done at different horizons. It can right from intraday ( holding periods of a few minutes to hours ) to multi-day or multi-month holding periods. Different trading strategies use different datasets for better accuracies. Some of the datasets which can be used are Price ( OHCLV data ), News, Fundamental information, Economic data etc.. . ML models are built with these datasets with the objective of predicting the future direction or return of the stock or stocks.
Trading is also a sequential decision making problem and fits naturally in the framework of RL.
Few Papers discussing RL in trading :
- FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance
- Deep Reinforcement Learning in Quantitative Algorithmic Trading: A Review
- Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning
- Deep Reinforcement Learning for Trading
- An Application of Deep Reinforcement Learning to Algorithmic Trading
Portfolio Management
Portfolio Management is the process of selecting assets ( sometimes from different classes i.e stocks, bonds or cash etc.. ) to maximise a certain objective. This objective is usually to maximise profit along with some notion of risk management. Portfolio Management involves the following main steps
- Portfolio Allocation : Allocation refers to the task of selecting which assets are part of the portfolio and how much weightage it receives. Changing the allocation can impact the performance of the portfolio quite drastically.
- Diversification : Diversification refers to the selection of ‘diverse’ or uncorrelated assets. This ensures that we pick assets which behave differently thereby reducing our risk. Picking uncorrelated assets ensures we are hedged against market movements or cycles. This is essentially of form of risk management.
- Rebalancing Portfolio : Market conditions are always changing due to evolving economic or political news or situations. Therefore our Portfolio Management system should be able to respond to these changes and reallocate & diversify over time. This process of changing allocation periodically or in response to events is called rebalancing. This ensures our portfolio aligns with the market regime and performs accordingly.
Few Papers discussing RL in Portfolio Management :
- Detecting and adapting to crisis pattern with context based Deep Reinforcement Learning
- Continuous-Time Mean-Variance Portfolio Selection: A Reinforcement Learning Framework
- Reinforcement Learning Portfolio Manager Framework with Monte Carlo Simulation
- MAPS: Multi-agent Reinforcement Learning-based Portfolio Management System
Options Pricing
Options are derivatives i.e the options price is derived from an underlying stock or asset price. An options contract offer the holder to buy or sell the underlying asset at a particular price. The holder is not obligated to buy or sell the asset but its an option. Each contract has an expiration date by which the holder should excise their option. The stated price of the option is called the strike price.
Options are priced using the well known Black-Scholes Merton pricing equation. The idea behind options pricing is that it can be priced as a portfolio of other tradable assets. This is known as dynamic option replication. The option is replicated using a portfolio of shares of the underlying asset and cash component. The portfolio is continuously rebalanced to match the option price. Cash inflow or withdrawal is not allowed in the portfolio.
Few papers discussing RL in Options Pricing :
- Dynamic Replication and Hedging: A Reinforcement Learning Approach
- QLBS: Q-Learner in the Black-Scholes (-Merton) Worlds
- The QLBS Q-Learner Goes NuQLear: Fitted Q Iteration, Inverse RL, and Option Portfolios
- Gamma and Vega Hedging Using Deep Distributional Reinforcement Learning
Optimal Execution
Optimal execution is a fundamental problem in financial modelling. The simplest version is the case of a trader who wishes to buy or sell a given amount of a single asset within a given time period. This is a challenge because any activity in the market ( buying or selling ) has a market impact. For example if you are buying a large amount of shares then the avg price you will get is higher because the action of buying will increase the prices and similarly if you are selling the price will decrease giving you a lower price. The market impact is always negative hence the trader who is buying or selling will have to take into account this impact else it can have negative impact on the transaction.
Few papers discussing RL in Optimal Execution :
- Double Deep Q-Learning for Optimal Execution
- Multi-Agent Deep Reinforcement Learning for Liquidation Strategy Analysis
- Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution
- A reinforcement learning extension to the Almgren-Chriss framework for optimal trade execution