Financial Metaverse as a Playground for Financial Machine Learning
Echo TIME’s new article “The Metaverse Will Reshape Our Lives. Let’s Make Sure It’s for the Better” [1].
We believe the metaverse will also reshape our financial lives. Open-source community will make sure it’s for the better.
FinRL-Meta is open-source on GitHub, demonstrating the great potential of financial reinforcement learning.
Why Deep Reinforcement Learning?
Starting from the digitalization era late 20th century, computer science has penetrated into many fields. Algorithmic trading is certainly one. The trading form changed from physically locating at an exchange, to calling a broker, and to simply clicking a mouse or tapping on a smart phone. While the theme is always finding an edge and betting on it. The explosion of machine learning gave people a brand new edge: train an agent on financial big data and trade automatically.
Academia started to apply different machine learning methods to stock trading, trying to see whether the computer program can learn to trade from the market data. The development of financial big data provided abundant training data. Strengthened by neural networks, deep reinforcement learning (DRL) got into the sight of scholars. FinRL chose DRL instead of conventional supervise learning because of a certain advantage of DRL in financial tasks.
Value of Financial Big Data
An inevitable result of digitalization is the explosion of data. It’s crucial to know how to extract useful information from the sea of data, and the research of financial big data was born in this background. According to Marcos López de Prado’s book Advances in Financial Machine Learning [2], there are four essential types of financial data: fundamental data (assets, sales), market data (price, volume), analytics (analyst recommendations, credit ratings), alternative data (google searches, social media, metadata). Each type of data contains valuable information that could be used in certain area through different ways.
Fundamental data are mostly accounting data, released quarterly. They are usually public so that easily accessible, and the hidden information are simple and straightforward.
Market data cover data that useful in an exchange. They directly influence trading strategies made by people or agents. Market data are very abundant and are updated in an extremely high frequency. Our DRL agents mainly reference to this kind of data during training.
Analytics are data that manually processed from raw data like fundamental data, market data, or alternative data. The advantage is that analytics already extract important information from raw data. But the negative part is analytics could be expensive on price and subjective according to different analysts.
Alternative data could be produced by individuals, business processes, and sensors. They are “alternative” — not belonging to fundamental data and market data. There are difficulty on how to use alternative data in the correct way. But it’s definitely a huge advantage if a company can use it better than its competitors.
Importance of Market Environments
FinRL proved the feasibility of DRL in finance. We notice a large voice in the rapidly growing community for market environments. Environment plays an important role in DRL. It is a testbed embodied a MDP process, including state, action, and reward. A DRL agent interacts with an environment and learns from the generated data (transitions in the form of (state, action, reward, next_state)).
In financial reinforcement learning, we construct an environment using market data, for example, we use data of Dow 30 during a certain time period. State denotes current capital and share holdings, action denotes change of share holdings (buy, sell, or hold), and reward denotes the change of total property (capital + stock value). Appreciating the success of MuJoCo (a library that provides high quality environments for robotic tasks) in robotics, we build a library that provides standard OpenAI-gym style market environments for the community.
FinRL-Meta builds environments in a dynamic way. We connect with several markets such as stock, cryptocurrency, ETFs, etc. Users can choose a proper data source to fetch data, only filling in few arguments like starting date, ending date. Now, FinRL-Meta has supported more than 30 data sources. Thus, FinRL-Meta can easily provide environments with different data types and period based on users’ demand.
Demos and Benchmarks
Demonstration and tutorial are crucial to a library. They help users understand the contents and get start quickly. FinRL-Meta provides tens of Jupyter notebooks as demos, each with a different task or illustration of some code functionality. We encourage new comers to run few demos first and have a general idea of how FinRL-Meta works. Users familiar with this topic can use our notebooks as stepping stones for their own trading tasks. In the future, we will add more demos to serve the demands of the community.
Benchmarks lay the groundwork of further research. We reproduced several published papers, both giving users a broader view of current work, and demonstrating the validity of FinRL-Meta. Reproduced code and results are also given in Jupyter notebooks for users’ convenience.
The following figure shows how we collect the core notebooks and design a four-phase curriculum. Users with different levels of proficiency can navigate in a flexible manner.
Goals of FinRL-Metaverse
This project was previously called Neo-FinRL, which stands for Near real-market Environments for data-driven Financial Reinforcement Learning. We changed the name to FinRL-Meta because now we intend to go one step further, not limited to just environments, but also a metaverse as a playground for financial machine learning.
The roadmap and vision we have in mind are as follows:
- Curriculum for newcomers: We provide an educational curriculum, as shown in the figure above. Users can grow programming skills by gradually changing the data/environment layer following instructions on our website.
- Benchmarks on cloud: We provide demos on a cloud platform, Weights & Biases 3, to demonstrate the training process. We define the hyperparameter sweep, training function, and initialize an agent to train and tune hyperparameters. On the cloud platform Weights & Biases, users are able to visualize their results and assess the relative performance via community-wise competitions.
- Curriculum learning for agents: Based on FinRL-Meta (a universe of market environments, say ≥ 100), one is able to construct an environment by sampling data samples from multiple market datasets, similar to XLand. In this way, one can apply the curriculum learning method to train a generally capable agent for several financial tasks.
Moreover, it will be interesting to explore the evolutionary perspectives to simulate the markets. We believe that FinRL-Meta will provide insights into complex market phenomena and offer guidance for financial regulations and risk control.
References
[1] TIME’s article: The Metaverse Will Reshape Our Lives. Let’s Make Sure It’s for the Better, July 18th, 2022.
[2] Marcos Lopez De Prado. Advances in Financial Machine Learning. John Wiley & Sons, 2018.
