Modelling Emergent Behavior in Financial Agents with Q-Learning: Phase 2

Continuing where I left off…

Previously, I talked about using Q-Learning for trading, modelling in NetLogo, and a glimpse into financial networks which provided some idea of how all of it would come together. Given I’ve worked towards achieving that goal since my last publication, I will elaborate more on that now, starting with network design and analysis.

Network Analysis

The distribution of connectivity in an environment is critical in fostering diffusion processes, or the spread of sentiment. A network highly dense in connections expected to exhibit more diffusion. The extent to which an individual agent is connected indirectly to other agents through intermediary agents will determine the level of clustering, which in turn will have a significant effect on cooperating behavior. High clustering, for example, can help reinforce behaviors that exhibit complementarities. Furthermore, the level of susceptibility to neighbors’ sentiment and segregation patterns can then also either facilitate or hinder this diffusion.

After taking this into account, I decided that the model should have some control over network structure and the propensity of each trading agent to adopt the sentiment of agents it is connected to, as well as the regularity in the network since under some regularity conditions, agents in large networks will converge to the same beliefs. (Recall the notion of regularity in graphs, illustrated in the following graph again)

Figure 1. Regularity and randomness in networks

In order to control the degree of regularity, we employ the Maslov-Sneppen rewiring algorithm. This method takes the initial network and generates a series of randomized graphs by iterating a fundamental rewiring step that preserves the desired properties. This algorithm will preserve the degree of every vertex with the sequence of steps shown below. After enough iterations, this procedure generates a randomized variant of the original network, or more specifically, a null model.

Figure 2. An illustration of the Maslov-Sneppen rewiring algorithm

Emulating a financial market

Now that we have a better understanding of financial networks themselves, let’s look at how to proceed. NetLogo offers us the ability to create networks as either immobile 2-D (or 3-D) lattices or as irregularly placed agents connected through links. For my purpose I adopted a network that matched relevant criteria which included external news input to emulate influencing factors in markets and inherent bounded rationality in agents, with this bounded rationality arising from different interpretation of the news. The external news was then modelled as a Gaussian distribution with controllable mean and deviation. Note that the mean in this context would just represent the overall sentiment arising out of the news while the deviation would denote the volatility in the markets. Recall the older model from Phase 1, which had a patch-based approach. For my new, complex network based approach, we can see how it plays out below.

Figure 3. The interface for the Artificial Financial Market model which I implemented

The randomization aspect is controlled through the parameter R above, which is just the number of iterations for the Maslov-Sneppen rewiring approach. At each stage, the collective sentiment of the agents determine the logarithmic price of the security being traded in. The GUI shows the live status in the top right. The trading agents are turtles connected with links, with the shade of red and green indicating the degrees of negative and positive sentiments respectively.

Establishing a Simulation Pipeline

In the last post, I mentioned I would use NL4Py, a controller package in Python for NetLogo. By utilizing NL4Py, we were able to take the steps illustrated in Figure 4 towards an integrative model which would link the model in NetLogo with the Python script dictating each agent’s move through prior Q-Learning.We used Pandas to extract the simulation data from the Artificial Financial Market in NetLogo and query the state in the Q-table where the state was based on a combination of technical and fundamental indicators of the stock.

Figure 4. A pipeline showing the processes required to achieve successful simulation

Some Results and Future Steps

The simulation has yielded some interesting trends with regards to localized bubbles and phase transitions. For example, observer Figure 5. In this case, we had a moderate positive bias in news (0.05), but the volatility was relatively high at 0.20. This scenario reflected a market with high optimism accompanied by high speculation. While the overall trend is an increase in the logarithmic price, we do see a rapid rise followed by an almost equally rapid descent which corresponds to a series of negative events as well as fast propagation due to higher interconnectivity and propensity to be influenced by link partners.

Figure 5. The logarithmic price as informed by the previous price and the trader sentiment at each point in time.

Crises are amplified if the epicenter is better integrated into the trade network, while target agents affected by such a shock are in turn better able to dissipate the impact if they are well integrated into the network, which can be explained with a network-based approach. One possible future step would be to use the model to emulate this and target certain agents within our model, which will feed ripple movements throughout the network. This could perhaps be achieved by restricted access to information, in line with the notion of bounded rationality. The information will then propagate in some manner throughout the network, and the response can then be analyzed over time. I could also conduct a sensitivity analysis on the parameters in the model which are controllable through NetLogo.

References

Goncalves, Carlos Pedro (2003). “Artificial Financial Market.” In:http://ccl.northwestern.edu/netlogo/models/community/ArtificialFinancialMarket

Gunaratne, Chathika and Ivan Garibay (2018). “NL4Py: Agent-Based Modeling in Python with Parallelizable NetLogo Workspaces.” In: arXiv preprint arXiv:1808.03292

Jackson, Matthew O, Brian W Rogers, and Yves Zenou (2017). “The economic consequences of social network structure.” In: Journal of Economic Literature 55.1, pp. 49–95.

Kali, Raja and Javier Reyes (2010). “Financial contagion on the international trade network.” In: Economic Inquiry 48.4, pp. 1072–1101

Maslov, Sergei and Kim Sneppen (2002). “Specificity and stability in topology of protein networks.” In: Science 296.5569, pp. 910–913.

.