Market regime detection using Statistical and ML based approaches

Overview

Financial markets microstructure behaviour can vary over time, being affected by a wide range of external and internal factors. This can result in different market regimes or periods of continuous similar market conditions. Financial market participants aim to detect these market regimes and their shifts to handle potential risks and make better-informed investment decisions. In the scope of this article, we aim to identify normal (growth) or crash (rapid decline) market states for S&P 500 using several statistical and ML models, including gaussian HMM, k-means clustering, and gaussian mixture models. Additionally, we build a simple investment strategy based on the identified market states and compare that with the base “Buy and Hold” strategy of the same asset throughout the analysis period.

The full article can be found on LSEG’s Developer Portal.

Data Ingestion and Engineering

Data Ingestion

In the article, we will be using futures contracts on S&P 500 (ESc1). In this section, we will show how to ingest historical prices for ESc1 using Refinitiv Data Libraries for Python, then prepare for feeding to regime detection algorithms.

Historical prices for S&P 500 futures

Data Engineering

In this section we provide a function which derive a new feature from our data which is the log return of the 7-day moving average of historical close prices. More on why we took this approach can be found in the main article.

Log returns on 7-day moving averages

Regime Detection

Building Regime Detection models

In this section, we briefly introduce several statistical and machine learning algorithms which we have used in this prototype to determine a market regime, including Agglomerative Clustering, Gaussian Mixture and Hidden Markov models. Then we will use OOP to implement the regime detection process on ESc1. Finally, we will present the results of both in-sample testing.

Market regimes identified by Hidden Markov Model from in sample testing

The details, complete codes, interpretations and other techniques can be found in the main article.

Rolling training and out of sample testing

Here, we introduce a function, which implements a rolling training and applies out of sample testing for the given model. The graph below presents the result from HMM model.

Market regimes identified by Hidden Markov Model from out-off-sample testing

Implementing an Investment Strategy

In this section, we implement a simple investment strategy based on the predicted hidden states, which is generating:

• A long signal when the market is expected to stay in a normal state
• A short signal when the market is in a crash or high volatility state

In this article, we have not built an end-to-end investment strategy considering transaction fees, slippage costs, margin calls, etc.; instead, we simply accumulate daily profit/loss based on the price change and our active position of the day. We then compare our strategy’s profit and loss results with a base Buy and Hold strategy. Below, are the results from the strategy:

References

• Refinitiv Data Libraries for Python
• Agglomerative clustering
• Gaussian Mixture Models
• Hidden Markov Models

Downloads

• Github — Market regime detection using Statistical and ML based approaches
• Explore API Samples on GitHub

Related Blueprints

• Detecting flash crash events using deep reinforcement learning agents — Dr. Skevofylakas Marios, Haykaz Aramyan
• Modelling and Evaluation — Artificial Intelligence regression and classification performance metrics — Skevofylakas Marios
• Introduction to Energy Modelling — A ML Approach — Jason Ramchandani
• Predicting M&A Targets Using ML: Unlocking the potential of NLP based variables — Haykaz Aramyan
• Explore all Blueprints