Experiments with denoising/shrinkage of a covariance matrix

I run some experiments on the impact of covariance matrix construction on portfolio stability. In particular, the following ways to generate a covariance matrix will be attempted:

  • Sample covariance — the most basic approach to construct a simple covariance matrix based on historical return data
  • Shrinkage of covariance matrix — additional shrinkage applied to all off-diagonal elements of the sample covariance matrix
  • Ledoit-Wolf shrinkage — additional tuning of shrinkage coefficient to achieve optimal mean square error between estimated and the real covariance matrix

Details of the implementation of the covariance matrix estimation methods can be found in [1].

We further assess the implication of shrinkage on portfolio optimization through the simulation of efficient frontiers with noisy asset returns.

Getting raw assets’ return data

Using historical returns from the Quantiacs Q18 NASDAQ-100 Stock Long-Short contest. Refer to https://quantiacs.com/documentation/en/user_guide/local_development.html


Function to construct efficient frontier

For further experiments, we will need a function constructing an efficient frontier based on a classical minimum variance optimization process. I could not find a suitable package for this task, so the section presents the implementation of efficient frontier construnction functions.

Min variance portfolio

(Yes, I’ve run it through ChatGPT to generate docstrings.)

Efficient frontier


Experimenting with the stability of portfolio construction

To visualize the impact of the noise in the covariance matrix, we take the approach suggested in one of the exercises in [2]:

Compute one hundred efficient frontiers by drawing one hundred alternative vectors of expected returns with a Normal-distributed noise with zero mean and 1% standard deviation.

Let’s begin with constructing the three versions of covariance matrices.

The level of noise in the covariance matrix is defined by its condition number, calculated as a ratio between the maximum and minimum eigenvalue of the matrix. We estimate this for the three covariance matrices constructed.

With a relatively aggressive shrinkage, the shrunk version of the covariance matrix achieved the largest reduction in the condition number, with Ledoit-Wolf implying more gentle shrinkage and hence moderate reduction in the condition number.

Compare reference efficient frontiers

Continuing to the main part of the experiment, we sample one hundred efficient frontiers based in all three versions of the covariance matrices, with the results presented below.


My take on the resulting graphs is as follows:

  • “Raw” sample covariance implies the best returns of the portfolio driven by higher diversification opportunities, however, the variance of the outcome can be extreme even for a relatively moderate noise in the assets returns;
  • Shrinkage is a powerful technique in tackling noise, but it is difficult to define the right level of shrinkage coefficient to balance the bias in the outcomes. In our results the deviation of the efficient frontiers from the source is quite significant for the shrinkage coefficient selected;
  • Ledoit-Wolf suggests the optimal out-of-the-box shrinkage, with significant improvement in the level of noise and moderate deviation from the source efficient frontier.


[1] https://scikit-learn.org/stable/modules/covariance.html#shrunk-covariance
[2] Marcos M. López de Prado “Machine Learning for Assets Managers”



Hi there, I am going to demonstrate some use cases of ML algorithms in market analysis and trading strategies. https://www.linkedin.com/in/alex-botsula-0421232

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alex Botsula

Hi there, I am going to demonstrate some use cases of ML algorithms in market analysis and trading strategies. https://www.linkedin.com/in/alex-botsula-0421232