On Building Diversified Portfolios That Outperform Out-of-Sample

Careful, It is all about prior

james rafco
Nov 1, 2020 · 6 min read
Image for post
Image for post

Quantitative finance is all about finding trading strategies using past data to generate returns in the future. In a machine learning dialect, futures returns are by definition out-of-sample so any trading strategy that (out)performs out-of-sample is the Graal.

A few years ago a paper with a new portfolio technic has been released, Building Diversified Portfolios that Outperform Out-of-Sample. The paper has been downloaded thousands of times and is ranked in the top 300 most viewed papers on SSRN.

The author introduces the Hierarchical Risk Parity (HRP) portfolio and claims its superiority to minimum variance portfolios in producing better out-of-sample returns, at least for long-only fully invested allocation. The paper includes the source code to reproduce the results, not that common for a financial paper, and that deserves to be recognized.

We won’t go into technical details, the paper being available but will highlight some findings. The HRP is compared to two other traditional portfolios. The Inverse Variance Portfolio (IVP) and the Markowitz portfolio’s solved using the Critical Line Algorithm (CLA). Both are very known and common in finance.

Back to 1952

Mean-variance and minimum variance allocation were developed by Harry Markowitz and published in 1952. The global idea is to minimize the portfolio variance using the correlation between assets and possibly expected returns. Computer sciences were far from what they are today and Markowitz developed the CLA algorithm to solve his portfolio problem. Fortunately today we have quadratic programming and we can solve the problem in a few lines of code using cvxpy .

Reproducing the paper

In the paper, the CLA portfolio is implemented in the getCLA function. In fact, it is the minimum variance (MV) portfolio, the last turning point of the CLA algorithm. That is why the function returns cla.w[-1].flatten() .

CLA version of the MV portfolio

As we couldn’t get the CLA class working out of the paper, a cvxpy version,getMV has been coded:

In addition to IVP and CLA portfolios, one other portfolio has been added, the Equal Risk Contribution portfolio (ERC) by Maillard et al. (2010). This portfolio allocation method is a more realistic benchmark to HRP:

  • It takes into account the correlation structure as HRP
  • It does not suffer from instability due to covariance matrix inversion as HRP
  • It equalizes the contribution of each asset has to portfolio risk (the variance).

Just as MV and IVP portfolios, the ERC portfolio has a clearly defined objective function where the solution is optimal. To our knowledge, HRP has no objective function to optimize so it is less clear to understand under which condition the HRP is optimal and its mathematical properties.

As we can see in the next table we can reproduce the portfolios exactly and the CLA (from the paper) matches our MV:

Image for post
Image for post
Comparison of allocations (in %)

We observe that while HRP concentrates 62.57% in the top five holdings, the ERC portfolio has only 55.36%, giving a much better diversification in the weights space. Now let’s have a look at the risk contributions, the risk being defined as the portfolio variance:

Image for post
Image for post
Comparison of risk contributions (in % of the total risk)

By definition ERC equalizes the contribution of each asset to the portfolio risk, so each asset has a 10% contribution. The MV has the property of having the weights equal to the risk contribution. As you can see HRP still have high-risk concentrations on asset 4 and 10, but much better than the MV which suffers from weights concentration.

Now we have all the ingredients to recompute the simulations and even more. Let’s first run the Monte-Carlo simulations that reproduce the main finding of the paper. We are running 1000 simulations with the same parameters as in the paper. The variance of each portfolio allocation is evaluated out-of-sample and reported in the table below. The rank shows the best portfolio in terms of out-of-sample variance (the lower the better).

Image for post
Image for post
Monte-Carlo simulations with shocks and correlations

We observe the same figures as the paper. The HRP overperforms both the IVP portfolio and the MV in terms of out-of-sample variance. The out-of-sample variance of the MV portfolio is the worst, about 76% greater than HRP. But the interesting part is that HRP underperforms the ERC, with an out-of-sample variance 25% lower. Now the question is have we found a portfolio allocation that is better than the HRP?

Beating the unbeatable

The answer is we can not conclude at all. Actually, we can not even conclude that ERC or HRP is better than MV and IVP portfolios. Indeed, Markowitz never had a chance, here is why.

First, the MV portfolio is optimal for a Gaussian distribution because it is characterized by its mean and its covariance, the two moments used in Markowitz's theory. Adding shocks and you are losing the optimality as you are adding a third and a fourth moment. As the next table shows, the Monte-Carlo experiment is far from a Gaussian distribution as the skewness and kurtosis are far from 0 for multiples assets.

Image for post
Image for post

Let’s remove the shocks and run again the simulation. The table now looks different:

Image for post
Image for post
Monte-Carlo simulations without shocks and with correlations

The MV portfolio is now the best portfolio to hold, or at least you can not really tell that it is worst than HRP and ERC in terms of minimizing the out-of-sample variance. The problem was not correlation or the matrix inversion, it was the shocks. Matrix inversion is a bigger problem when the number of observations (days in time series) is larger than the number of assets (features). The IVP is the worst, having a variance 20% higher.

Second, the IVP does not take correlations into account. In fact, the IVP portfolio is the MV portfolio when the correlation between the assets is null. So, the same assumptions hold. Adding correlations and shocks and you are losing the optimality of IVP. Let’s remove the correlations and run again.

Image for post
Image for post
Monte-Carlo simulations without shocks and correlations

This time none of the allocation leads but IVP would be the fastest to compute and the right to pick.


What we can conclude is the following: given the assumptions of Gaussian returns with shocks and correlations as simulated in the experiment, ERC seems to be a better choice than HRP. But, both ERC and HRP seem to be robust to shocks. When there are no shocks, all allocations perform the same and are better choices than IVP.

All results are available on Google Colab.

To use correctly any portfolio allocation, we need to have, at least, a prior on the underlying returns distribution. Having the wrong prior yields to the wrong allocation. In the end, the portfolio that beats them all stays to be discovered.

The Startup

Medium's largest active publication, followed by +756K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store