End To End Python Implementation Of Finding Optimised Efficient Investment Portfolios

Explaining How To Build The Efficient Investment Portfolios From Start To End

Farhad Malik
Aug 24 · 11 min read

One of the milestones of the investment management application was to implement an end to end solution that starts by fetching company stock prices and builds a set of efficient and optimum portfolios using optimisation routines.

This article will demonstrate how to achieve it.

Please read FinTechExplained disclaimer. This application is based on my opinions and they could be wrong. Always seek advice from a professional financial advisor before investing your money.

1. Article Aim

This article will document the following key points:

  1. The Theory Of Efficient Frontier
  2. The Process Flow Of The End To End Solution
  3. Implementation in Python
  4. Path Of The Complete Code
  5. Next Steps

2. The Theory Of Efficient Frontier

We hear the term “efficient frontier” quite a lot but what does it mean exactly? And what is the Modern Portfolio Theory all about?

Burton G Malkiel expressed it in a perfect manner:

“The theory provides a firm foundation for the intuition that you should not put all your eggs in one basket and shows investors how to combine securities to minimize risk” (Burton G. Malkiel).

Key Note: Never put all of your eggs in one basket. Make sure you diversify!

Photo by Diggity Marketing on Unsplash
  • Let’s consider you have $10'000 of cash available and you are interested in investing it.
  • Your aim is to invest the money for a year.
  • Like any rational investor, you expect the final amount in a years time to be higher than the $10'000 amount you want to invest.

So far, so good!

Key Note: I am a strong believer in maintaining focus when we are attempting to reach a goal. And the key to meet that goal is to adapt swiftly to the changing environment.

Read this article if you want to gain an understanding of investment management:

Photo by Samuele Errico Piccarini on Unsplash

2.1 Understanding Risk-Return Relationship

There are many investment options available, such as buying a T-bill or company shares, etc. Some of the investment options are riskier than the others because they attract us to gain higher returns.

Hence, the point to note is that there exists a risk-return trade-off.

As an investor, our sole aim might be to invest in the least risky investment option that yields the highest return.

If we buy a number of assets such as shares of different companies then the total risk of the portfolio can be reduced due to diversification. This means that an investor can reduce the total risk and increase the return by choosing different assets with different proportions in a portfolio. This is due to the fact that the assets can be correlated with each other.

This brings the question: What Is A Good Portfolio?

A good portfolio is more than a long list of good stocks and bonds. It is a balanced whole, providing the investor with protections and opportunities with respect to a wide range of contingencies. — Harry Markowitz

The Modern Portfolio Theory of Nobel prize winner, Harry Markowitz can be implemented by using optimisation routines.

Photo by Will Tribe on Unsplash

3. The 7 Steps Of Efficient Frontier Process

Before I present the end-to-end implementation of the solution, I wanted to highlight a key technical note here. The core of the design of this application revolves around just one concept:

When the application runs, it will produce a large amount of data such as stock prices, their returns, covariance matrix, portfolios and their allocations along with their risk, return and Sharpe ratio. The Python code will also save all of this data in an Excel spreadsheet.

We need to perform the following 7 steps:

We Can Think Of The Steps As Three Main Stages

  1. The first stage (steps 1–3) is to calculate the daily returns of our stocks.
  2. The second stage is step 4 whereby we need to optimise the allocations to retrieve the best portfolio for each of our target returns.
  3. The last stage is about using the allocations to calculate the risk and return and plot the portfolios.
Photo by Ion Şipilov on Unsplash

4. Starting The Implementation In 3 Stages

This section will demonstrate the Python code and how I have implemented the solution. The following packages are required:

  • Pandas
  • Numpy
  • Matplotlib
  • SciPy

4.1 First Step Is To Fetch The Stock Prices

We built a component that fetched the past historical stock prices:

end_date = settings.get_end_date()
start_date = settings.get_start_date(end_date)
closing_prices = price_extractor.get_prices(settings.PriceEvent, start_date, end_date)
#plot stock prices & save data to a file
cp.plot_prices(closing_prices)
fr.save_to_file(closing_prices, 'StockPrices')

The stock prices are time-series data.

This is the snapshot of the data. The code stored the data in the StockPrices sheet of the excel spreadsheet as shown below.

As you can see, we cannot simply compare ZM with SYMC because the stock prices are of different magnitude. Hence, we need to compute the returns to standardise the stock prices.

4.2 We Need To Generate The Returns From The Prices

We need to compute their geometric returns by calculating the following equation:

The returns are generated so that we can standardised the stock prices so that they can be compared.

returns = settings.DailyAssetsReturnsFunction(closing_prices, settings.ReturnType)#plot stock prices & save data to a file
cp.plot_returns(returns)
fr.save_to_file(returns, 'Returns')

The returns are time-series data. This is the snapshot of the data which is stored in the Returns sheet of the excel spreadsheet as shown below.

4.3 Asset Expected Mean Returns And Covariance Matrix

There are essentially two steps involved:

expected_returns = settings.AssetsExpectedReturnsFunction(returns)
covariance = settings.AssetsCovarianceFunction(returns)
#Plot & Save covariance to file
cp.plot_correlation_matrix(returns)
fr.save_to_file(covariance, 'Covariances')

One of the most common ways is to compute the mean of the returns which is known as the expected returns.

To compute the asset expected mean return, we need the of the returns of each stock:

From the asset returns, we can compute the portfolio returns. The assets within the portfolio have been allocated a proportion of the total investment amount. As an instance, the portfolio might hold 40% of asset ABC and 60% of asset DEF.

Note: Are historical returns the right choice? Should we instead generate different measures? I will implement a superior application in the future that demonstrates how we can implement time-weighted returns instead.

The code then prepares the covariance matrix:

The covariance matrix is stored in the Covariances sheet of the Excel spreadsheet as shown below:

The volatility of the portfolio is the risk of the portfolio. The volatility is computed by calculating the standard deviation of the returns of each stock along with the covariance between each pair of the stocks by using this formula:

Volatility in this instance is the standard deviation i.e. the total risk of the portfolio.

Standard deviation measures the dispersion of the values around the mean.

4.4 Now Important Point: The Monte-Carlo Simulation & The Efficient Portfolios Line

Let’s first generate 100'000+ portfolios via Monte Carlo simulation. The allocations will be generated randomly. This technique relies upon the theory that as we repeat the experiment of finding the portfolio, we will eventually reach the formation of the true optimum portfolio.

This article demonstrates the Monte-Carlo simulation

If we plot the risk and return for each of the portfolios on a chart then we will see an arch line at the top of the portfolios.

This yellow line is essentially pointing at the portfolios that are the most efficient. This line is known as the efficient frontier.

The efficient frontier is a set of portfolios that give us the highest return for the lowest possible risk.

Every other portfolio that does not reside on the efficient frontier is not as efficient because it offers the same return as a portfolio on the efficient frontier but by taking a higher risk.

Any other portfolio is therefore inferior to the portfolios on the efficient frontier. As a result, we can literally ignore the portfolios that are not on the efficient frontier line.

Executing Monte-Carlo Simulator:

portfolios_allocations_df = mcs.generate_portfolios(expected_returns, covariance, settings.RiskFreeRate)portfolio_risk_return_ratio_df = portfolios_allocation_mapper.map_to_risk_return_ratios(portfolios_allocations_df)

The portfolios are stored in the MonteCarloPortfolios sheet of the Excel spreadsheet as shown below:

5. The Key Step — Portfolio Optimisation

We are going to use the SciPy package. The scipy.optimize module, within the SciPy package, offers a variety of optimisation algorithms. The module includes un/constrained algorithms, global optimisation routines, least-squares minimisation, scalar and multivariate minimisers etc.

It offers Newton-Krylov, Newton Conjugate Gradient, SLSQP, curve fit and dual annealing algorithms amongst others.

5.1 SciPy minimize function

There is a minimize function within the scipy.optimize module that performs the minimisation of a scalar function of one or more variables. The minimize function makes it easier for us to execute the required algorithm on an objective function.

The signature of the method is:

funx0args=()method=Nonejac=Nonehess=Nonehessp=Nonebounds=Noneconstraints=()tol=Nonecallback=Noneoptions=None

The key arguments are:

  • This is the objective function which we want to minimise. In our case, our portfolio is composed of a number of assets. We want to find the allocation of each asset that gives us the least amount of risk. The first parameter of the function is the 1-D array of allocations. We can also pass in a tuple of other parameters to the function too.
  • : This is the initial guess. We can assign a value of 1/number of assets, as the initial value of x.
  • : This is a tuple that can contain the extra optional arguments. As an instance, if our target risk function takes in the covariance matrix then we can pass it in the args tuple.
  • : This is a string argument where we can pass in the algorithm name such as SLSQP
  • : This is where we will pass in the constraints such as the sum of allocations should be 1.
  • : These are essentially the minimum and maximum pairs of each element in x. As an instance, we can indicate that we don’t want the value of allocation to be negative implying that we don’t want to sell an asset.

5.2 Portfolio Optimisation Routine

To perform constrained minimisation for multivariate scalar functions, we can use the minimize function using the SLSQP algorithm.

SLSQP stands for Sequential Least SQuares Programming.

def solve(self, x0, constraints, bounds, covariance):
return minimize(self.__risk_function, x0,
args=(covariance), method='SLSQP',
#prints covergence msgs
options={'disp': True},
constraints=constraints,
bounds=bounds)

5.3 Initial Guess And Bounds

We specified an initial guess of an equal proportion portfolio and a lower bound of 0 along with an upper bound of 1 on each of the allocations.

x0 = np.ones(self.__portfolio_size) * (1.0 / self.__portfolio_size)
bounds = ((0, 1),) * (self.__portfolio_size)

5.4 Constraints

We also specified the constraints. The idea is to minimise a function of x subject to functions of constraints. The constraints can be linear and non-linear. They are defined as dictionaries. Constraints on the allocations are added which ensure that we maximise return and the sum of the allocations is 1. The keys of the dictionary include:

  • : This specifies the type of constraints such as ineq (inequality) or eq (equality). If you have an equality constraint such as:

A + B = C then it will be represented as A+B-C

def con():
return A + B - C

It would be equality (type='eq') constraint, where you make a function that must equal zero:

def con(t):
return A + B - C

If you have an inequality constraint such as:

A + B ≥ C then it will be represented as A+B-C

def con():
return A + B - C

It would be an inequality (type='ineq') constraint, where you make a function that must be greater than C:

def con(t):
return A + B - C
  • : The function of the constraint such as the sum of allocations of assets in the portfolio should be equal to 1.
constraints=[]constraints.append({'type': 'eq', 'fun': lambda inputs: 1.0 - np.sum(inputs)})
constraints.append({'type': 'eq', 'args': (returns,),
'fun': lambda allocations, returns:
my_return - self.__return_function(returns, allocations)})

5.5 Execution Of Portfolio Optimiser

optimiser = obj_factory.get_optimiser(targets, len(expected_returns.index))    
portfolios_allocations_df = optimiser.generate_portfolios(expected_returns, covariance, settings.RiskFreeRate)
portfolio_risk_return_ratio_df = portfolios_allocation_mapper.map_to_risk_return_ratios(portfolios_allocations_df)
#plot efficient frontiers
cp.plot_efficient_frontier(portfolio_risk_return_ratio_df)
cp.show_plots()
#save data
print('7. Saving Data')
fr.save_to_file(portfolios_allocations_df, 'OptimisationPortfolios')
fr.close()

As expected, the chart plotted an arch line. We specified a list of our target returns.

def get_my_targets():
return np.arange(0, 1.5, 0.05)

For each of the returns, the optimiser ran the optimisation routine and produced the most optimum and best portfolios which are known as the efficient frontier.

For each target, it returned the most optimum portfolio. The array of allocations are the weights (proportions) of the assets which make up the portfolio for each of our target return

These are the optimised target portfolios.

6. Complete Code

The complete code is uploaded on GitHub

7. Next Steps:

We have a long way to go but we are getting there! The next set of steps is about:

  1. Adding superior measures to calculate the return
  2. Enhanced risk metrics to calculate the risk of the portfolio
  3. Implementing a forward-looking correlation matrix
  4. Allocation of weights to the actual amount
  5. Company extractor to fetch us the right companies and enriching additional information such as sector, volume, ratings etc
  6. Adding additional constraints on the sector, currency, ratings, etc
  7. Appropriate allocations by units of shares

8. Notes Worth Mentioning

  • We need to consider transaction costs.
  • Additionally, we need to remember that the past does not always dictate the future accurately.
  • The risk and return measures are not superior and have many flaws. We need to explore other measures such as weighted average returns and expected shortfall risk measures along with forward looking covariance matrix.
  • We need to start considering the factor models.

9. Summary

This article documented the following sections:

  1. The Theory Of Efficient Frontier
  2. The Process Flow Of The End To End Solution
  3. Implementation in Python
  4. Path Of The Complete Code
  5. Next Steps

FinTechExplained

This blog aims to bridge the gap between technologists, mathematicians and financial experts and helps them understand how fundamental concepts work within each field. Articles

Farhad Malik

Written by

My personal blog, aiming to explain complex mathematical, financial and technological concepts in simple terms. Contact: FarhadMalik84@googlemail.com

FinTechExplained

This blog aims to bridge the gap between technologists, mathematicians and financial experts and helps them understand how fundamental concepts work within each field. Articles

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade