S&P500 vs Stock picking : Building a stock market portfolio using Data Mining ⛏️

Uncover the Power of Data Mining in Crafting a Market-Beating Portfolio.

“ …However, we report majority underperformance in 11 out of 17 fixed income categories, topping out at 95% for actively managed Government Intermediate funds. … ” SPIVA® U.S. Scorecard

S&P500 vs Stock picking

Introduction
1. Understanding the S&P 500
2. The Art of Stock Picking
3. Data Mining in Stock Market Analysis
4. Building a Portfolio Using Data Mining
5. Comparing Strategies
6. The Future of Investing with Data Mining
Conclusion

Introduction

In the ever-evolving landscape of investment strategies, two schools of thought have long vied for supremacy: the passive approach of tracking major indices like the S&P 500 and the active pursuit of stock picking. The S&P 500, a barometer of the U.S. stock market’s health, offers a straightforward path to investing by mirroring the performance of 500 leading companies. On the other hand, stock picking is an art form that requires a blend of intuition, research, and sometimes, a bit of luck. It’s a method favored by those who seek to beat the market by selecting individual stocks they believe are poised for success.

But as we stand on the cusp of a data-driven era, a new contender has entered the arena: data mining. This technique, which involves extracting patterns and insights from large datasets, is revolutionizing the way we approach stock market analysis. With the wealth of financial data available on platforms like Yahoo Finance, investors now have the tools to sift through the noise and uncover investment opportunities that might otherwise go unnoticed.

This article aims to demystify the process of building a stock market portfolio using data mining and compare it with the traditional approach of investing in the S&P 500. We will delve into the historical performance data from Yahoo Finance, explore the methodologies behind data mining, and examine whether a synergy of technology and analysis can truly give investors an edge. Whether you’re a seasoned investor or taking your first steps into the world of finance, understanding these strategies is crucial in making informed decisions that align with your financial goals.

1. Understanding the S&P 500

The S&P 500, or Standard & Poor’s 500, is a stock market index that serves as a barometer for the overall health of the United States economy. Comprising 500 of the largest publicly traded companies in the U.S., it represents approximately 80% of the total market capitalization, making it a key indicator of the market’s overall performance.

Composition of the S&P 500

The S&P 500 is a market-capitalization-weighted index, meaning that companies with larger market values have a greater impact on the index’s movements. This weighting method reflects the actual economic footprint of the companies and ensures that the index movements are representative of shifts in the broader economy.

Advantages of Index Investing

Investing in the S&P 500 offers several advantages. Firstly, it provides instant diversification, as purchasing an S&P 500 index fund or exchange-traded fund (ETF) is similar to holding a small piece of 500 different companies. This diversification helps to mitigate company-specific risk.

Moreover, index investing is often associated with lower management fees compared to active management. Funds that track the S&P 500 simply aim to replicate the performance of the index, which requires less intervention and analysis by fund managers, thus resulting in lower costs for the investor.

Historical Performance

Historically, the S&P 500 has provided solid returns over the long term. According to data from Yahoo Finance, the index has experienced an average annual return of around 10% before inflation over the last 90 years. While past performance is not indicative of future results, the S&P 500’s track record has made it a popular choice for investors seeking to build wealth over time.

S&P 500 as a Benchmark

The S&P 500 is also widely used as a benchmark against which the performance of stocks, mutual funds, and investment strategies are measured. Professional money managers often compare their returns to the S&P 500 to demonstrate their value in terms of beating the market. For individual investors, it serves as a reference point to gauge the success of their investment choices.

2. The Art of Stock Picking

Stock picking is the practice of selecting individual stocks to invest in, with the aim of outperforming the market or a benchmark index like the S&P 500. It’s a strategy that requires a blend of analytical skills, market knowledge, and often, a bit of intuition.

Fundamental Analysis

At the core of stock picking is fundamental analysis, which involves delving into a company’s financial statements to evaluate its viability and growth potential. Investors scrutinize earnings reports, balance sheets, and cash flow statements to assess factors such as profitability, debt levels, and operational efficiency. Ratios like price-to-earnings (P/E), debt-to-equity, and return on equity (ROE) are key metrics that can help determine whether a stock is undervalued or overvalued.

Technical Analysis

Some investors also employ technical analysis, which is the study of statistical trends gathered from trading activity, such as price movement and volume. Unlike fundamental analysts, technical traders are more concerned with patterns and signals on charts that suggest future activity, rather than a company’s financials.

Qualitative Analysis

Qualitative factors also play a significant role in stock selection. This includes evaluating the company’s business model, competitive advantage, management quality, and market share. Factors such as industry growth, brand loyalty, and regulatory environment can also influence a stock’s potential.

Risk and Reward

The allure of stock picking lies in the potential for significant returns if one can identify undervalued stocks or predict industry trends. However, it comes with higher risks, as individual stocks can be subject to volatile swings due to market sentiment, earnings reports, and other factors.

Diversification and Portfolio Management

While stock picking can lead to high rewards, it’s also important to manage risk through diversification. A well-rounded portfolio typically includes a mix of stocks across different sectors and industries, reducing the impact of any single stock’s poor performance.

Behavioral Aspects

Investor psychology is another critical aspect of stock picking. Emotional biases and market sentiment can heavily influence investment decisions. Successful stock pickers often have the discipline to stick to their strategies and avoid the pitfalls of emotional trading.

In essence, the art of stock picking is about finding the hidden gems in the market that can provide substantial returns. It requires diligence, patience, and a deep understanding of market dynamics. For those with the time and inclination to research and analyze individual stocks, stock picking can be a rewarding strategy that offers the potential for above-average returns.

3. Data Mining in Stock Market Analysis

Data mining in stock market analysis is the process of using algorithms and statistical methods to uncover patterns and insights from large datasets. This approach has gained popularity with the advent of big data and advanced computing power, allowing investors to make more informed decisions.

Understanding Data Mining

Data mining involves the extraction of hidden predictive information from vast amounts of data. In the context of the stock market, it can help identify stock trends, forecast market movements, and develop trading strategies. Techniques such as machine learning, artificial intelligence, and neural networks are commonly used to analyze historical data and predict future stock performance.

Machine Learning and Predictive Analytics

Machine learning algorithms can learn from and make predictions on data by building models from sample inputs. In stock market analysis, these models can be trained on historical price data, financial ratios, and other relevant metrics to forecast future stock prices or market trends.

Quantitative Analysis

Quantitative analysis is a data-driven approach that relies on mathematical and statistical models. It’s used to evaluate investments and identify trading opportunities by processing large datasets to find patterns that are not easily visible to the human eye.

Big Data in Finance

The financial sector generates a massive amount of data daily, including stock prices, trading volumes, and economic indicators. Big data technologies enable the processing and analysis of this data in real time, providing a significant advantage in market analysis and decision-making.

Challenges and Considerations

While data mining can provide valuable insights, it also comes with challenges. Overfitting, where a model is too closely tailored to past data, may not perform well in real-world trading. Additionally, the stock market is influenced by unpredictable factors such as political events and investor sentiment, which can be difficult to quantify and incorporate into models.

4. Building a Portfolio Using Data Mining

Building a portfolio using data mining involves leveraging advanced analytics to make informed investment decisions. By extracting patterns and insights from historical and real-time data, investors can construct a portfolio that aligns with their risk tolerance and investment goals.

In this section, I am going to show step by step the entire process of creating a portfolio using S&P500 data from yahoo finance by displaying the code used. Here, we are working with jupyter notebook to write our python code. For some part, I am going to insert the code but for others I have to insert an image to illustrate the work.

a . Install and import all the dependancies

!pip install pandas-ta

pandas-ta is a Technical Analysis library useful to do feature engineering from financial time series datasets (Open, Close, High, Low, Volume). It is built on Pandas and Numpy. Here is the documentation for more details : Technical analysis in Python

from statsmodels.regression.rolling import RollingOLS
import pandas_datareader.data as web
import matplotlib.pyplot as plt
import statsmodels.api as sm
import pandas as pd
import numpy as np
import datetime as dt
import yfinance as yf
import pandas_ta
import warnings
warnings.filterwarnings('ignore')

import matplotlib.pyplot as plt
import seaborn as sns

The different dependancies are just basics for data analysis in python. We are using pandas, numpy and ploting tools.

b . Download/Load SP500 stocks prices data

The code begins by fetching the list of S&P 500 companies from Wikipedia, which is a common reference for investors looking to analyze large-cap U.S. equities. The pd.read_html function is used to scrape the table containing the S&P 500 companies directly into a pandas DataFrame.

We can see a couple of rows in the image below

We observe that the dataset comprises 994,448 rows and 6 columns, encompassing market data from the years 2025 to 2023. This ensures that our analysis incorporates the most up-to-date information available.

We can check display the price variation from 2015 to 2023 of Apple stock as example.

# Sélectionner uniquement les données pour AAPL
aapl_data = df.loc[df.index.get_level_values('ticker') == 'AAPL']

# Tendances temporelles pour AAPL
plt.figure(figsize=(12, 6))
aapl_data['adj close'].plot(label='AAPL', color='blue')
plt.title('Tendances Temporelles pour AAPL')
plt.xlabel('Date')
plt.ylabel('Prix des Actions (Adj Close)')
plt.legend()
plt.show()

It’s evident that Apple’s stock has been on an upward trajectory since 2015. An initial investment made at the beginning of this period could have yielded substantial profits. Considering a stock picking strategy and examining the returns, if one had purchased Apple stock in 2015 at a price of $25 and it has now reached nearly $200 in 2023, the investment would have seen a significant appreciation in value.

c . Calculate features and technical indicators for each stock.

Following the data acquisition phase, the next step in building a data-driven portfolio is to calculate features and technical indicators for each stock. These indicators are crucial for analyzing market trends and making informed investment decisions. Here are some of the key technical indicators that you might calculate:

• Garman-Klass Volatility: This measure provides an estimate of a stock’s volatility based on high, low, opening, and closing prices. It’s particularly useful for capturing the price movement within a trading day.

df['garman_klass_vol'] = ((np.log(df['high'])-np.log(df['low']))**2)/2 - 
(2*np.log(2)-1)*((np.log(df['adj close'])-np.log(df['open']))**2)

This line calculates the Garman-Klass volatility for each stock, which is a measure of volatility that takes into account the high, low, opening, and closing prices.

• RSI (Relative Strength Index): The RSI is a momentum oscillator that measures the speed and change of price movements. It can help identify overbought or oversold conditions in a stock.

df['rsi'] = df.groupby(level=1)['adj close'].transform(lambda x: 
pandas_ta.rsi(close=x, length=20))

The RSI is calculated using the pandas_ta library, which is applied to the adjusted closing price of each stock over a window of 20 periods.

• Bollinger Bands: These are volatility bands placed above and below a moving average. The bands widen or contract based on the volatility of prices, helping to identify potential overbought or oversold levels.

df['bb_low'], df['bb_mid'], df['bb_high'] = zip(*df.groupby(level=1)['adj close']
.transform(lambda x: pandas_ta.bbands(close=np.log1p(x), length=20).iloc[:, [0, 1, 2]]))

Bollinger Bands are calculated for the lower, middle, and upper bands using the pandas_ta library. The np.log1p function is used to apply a log transformation to the adjusted close prices before calculating the bands.

• ATR (Average True Range): The ATR is an indicator that measures market volatility by decomposing the entire range of an asset price for that period.

def compute_atr(stock_data):
    atr = pandas_ta.atr(high=stock_data['high'],
                        low=stock_data['low'],
                        close=stock_data['close'],
                        length=14)
    return atr.sub(atr.mean()).div(atr.std())

df['atr'] = df.groupby(level=1, group_keys=False).apply(compute_atr)

The ATR is calculated using a custom function that applies the pandas_ta.atr function to each group of stock data. The result is standardized by subtracting the mean and dividing by the standard deviation.

• MACD (Moving Average Convergence Divergence): This trend-following momentum indicator shows the relationship between two moving averages of a stock’s price. It can signal changes in the strength, direction, momentum, and duration of a trend in a stock’s price.

def compute_macd(close):
    macd = pandas_ta.macd(close=close, length=20).iloc[:,0]
    return macd.sub(macd.mean()).div(macd.std())

df['macd'] = df.groupby(level=1, group_keys=False)['adj close'].apply(compute_macd)

The MACD is calculated using a custom function that applies the pandas_ta.macd function to the adjusted close price of each stock. The result is standardized in a similar manner to the ATR.

• Dollar Volume: This is a measure of the total amount of money traded in a particular stock over a given period. It’s calculated by multiplying the volume of shares traded by the price of the shares.

df['dollar_volume'] = (df['adj close']*df['volume'])/1e6

The dollar volume is calculated by multiplying the adjusted close price by the volume of shares traded, then dividing by 1 million to scale the values.

You can see the full code in the following image :

After running this code, the DataFrame df will have new columns for each of the calculated technical indicators, which can be used for further analysis and to inform investment decisions.

Here is the output :

d . Filtering the 150 most liquid Stocks

This part is crucial for a bunch of reasons. In fact, focusing on the 150 most liquid stocks is useful for several reasons, particularly when implementing a stock picking strategy:

• Ease of Entry and Exit: Liquid stocks have higher trading volumes, which means that large quantities of shares can be bought or sold without significantly affecting the stock price. This is crucial for investors who may need to enter or exit positions quickly without incurring substantial slippage costs.
• Price Discovery: Liquid stocks tend to have tighter bid-ask spreads, which means the buying price and selling price are closer together. This leads to better price discovery and ensures that the stock price more accurately reflects the underlying value and market sentiment.
• Reduced Transaction Costs: Trading in liquid stocks often results in lower transaction costs because the tight bid-ask spread means less money is lost when crossing the spread. This is particularly important for strategies that involve frequent trading.
• Market Representation: The most liquid stocks are often large-cap stocks that have a significant impact on the market and are usually part of major indices like the S&P 500. By focusing on these stocks, your portfolio is more aligned with the broader market trends.
• Financial Stability: Highly liquid stocks are typically associated with well-established companies that have a stable financial background. This can reduce the risk of investing in companies with uncertain futures.
• Information Efficiency: Liquid stocks are closely followed by many investors and analysts, which means information is quickly incorporated into the stock price. This leads to a more efficient market and allows investors to make decisions based on the most current information available.

First of all, we created a different data-frame that only contains the indicators calculated below. Those indicators help a lot for checking volatility of an asset in the market.

Typically, liquidity is measured by the volume of shares traded and the dollar volume. Since we have already calculated the dollar volume, we can use this as a primary measure of liquidity.

Here’s a step-by-step approach to select the 150 most liquid stocks:

• Calculate Average Dollar Volume: Compute the average dollar volume for each stock over the period you’re analyzing to get a sense of its typical liquidity.
• Rank Stocks by Liquidity: Rank the stocks based on their average dollar volume, with the highest values representing the most liquid stocks.
• Select Top Stocks: Select the top 150 stocks from this ranking to focus on for your portfolio.

Here’s how we implemented this in code:

data['dollar_volume'] = (data.loc[:, 'dollar_volume'].unstack('ticker').rolling(5*12, min_periods=12).mean().stack())

data['dollar_vol_rank'] = (data.groupby('date')['dollar_volume'].rank(ascending=False))

data = data[data['dollar_vol_rank']<150].drop(['dollar_volume', 'dollar_vol_rank'], axis=1)

data

We can see the output below.

Here are out 150 most liquid stocks based on the previous calculations.

e . Calculate stocks returns for different periods

The function calculate_returns is designed to calculate the investment returns for each stock over different time periods, specified as lags in months.

def calculate_returns(df):

    outlier_cutoff = 0.005

    lags = [1, 2, 3, 6, 9, 12]

    for lag in lags:

        df[f'return_{lag}m'] = (df['adj close']
                              .pct_change(lag)
                              .pipe(lambda x: x.clip(lower=x.quantile(outlier_cutoff),
                                                     upper=x.quantile(1-outlier_cutoff)))
                              .add(1)
                              .pow(1/lag)
                              .sub(1))
    return df


data = data.groupby(level=1, group_keys=False).apply(calculate_returns).dropna()

data

After running this code, the data DataFrame will have new columns for each lag period, named return_1m, return_2m, return_3m, return_6m, return_9m, and return_12m, representing the investment returns over 1, 2, 3, 6, 9 and 12 months, respectively.

f . Fama-French Factors et Rolling Factor Betas

After calculating the investment returns for each stock, the next step in your analysis is to incorporate the Fama-French factors and calculate rolling factor betas for each stock. The Fama-French three-factor model is an asset pricing model that expands on the capital asset pricing model (CAPM) by adding size risk (SMB, Small Minus Big) and value risk (HML, High Minus Low) to the market risk factor.

Here’s an outline of what you would typically do in this step:

• Obtain Fama-French Factors: You need to acquire the historical factor data for the market risk premium, SMB, and HML. This data is often available from research databases or directly from the websites of researchers like Kenneth French.
• Merge Factor Data with Stock Returns: Combine the Fama-French factor data with your stock returns data, ensuring that the dates align correctly.
• Calculate Rolling Betas: For each stock, calculate the rolling betas with respect to each Fama-French factor. Rolling betas are calculated using a rolling window (e.g., 36 months) to estimate how sensitive a stock is to each factor over time.
• Regression Analysis: Perform a regression analysis for each stock over the rolling window, with the stock’s excess returns as the dependent variable and the Fama-French factors as independent variables. The coefficients from these regressions are the rolling betas.
• Store the Results: Save the rolling beta coefficients for each stock and each factor. These betas can be used to understand the stock’s exposure to different risk factors and to build a diversified portfolio.

Here is the implementation on how we calculated the rolling factor bettas

factor_data = web.DataReader('F-F_Research_Data_5_Factors_2x3', 'famafrench',
                               start='2010')[0].drop('RF', axis=1)

factor_data.index = factor_data.index.to_timestamp()

factor_data = factor_data.resample('M').last().div(100)

factor_data.index.name = 'date'

factor_data = factor_data.join(data['return_1m']).sort_index() 

factor_data

Here, we deleted the stock with less than 10 months of data.

observations = factor_data.groupby(level=1).size()

valid_stocks = observations[observations >= 10]

factor_data = factor_data[factor_data.index.get_level_values('ticker').isin(valid_stocks.index)]

factor_data

• The rolling factor

betas = (factor_data.groupby(level=1,
                            group_keys=False)
         .apply(lambda x: RollingOLS(endog=x['return_1m'],
                                     exog=sm.add_constant(x.drop('return_1m', axis=1)),
                                     window=min(24, x.shape[0]),
                                     min_nobs=len(x.columns)+1)
         .fit(params_only=True)
         .params
         .drop('const', axis=1)))

betas

After that, we join the rolling factors data to the principal dataframe.

factors = ['Mkt-RF', 'SMB', 'HML', 'RMW', 'CMA']

data = (data.join(betas.groupby('ticker').shift()))

data.loc[:, factors] = data.groupby('ticker', group_keys=False)[factors].apply(lambda x: x.fillna(x.mean()))

data = data.drop('adj close', axis=1)

data = data.dropna()

data.info()

g . Applying K-means clustering to group similar stocks

Applying K-means clustering to group similar stocks based on their characteristics is a common technique in portfolio management to create clusters of stocks with similar features, which can help in diversification and risk management. Here’s how we can apply K-means clustering to your stock data:

# Code permettant de faire le clustering sur chaque mois en fixant un nombre de cluster de 4
# avec une initialisation random des centroides

from sklearn.cluster import KMeans

# data = data.drop('cluster', axis=1)

def get_clusters(df):
    df['cluster'] = KMeans(n_clusters=4,
                           random_state=0,
                           init='random').fit(df).labels_
    return df

data = data.dropna().groupby('date', group_keys=False).apply(get_clusters)

data

The image below displays the results generated by the aforementioned function, illustrating the assigned cluster for each stock on a monthly basis.

We have four distinct groups of stocks, each categorized based on their unique characteristics. Our stock selection strategy will utilize these diverse groups as a foundation for constructing our investment portfolio.

g . Select stocks based on cluster for building a portfolio, for each month

To select stocks for building your portfolio based on their cluster assignments each month, you can create a function that iterates through each cluster and selects stocks according to your predefined criteria.

def select_stocks_by_cluster(data, selection_criteria):
    portfolio_selection = {}
    clusters = data['cluster'].unique()
    
    for cluster in clusters:
        # Filter data for the current cluster
        filtered_df = data[data['cluster'] == cluster].copy()
        filtered_df = filtered_df.reset_index(level=1)
        filtered_df.index = filtered_df.index + pd.DateOffset(1)
        filtered_df = filtered_df.reset_index().set_index(['date', 'ticker'])
        
        # Get unique dates
        dates = filtered_df.index.get_level_values('date').unique().tolist()
        
        # Apply selection criteria for each date
        for d in dates:
            tickers = filtered_df.xs(d, level=0).index.tolist()
            selected_tickers = selection_criteria(tickers)
            portfolio_selection[d.strftime('%Y-%m-%d')] = selected_tickers
    
    return portfolio_selection

# Define your selection criteria function
def selection_criteria(tickers):
    # Implement your selection logic here
    # For example, select a subset of tickers based on your criteria
    return tickers[:5]  # Placeholder: select the first 5 tickers

# Apply the function to your data
portfolio = select_stocks_by_cluster(data, selection_criteria)

In this extended function, selection_criteria is a placeholder for the actual logic we used to select stocks. We can replace this with our specific criteria, such as selecting the top-performing stocks based on past returns, lowest volatility, or any other metric that suits your investment strategy. The portfolio dictionary will then contain the selected stocks for each date, organized by cluster.

The image below presents the output, where a set of 5 chosen stocks for each month is displayed, representing our stock selection process.

h. Define a portfolio optimization function

The importance of the portfolio optimization step lies in its ability to systematically allocate capital across the selected stocks in a way that aims to maximize returns for a given level of risk, or alternatively, to minimize risk for a given level of expected return. Here are the key reasons why this step is crucial:

• Maximizing the Sharpe Ratio: By optimizing for the Sharpe ratio, you are effectively seeking the highest possible expected return per unit of volatility (risk). A higher Sharpe ratio indicates a more efficient portfolio with better risk-adjusted returns.
• Risk Management: Portfolio optimization takes into account the covariance between the returns of the assets, which helps in diversifying the portfolio. Diversification can reduce the portfolio’s overall risk because the poor performance of some investments may be balanced by the better performance of others.
• Constraint Satisfaction: Applying constraints on the weights of individual stocks ensures that the portfolio is not overly concentrated in a few assets, which can expose you to idiosyncratic risks. Setting a minimum and maximum weight for each stock helps maintain a balanced and diversified portfolio.
• Systematic Approach: Using a mathematical and systematic approach to determine the optimal weights of the stocks in the portfolio removes emotional biases and subjective judgment from the investment process.
• Adaptability: The optimization process can be repeated periodically to adjust the portfolio weights in response to changing market conditions or changes in the investor’s risk appetite, investment horizon, or financial goals.

Here is an implementation of the optimization function :

This process is important because it uses the latest available data to ensure that the portfolio is optimized based on the most recent market conditions. By regularly updating the portfolio weights using fresh data and the optimization function, we can maintain a portfolio that is aligned with our investment goals and risk tolerance.

The next step involves iterating over a set of rebalancing dates, optimizing the stock weights for each period to maximize the Sharpe ratio, calculating the weighted log returns for the portfolio, and compiling these into a single DataFrame that tracks the portfolio’s strategy returns over time. If the optimization fails, equal weights are used. The final DataFrame is cleaned to remove any duplicate entries, ensuring an accurate representation of the portfolio’s performance. Here is the code :

returns_dataframe = np.log(new_df['Adj Close']).diff()

portfolio_df = pd.DataFrame()

for start_date in fixed_dates.keys():

    try:

        end_date = (pd.to_datetime(start_date)+pd.offsets.MonthEnd(0)).strftime('%Y-%m-%d')

        cols = fixed_dates[start_date]

        optimization_start_date = (pd.to_datetime(start_date)-pd.DateOffset(months=12)).strftime('%Y-%m-%d')

        optimization_end_date = (pd.to_datetime(start_date)-pd.DateOffset(days=1)).strftime('%Y-%m-%d')

        optimization_df = new_df[optimization_start_date:optimization_end_date]['Adj Close'][cols]

        success = False
        try:
            weights = optimize_weights(prices=optimization_df,
                                   lower_bound=round(1/(len(optimization_df.columns)*2),3))

            weights = pd.DataFrame(weights, index=pd.Series(0))

            success = True
        except:
            print(f'Max Sharpe Optimization failed for {start_date}, Continuing with Equal-Weights')

        if success==False:
            weights = pd.DataFrame([1/len(optimization_df.columns) for i in range(len(optimization_df.columns))],
                                     index=optimization_df.columns.tolist(),
                                     columns=pd.Series(0)).T

        temp_df = returns_dataframe[start_date:end_date]

        temp_df = temp_df.stack().to_frame('return').reset_index(level=0)\
                   .merge(weights.stack().to_frame('weight').reset_index(level=0, drop=True),
                          left_index=True,
                          right_index=True)\
                   .reset_index().set_index(['Date', 'index']).unstack().stack()

        temp_df.index.names = ['date', 'ticker']

        temp_df['weighted_return'] = temp_df['return']*temp_df['weight']

        temp_df = temp_df.groupby(level=0)['weighted_return'].sum().to_frame('Strategy Return')

        portfolio_df = pd.concat([portfolio_df, temp_df], axis=0)

    except Exception as e:
        print(e)

portfolio_df = portfolio_df.drop_duplicates()

portfolio_df

5. Comparing Strategies

In this Section , we delve into a critical analysis where the performance of our custom stock picking strategy is meticulously evaluated against a well-established market benchmark, the S&P 500 index. This comparison is pivotal as it provides a clear perspective on the efficacy of the stock selection and portfolio optimization techniques employed in our strategy. By juxtaposing the returns of our tailored portfolio with the SPY ETF — a proxy for the S&P 500’s performance — we can discern whether our approach has succeeded in generating excess returns or if it has fallen short of the broad market’s performance. This benchmarking exercise not only highlights the relative success of our investment decisions but also informs potential adjustments to enhance future strategy outcomes. It is a testament to the importance of rigorous performance evaluation in the realm of investment strategy development.

Here is the code for juxtaposing the S&P500 returns with our strategy ‘s performance.

spy = yf.download(tickers='SPY',
                  start='2015-01-01',
                  end=dt.date.today())

spy_ret = np.log(spy[['Adj Close']]).diff().dropna().rename({'Adj Close':'SPY Buy&Hold'}, axis=1)

portfolio_df = portfolio_df.merge(spy_ret,
                                  left_index=True,
                                  right_index=True)

portfolio_df

The output :

To visualize the comparative performance of our custom stock picking strategy alongside the S&P 500 index, we’ll employ matplotlib to create a cohesive graph that plots the returns of both strategies on the same chart. This graphical representation will allow us to easily compare the trajectories of the returns and gain immediate insights into how our strategy stacks up against the market benchmark over time. The implementation in matplotlib will enable us to produce a clear and informative visual aid for our analysis.

import matplotlib.ticker as mtick

plt.style.use('ggplot')

portfolio_cumulative_return = np.exp(np.log1p(portfolio_df).cumsum())-1

portfolio_cumulative_return[:'2023-09-29'].plot(figsize=(16,6))

plt.title('Unsupervised Learning Trading Strategy Returns Over Time')

plt.gca().yaxis.set_major_formatter(mtick.PercentFormatter(1))

plt.ylabel('Return')

plt.show()

Presented below is the output graph that illustrates the comparison between the returns of our custom stock picking strategy and the S&P 500 index.

The output graph reveals that following the 2020 COVID-19 market crash, the S&P 500’s buy-and-hold approach has consistently outperformed our custom stock picking strategy. Despite this, there have been instances where our strategy nearly matched the benchmark’s performance, indicating moments of close competition. This observation underscores the resilience and potential of our strategy, even in the face of significant market upheavals, and suggests areas for further refinement to enhance its competitiveness.

6. The Future of Investing with Data Mining

As we stand on the cusp of a new era in finance, the role of data mining in investing is becoming increasingly pivotal. The vast amounts of data generated daily hold untapped potential that, when harnessed correctly, can lead to significant competitive advantages. In this section, we explore the transformative impact that data mining and advanced analytics are expected to have on the future of investing.

The advent of sophisticated machine learning algorithms and artificial intelligence has opened up new frontiers for identifying patterns, trends, and correlations within complex datasets that were previously inaccessible or too intricate to analyze. These technologies enable investors to predict market movements with greater accuracy, optimize portfolio allocations, and manage risks in ways that were not possible before.

Moreover, the rise of alternative data sources, such as social media sentiment, satellite imagery, and transactional data, provides a richer, more granular view of market dynamics. Investors who can effectively integrate and interpret these diverse data streams are likely to gain an edge in identifying undervalued assets and anticipating shifts in consumer behavior.

However, the proliferation of data mining in investing also raises important questions about privacy, data security, and ethical use of information. As regulations struggle to keep pace with technological advancements, the industry must navigate these challenges with care to maintain trust and integrity in the markets.

Furthermore, the democratization of data and analytics tools has leveled the playing field, allowing retail investors to access resources that were once the exclusive domain of institutional players. This shift is likely to spur innovation and competition, leading to more inclusive and efficient markets.

Conclusion

In conclusion, our journey through the development and evaluation of a custom stock picking strategy has provided valuable insights into the complexities of portfolio management. By comparing our strategy’s performance with the S&P 500 benchmark, we have highlighted the importance of benchmarking and the need for continuous improvement. The practical considerations discussed serve as a reminder of the myriad factors that can influence investment outcomes. Looking ahead, the integration of data mining and advanced analytics into investing practices holds the promise of more informed and dynamic strategies. As the financial industry continues to embrace technological innovation, investors who adapt and learn to harness the power of data will be well-positioned to thrive in the markets of tomorrow.

References :

• SPIVA® U.S. Scorecard
• corcel.io
• Algorithmic Trading