Deep Learning for Stock Price Forecasting: A Comprehensive Tutorial with Python and the S&P 500

Due to the multitude of economic, political, and social factors that affect stock prices, predicting stock market trends is a very complex task. Yet deep learning advancements have opened up new possibilities for forecasting stock prices based on historical data.

The purpose of this article is to demonstrate how to use deep learning methods, specifically LSTM models, to predict future stock prices using Python. Based on the closing prices of the previous nine days, we will forecast the closing price of a single stock for the following day, and we will estimate the closing price of all stocks in the S&P 500 based on historical data.

To begin with, we will focus on the simpler scenario of forecasting a single stock’s closing price based on the previous nine days’ closing prices.

As a next step, we will examine all stocks within the S&P 500. In this tutorial, we will demonstrate how to obtain historical prices for all S&P 500 components from Wikipedia and Yahoo Finance, preprocess the data, train the model, and generate predictions. Furthermore, we will illustrate how the model’s performance can be visualized by comparing predicted prices and actual prices.

Once you have completed this article, you will have a thorough understanding of how deep learning can be employed for stock price prediction using Python, and how these techniques can be applied to a real-world dataset.

PART 1: Forecast for a single stock First, we can retrieve historical data from Yahoo Finance using the yfinance library. An example of how to download historical data for an individual stock symbol is shown in the following code cell.

import yfinance as yf
symbol = 'AAPL'
start_date = '2010-01-01'
end_date = '2023-02-17'
data = yf.download(symbol, start=start_date, end=end_date)

Visually displaying historical data is possible with the plot function.

import matplotlib.pyplot as plt
plt.plot(data['Close'])
plt.title(symbol + ' Close Price')
plt.xlabel('Date')
plt.ylabel('Price')
plt.show()

Based on historical data, we can now construct an LSTM model to forecast future prices. LSTMs are recurrent neural networks capable of learning from sequential data, such as time series data. Using Keras, a widely used deep learning library, the following code cell creates an LSTM model.

from keras.models import Sequential
from keras.layers import LSTM, Dense
import numpy as np
from sklearn.preprocessing import MinMaxScaler

lookback = 9
def create_dataset(dataset):
    X, Y = [], []
    for i in range(len(dataset)-lookback-1):
        X.append(dataset[i:(i+lookback), 0])
        Y.append(dataset[i+lookback, 0])
    return np.array(X), np.array(Y)
# Scale data
data = np.array(data['Close']).reshape(-1, 1)
scaler = MinMaxScaler(feature_range=(0, 1))
data = scaler.fit_transform(data)
# Train and test split
train_size = int(len(data) * 0.8)
train_data = data[:train_size]
test_data = data[train_size:]
train_X, train_Y = create_dataset(train_data)
test_X, test_Y = create_dataset(test_data)
train_X = np.reshape(train_X, (train_X.shape[0], train_X.shape[1], 1))
test_X = np.reshape(test_X, (test_X.shape[0], test_X.shape[1], 1))
# Create model
model = Sequential()
model.add(LSTM(50, input_shape=(lookback, 1)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(train_X, train_Y, epochs=50, batch_size=64, verbose=2)

With the create_dataset function, we generate sequences of prior prices along with their corresponding future prices based on the data divided into training and testing sets. After normalizing the data, we reshape it to conform to the LSTM model’s input format using a MinMaxScaler.

An LSTM model consists of an input layer, an LSTM layer containing 50 units, and a dense output layer. Fitting the model to the training data is done using the mean squared error loss function and the Adam optimizer.

The model can be applied to test data after it has been trained. Based on the 9 previous prices, the following code cell illustrates how to predict the 10th price.

# Reshape the test data
test_X = test_X.reshape((test_X.shape[0], lookback, 1))

# Use the trained model to make predictions on the test data
predictions = model.predict(test_X)
# Convert the predicted values back to their original scale
predictions = scaler.inverse_transform(predictions)
# Convert the actual values back to their original scale
actuals = scaler.inverse_transform(test_Y.reshape(-1, 1))
# Print the first 10 predicted and actual values
for i in range(10):
    print(f"Predicted: {predictions[i][0]:.2f} Actual: {actuals[i][0]:.2f}")

As a first step, we reshape the test data according to the input format of the LSTM model. Next, we use the predict function to generate predictions and reverse the scaling to obtain actual prices. Below are the first 10 predicted prices and their actual prices.

Using the subsequent code cell, we can also visualize the predicted prices over time.

plt.plot(actuals, label='Actual')
plt.plot(predictions, label='Predicted')
plt.title(symbol + ' Close Price')
plt.xlabel('Date')
plt.ylabel('Price')
plt.legend()
plt.show()

Orange represents predicted prices and blue represents actual prices. Although the model fails to predict the exact price, it captures the overall trend of the stock price.

PART 2: Forecast for all stock symbols

A LSTM model was used in the previous section to predict the future price of a single stock. Our analysis of all S&P 500 stocks will be expanded in this section.

Using the same start and end dates as before, we need to obtain historical data from Yahoo Finance for all S&P 500 components.

# Import necessary libraries
import yfinance as yf
import pandas as pd

# Download the list of S&P 500 components from Wikipedia
url = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'
table = pd.read_html(url, header=0)[0]
symbols = list(table.Symbol)
# Define the start and end dates for the historical data
start_date = '2010-01-01'
end_date = '2023-02-17'
# Download historical data for all S&P 500 components
data = {}
for symbol in symbols:
    print(f"Downloading data for {symbol}")
    df = yf.download(symbol, start=start_date, end=end_date)
    if len(df) >= 50:
        data[symbol] = df

Using the code above, we download historical data on all components of the S&P 500 and store it as dataframes in a dictionary where key values represent symbols. The symbol data is only stored if it is 50 or greater in length.

For each symbol, we can now use the same create_dataset and train_test_split functions as before. LSTM will be trained on all data except for the last day and used to forecast each symbol’s price on the last day.

# Import necessary libraries
from sklearn.preprocessing import MinMaxScaler
import numpy as np
from tensorflow import keras

# Define functions to create and split the dataset
def create_dataset(data, lookback=9):
    X, Y = [], []
    for i in range(lookback, len(data)):
        X.append(data[i-lookback:i, 0])
        Y.append(data[i, 0])
    return np.array(X), np.array(Y)
def train_test_split(data, test_size=0.2, lookback=9):
    train_size = int(len(data) * (1 - test_size))
    train, test = data[:train_size], data[train_size-lookback:]
    return train, test
# Define the lookback and test size
lookback = 9
test_size = 0.2
# Create an empty list to store the results
results = []
# Iterate through each symbol and its corresponding data
for symbol, df in dfs.items():
    print(f"Processing {symbol}")
    # Prepare the data
    data = df['Close'].values.reshape(-1, 1)
    scaler = MinMaxScaler(feature_range=(0, 1))
    data = scaler.fit_transform(data)
    train, test = train_test_split(data, test_size=test_size, lookback=lookback)
    train_X, train_Y = create_dataset(train, lookback=lookback)
    test_X, test_Y = create_dataset(test, lookback=lookback)
    # Train the model
    train_X = train_X.reshape((train_X.shape[0], train_X.shape[1], 1))
    test_X = test_X.reshape((test_X.shape[0], test_X.shape[1], 1))
    model = keras.models.Sequential([
        keras.layers.LSTM(50, input_shape=(lookback, 1)),
        keras.layers.Dense(1)
    ])
    model.compile(loss='mse', optimizer='adam')
    model.fit(train_X, train_Y, epochs=100, batch_size=32, verbose=0)
    # Make predictions on the test data
    test_X = test_X.reshape((test_X.shape[0], lookback, 1))
    predictions = model.predict(test_X)[-1]
    predictions = scaler.inverse_transform(predictions.reshape(-1, 1))[0]
    actual = df['Close'][-1]
    # Append the results to the list
    results.append((symbol, actual, predictions))

Using historical data for each stock, the code above iterates through all the S&P 500 components and trains an LSTM model. As a result, the model predicts the closing price for each stock on the last day of data availability.

Each tuple contains the symbol, the actual price, and the predicted price. The following code can be used to compare predicted prices with actual prices:

import matplotlib.pyplot as plt
plt.figure(figsize=(16, 12))
for symbol, actual, predicted in results:
    plt.plot([symbol]*2, [actual, predicted], 'o-', label=symbol)
# plt.legend()
plt.title('Predicted vs Actual Prices for S&P 500 Stocks')
plt.ylabel('Price ($)')
plt.show()

In the above code, the predicted and actual prices are plotted on the last day of data available for each stock. An x-axis represents the symbol, and an y-axis represents the price. A blue dot indicates the actual price, while an orange line indicates the predicted price.

For some stocks, the LSTM model offers reasonably accurate predictions, while for others, the predictions are less accurate. Stock markets are inherently unpredictable and influenced by multiple external factors, which is to be expected.

We have illustrated how to use deep learning, specifically LSTM models, to predict future stock prices using Python in this article. This analysis can also be broadened to encompass all stocks in the S&P 500, and the predicted prices can be compared to the actual prices.