Applied RL: Custom RL for Algo Trading — Data Preprocessing

We are going to build a custom Gym environment for multi-stock trading with a customized policy in stablebaselines3 using the PPO algorithm

Applied RL: Customized Deep Reinforcement Learning for Algorithmic Trading

Unlike most AI tasks, AI in Algo trading benefits from the fact that the datasets are readily available with financial data providers such as YahooFinance, Quantopian, or your broker. It also helps that most tasks in Financial time-series-based AI don’t have a strict requirement for annotation.

A sample data frame of the raw data used from one of the stocks used to build the multi-stock trading environment.

The data set we utilize is a dataset collected for 30 stocks from the Indian stock market alongside the Sensex index values to make up for a total of 31 assets to be traded by our agent.

We loop over all the files and load them to data frames while also adding some trade time-based features as such:

for filename in data_files:
    df = pd.read_csv(filename)
    df['datetime'] = pd.to_datetime(df['datetime'])
    name = df['name'].iloc[0]
    # Adding the Time of Day and Day of Week features
    df['ToD'] = df['datetime'].dt.hour + df['datetime'].dt.minute/60
    df['DoW'] = df['datetime'].dt.weekday/6
    df.sort_values(['timestamp'], inplace=True)

Then we add more features using TA-Lib, a library for technical analysis in python.

def add_features(tic_df):
    # Returns in the last t intervals
    for t in range(1, 11):
        tic_df[f'ret{t}min'] =                                                                         tic_df['close'].div(tic_df['open'].shift(t-1)).sub(1)
    # Simple Moving Average based features
        tic_df['sma'] = talib.SMA(tic_df['close'])
        tic_df['5sma'] = talib.SMA(tic_df['close'], timeperiod=5)
        tic_df['20sma'] = talib.SMA(tic_df['close'], timeperiod=20)

We also add signals for buy and sell based on other Algo-trading strategies such as moving average cross-overs.

tic_df['sma_buy'] = np.where((tic_df['close'] < tic_df['5sma']), 1, 0)

Overall we add a total of 50 features for each stock at each timestep making our observation space of the size of (number_of_stocks, number_of_timesteps, number_of_features).

We then start to join all the data frames into one single data frame to align the timestamps and pad the timestamps with a null value.

updated_df = add_features(df)
updated_df['datetime'] = pd.to_datetime(updated_df['datetime'])
updated_df = df.set_index(pd.DatetimeIndex(updated_df['datetime']))
# Replace infinities with zeroes
updated_df.replace([np.inf, -np.inf], 0, inplace=True)
dfs = pd.concat([dfs,updated_df], axis=1)
dfs.interpolate(method='pad', limit_direction='forward', inplace=True)

We then split the data frame into data frames of each stock so that we can pass a list of data frames to our environment. We also need to create a data frame to only contain the closing prices of each asset in that trading interval

cols_per_asset = int(len(dfs.columns)/num_assets)
df_list = []
price_df = pd.DataFrame()
for i in range(num_assets):
    df = dfs.iloc[:,i*cols_per_asset:i*cols_per_asset+cols_per_asset]
    df.drop(['datetime'], axis=1, inplace=True)
    price_df[names[i]] = df['close']
    df_list.append(df)
cols_per_asset -= 1

That’s it! We are done with preprocessing the data frame for trading and we can now initialize the environment.

env = MultiStockTradingEnv(df_list, price_df, stock_dim=num_assets, initial_amount=1000000, trade_cost=0, state_space=cols_per_asset, action_space=num_assets, window_size=12 frame_bound = (12,len(price_df)-1500), tech_indicator_list=indicators)

Okay! now that we have established how we preprocess our data and pass it to our RL environment we can now go ahead and start implementing the environment.

Applied RL: Custom Gym environment for multi-stock RL based Algo trading

Disclaimer: Any part of this tutorial must not be misconstrued as investment advice. Most algorithmic trading systems — even the best in backtesting — lose money when deployed in the market.

If you liked my work, please show some love and leave a comment. The code for this article is available in my Github repo. Please leave a star if you liked it and intend to use it in your work. Cheers!