Deploying ANN Model using HyperParameter Tuning to Predict Remaining Useful Life of a Turbofan Engine
This post is continuation to my earlier post to predict Remaining Useful Life (RUL) on Turbofan Engine using Engine parameters.
Through these posts and my upcoming posts i will be exploring how we can utilize & deploy available data science frameworks to optimize lifecycle of assets in Industrial Systems.
In earlier post i have predicted RUL using various ML Models. In this post i have predicted RUL of engine using Deep Learning ANN Model and Hyperparameter Tuning using Keras Regressor.
Dataset and description about dataset can be referred through my earlier post.
Dataset contains:
Train trjectories: 100 engines ,Test trajectories: 100 engines. Fault Modes: ONE
- Datasets include simulations of multiple turbofan engines over time, each row contains the following information:
- Engine unit number
- Time, in cycles
- Three operational settings
- 21 sensor readings.
Deep Learning in short is replicating how our brain process input and responds. It is also described as multiple layer of Neural Network.
Most good thing in Deep Learning models is that you don’t have to perform feature engineering and feature extraction steps performed on data while feeding to ML Models.
Library used to build ANN Model is Keras which is open source library build over Tensorflow librarary.Keras is a powerful and easy-to-use free open source Python library for developing and evaluating deep learning models. It wraps the efficient numerical computation libraries Theano and TensorFlow and allows you to define and train neural network models in just a few lines of code.
STEP 1 : Importing Necessary Libraries
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
pd.set_option("display.max_rows", None)
STEP 2 : Importing the Dataset
# define column names for easy indexing
index_names = ['unit_nr', 'time_cycles']
setting_names = ['setting_1', 'setting_2', 'setting_3']
sensor_names = ['s_{}'.format(i) for i in range(1,22)]
col_names = index_names + setting_names + sensor_names# read data
train = pd.read_csv('train_FD001.txt',sep='\s+', header=None, names=col_names)
test = pd.read_csv('test_FD001.txt',sep='\s+', header=None, names=col_names)
y_test = pd.read_csv('RUL_FD001.txt', sep='\s+', header=None, names=['RUL'])# Train data contains all features (Unit Number + setting parameters & sensor parameters)
# Test data contains all features (Unit Number + setting parameters & sensor parameters)
# Y_test contains RUL for the test data.
train.head()
train['unit_nr'].unique()# There are 100 no unique engines.
Note: for Test Dataset Since the true RUL values for the test set are only provided for the last time cycle of each engine, therefore the test dataset will be subsetted to represent the same.
# Since the true RUL values (y_test) for the test set are only provided for the last time cycle of each enginge,
# the test set is subsetted to represent the same
test = test.groupby('unit_nr').last().reset_index().drop(['unit_nr','time_cycles'], axis=1)y_test.shape# RUL value for 100 no of engines.
Step 3: Data Pre-Processing
train.describe()
# Remove setting_3 column as we can see that it's value is not changing theerfore will not not add any information to our
# prediction
train=train.drop('setting_3',axis=1)Now we will add RUL (Remaining Useful Life) to the train dataset.
RUL = Max Life Time Cycle of Engine-Current Time Cycle
Max Life Time Cycle of Engine means time when engine has degraded.
def add_remaining_useful_life(df):
# Get the total number of cycles for each unit
grouped_by_unit = df.groupby(by="unit_nr")
max_cycle = grouped_by_unit["time_cycles"].max()
# Merge the max cycle back into the original frame
result_frame = df.merge(max_cycle.to_frame(name='max_cycle'), left_on='unit_nr', right_index=True)
# Calculate remaining useful life for each row
remaining_useful_life = result_frame["max_cycle"] - result_frame["time_cycles"]
result_frame["RUL"] = remaining_useful_life
# drop max_cycle as it's no longer needed
result_frame = result_frame.drop("max_cycle", axis=1)
return result_frametrain = add_remaining_useful_life(train)
train[sensor_names+['RUL']].head()
Step 4 : Seperating Train & Test Data
# train & y_train
# Calculated RUL variable is our Target variable.
y_train=train['RUL']
X_train=train.drop(['RUL'],axis=1)
X_train.head(5)# Test data set , keeping only train columns/features.
X_test=test[X_train.columns]
X_test.head(5)# Clipping of RUL at 125 as after 125 , RUL is responding to the sensor values after this value.
# This is done to improve performance of the applied models.y_train= y_train.clip(upper=125)
Step 5: Building Deep Learning ANN Model
Performing scaling of features using standardisation. Scaling is important in order to make sure gradient descent algorithm converges to global minimum point fast.
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train1 = sc.fit_transform(X_train)
X_test1 = sc.transform(X_test)Importing necessary libraries
# Importing the Keras libraries and packages
import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LeakyReLU,PReLU,ELU
from keras.layers import Dropout
from tensorflow.keras.optimizers import RMSpropCreating Model Architecture
# Initialising the ANN
regressor = Sequential()# Adding the input layer and the first hidden layer
regressor.add(Dense(20, kernel_initializer = 'he_uniform',activation='relu',input_shape=(12,)))# Adding the second hidden layer
regressor.add(Dense(20, kernel_initializer = 'he_uniform',activation='relu'))# Adding the third hidden layer
regressor.add(Dense(20, kernel_initializer = 'he_uniform',activation='relu'))# Adding the Fourth hidden layer
regressor.add(Dense(20, kernel_initializer = 'he_uniform',activation='relu'))# Adding the Fifth hidden Layer
regressor.add(Dense(20, kernel_initializer = 'he_uniform', activation = 'relu'))# Adding the Sixth hidden layer
regressor.add(Dense(20, kernel_initializer = 'he_uniform', activation = 'relu'))# Adding the Seventh hidden layer
regressor.add(Dense(20, kernel_initializer = 'he_uniform', activation = 'relu'))# Adding the output layer
regressor.add(Dense( 1, kernel_initializer = 'glorot_uniform', activation = 'linear'))# Compiling the ANN
#regressor.compile(optimizer = 'adam', loss = 'mean_absolute_error', metrics = ['mean_absolute_error'])
regressor.compile(optimizer = RMSprop(0.001), loss = 'mean_absolute_error', metrics = ['mean_absolute_error'])# RMSprop is optimizer like adam. 0.01 is learning
Training the model using our data
regressor.fit(X_train1,y_train,epochs=100)Epoch means one complete cycle of weight initialisation through forward propagation and weight updation through backward propagation.
These terms can be learned in detailed through this excellent article explaining about neural networks with proper examples.
Also one refer to following youtube playlist on Deep Learning.https://www.youtube.com/playlist?list=PLZoTAELRMXVPGU70ZGsckrMdr0FteeRUi
During trailing objective is to update weights in a way that our loss function gets minimized.
Predicting results on Train data
y_hat=regressor.predict(X_train1)
r2_score(y_train,y_hat)
R2 score value explains how much variance in your target variable (Here it is RUL) is explained by the independent features (Here we have sensor values + setting values)
Predicting Result on Test Data
y_hat_test=regressor.predict(X_test1)
r2_score(y_test,y_hat_test)
Step 6: ANN Model with HyperParameter Tuning using KERAS Regressor
Hyperparameter optimization or tuning is the technique of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are learned or final output of learned algorithm or model.
In Deep learning (ANN Model) Hyperparameter includes number of neurons , number of layers , no of epochs , batch size , weight initialization methods types , learning rate , activation function.
Here we will be dealing with batch size , number of epochs.
# Goal: tune the batch size and epochs# Import KerasRegressor class
from keras.wrappers.scikit_learn import KerasRegressor
from keras.layers import LeakyReLU# Import the GridSearchCV class
from sklearn.model_selection import GridSearchCV# Define the model trhough a user-defined function
def create_model():
model = Sequential()
# Defining Input Layer
model.add(Dense(15, activation=LeakyReLU(alpha=0.1),kernel_initializer = 'he_uniform', input_shape=(12,)))
# Defining first Hidden Layer
model.add(Dense(15, activation=LeakyReLU(alpha=0.1),kernel_initializer = 'he_uniform'))
# Defining second Hidden Layer
model.add(Dense(15, activation=LeakyReLU(alpha=0.1),kernel_initializer = 'he_uniform'))
# Defining third Hidden Layer
model.add(Dense(15, activation=LeakyReLU(alpha=0.1),kernel_initializer = 'he_uniform'))
# Defining Fourth Hidden Layer
model.add(Dense(15, activation=LeakyReLU(alpha=0.1) ,kernel_initializer = 'he_uniform'))
# Defining Fifth Hidden Layer
model.add(Dense(15, activation=LeakyReLU(alpha=0.1) ,kernel_initializer = 'he_uniform'))
# Defining Sixth Hidden Layer
model.add(Dense(15, activation=LeakyReLU(alpha=0.1) ,kernel_initializer = 'he_uniform'))
# Defining Output Layer
model.add(Dense(1,activation = 'linear',kernel_initializer = 'glorot_uniform'))
# Compiling the Model with Given Architecture
model.compile(loss='mean_absolute_error', metrics=['mean_absolute_error'], optimizer='adam') # compile the model
return model
model = KerasRegressor(build_fn=create_model, verbose=1)# Define the hyperparameters grid to be validated
batch_size = [10, 20, 30, 40, 60, 80,100]
epochs = [10,50,75,100]
param_grid = dict(batch_size=batch_size, nb_epoch=epochs)
model = KerasRegressor(build_fn=create_model, verbose=1)
grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=5, n_jobs=-1)# Run the GridSearchCV process
grid_result = grid.fit(X_train1,y_train,verbose = 2)# Print the results of the best model
print('Best params: ' + str(grid_result.best_params_))
Predicting Result on Train Data
y_hat=grid.predict(X_train1)
r2_score(y_train,y_hat)
Predicting Result on Test Data
y_hat_test=grid.predict(X_test1)
r2_score(y_test,y_hat_test)
R2 score can be further increased by trying out different tuning techniques as described above.
My tryst with use of ML & DL will continue i will be further exploring concepts of LSTM , Time Series Forecasting , Graph Neural Networks on this Engine dataset.
Complete Code can be found out at my Github account.
Available to collaborate/learn/feedback through my LinkedIn profile.
Thanks. Happy Learning
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a small portion of your membership fee if you use the following link, with no extra cost to you.