Best Model in PyTorch after training across all Folds
In this article I, am going to define one function which will help the community to save the best model after training a model across all the folds of a dataset .
For this, first we will partition our dataframe into a number of folds of our choice .
from sklearn import model_selection
dataframe["kfold"] = -1 # defining a new column in our dataset # taking a fraction of data from our dataframe and reset its index
dataframe = dataframe.sample(frac=1).reset_index(drop=True)# storing the target values in a list
y = dataframe.label.values# defining the number of required folds we want in our dataset
kf = model_selection.StratifiedKFold(n_splits= number_of_folds)# assigning the fold number with respect to the data in kfold column
for f, (t_, v_) in enumerate(kf.split(X=dataframe, y=y)):
dataframe.loc[v_, 'kfold'] = f
# looking into the new dataframe we created
dataframe
Now I am going to define the function which will return the best model after training across all folds .
from cpython.Lib import copya_string = "*" * 20def _run():
for i in range(number_of_folds): # just a print statement for fomatting
print(a_string, " FOLD NUMBER ", i, a_string)
# assigning fold numbers for training dataframe
df_train = new_train[new_train.kfold != i].reset_index(drop=True) # assigning fold numbers for validation dataframe
df_valid = new_train[new_train.kfold == i].reset_index(drop=True)
# defining an empty list to store all accuracies across the particular fold we are training
all_accuracies = []
for epoch in range(NO_OF_EPOCHS):
print(f"Epoch --> {epoch+1} / {NO_OF_EPOCHS}")
print(f"-------------------------------")
train_loss, train_accuracy_metric_output = train_loop_fn()
print('training Loss: {:.4f} & training accuracy : {:.2f}%'.format(train_loss, train_accuracy_metric_output*100))
valid_loss , validation_accuracy_metric_output = eval_loop_fn(val_dataloader, model, device)
print('validation Loss: {:.4f} & validation accuracy : {:.2f}%'.format(valid_loss , validation_accuracy_metric_output*100))
all_accuracies.append(valid_acc)
# comparing and saving the best model in best_model variable at each training and validation checkpoint
if i < 1:
best_accuracy = max(all_accuracies)
best_model = copy.deepcopy(model)
else:
if best_accuracy > max(all_accuracies):
continue
else:
best_accuracy = max(all_accuracies)
best_model = copy.deepcopy(model)
print('\n============Saving the best model==================')
torch.save(best_model.state_dict(),'__your best model__.pt')
print() # Printing out the best accuracy we got after training
print("& The highest accuracy we got among all the folds is {:.2f}%".format(best_accuracy*100)) return best_model
As there are lot of lines present in the above code , so I am going to explain them one by one and I also urge to read the commented lines present above the codes, in order to understand it properly.
In the first line I am importing the copy function from cpython library.
Then defining a string line just for formatting, that nothing much important.
Then defining a loop so that we can iterate on each fold one by one.
After that, we are using formatting variable to show the results obtained on training the model across each fold in an iterative manner.
Then defining df_train & df_valid which will consider the fold number only for df_valid and rest other folds will be assigned to df_train.
After this, we are defining an empty list to store all the accuracy values that we will obtain after training the model inside a specific fold number for validation data.
After that we are defining the loop to train our model for a number of epochs and using some formatting print lines to track the training and validation losses as well as metric and printing them out in the next couple of lines.
all_accuracies.append(valid_acc) this line is used for storing all the accuracies obtained within the fold training in the list declared above.
Next few lines are the most important lines where the actual magic is happening so pay close attention to this. So, we are checking here that if the fold on which we are training our model currently is less than 1 then we are storing the best accuracy and best model in two variables best_accuracy & best_model consecutively. Otherwise if the fold number is having value more that or equal to 1 then we are comparing its best accuracy with than the max accuracy we obtained in the last fold’s iteration . If current best accuracy is less than the previous accuracy we got then we are simply continuing else we are replacing the it with the values inside our required variables.
I hope that you have understood this pytorch coding well. If you still have any question, comments or concerns then don’t be shy to put it out in the comment section. I will definitely answer it. And until then enjoy learning.
