Scene-15 Image classification problem with Transfer Learning

Image classification is a process to classify the image within different classes. Image classification is one of the most important applications of deep learning. Transfer Learning is a highly efficient technique for image classification problems. Basically, in the transfer learning, we have to use “ImageNet” dataset weights to improve the prediction accuracy. To prove my points here I use the scene-15 dataset. You can found this dataset, here.

Scene-15

Some pictures from the scene-15 dataset

The Scene-15 dataset contains different pictures with 15 different classes as shown above. A total of 4485 images are available in 15 different classes among which we are going to use 100 images for training and the rest are for the tests. We have 1500 images for training and 2985 images for testing.

Let’s first import the libraries.

# importing libraries
import pandas as pd
import numpy as np
import os
import cv2
import random
import keras
import matplotlib.pyplot as plt
from scipy import misc
from plot_keras_history import plot_history
from keras import Model
from keras.applications.resnet50 import preprocess_input
from keras.applications import ResNet152V2
from keras.preprocessing.image import img_to_array, load_img
from keras.layers import ZeroPadding2D
from keras.models import Sequential
from keras.layers import Dense, Conv2D, MaxPooling2D,Dropout, Flatten, BatchNormalization, Concatenate, AveragePooling2D
from keras.callbacks import EarlyStopping, LearningRateScheduler

Splitting the data

It is important to split the data in training and testing. But here we have 15 different folders available so first try to split the data into the train/test folder. The following code will help to split the function into the Train and Test folder at the specified location.

# function to split images into train&test
def create_train_test(data_dir, class_folder):
    directory = data_dir + "/" + class_folder + "/"
    data = os.listdir(directory)
    random.shuffle(data)
    train = data[:100]  # random 100 images (random.shuffle) for train
    test = data[100:]  # rest for test
return train, test
# Getting available features
class_folder = []
path = "C:/Users/Cobra/Python Learning/Lakehead Assignments/Assignent_2_(part-1)/15-Scene/"
for category in sorted(os.listdir(path)):
    class_folder.append(category)
# splitting images into train & test
# Remember we just split the image into train & test (without preprocessing)
train_set = []
test_set = []
data_dir = 'give data dir path'
for class_name in class_folder:
 train, test = create_train_test(data_dir, class_name)
 train_data = [class_name, train]
 test_data = [class_name, test]
 train_set.append(train_data)
 test_set.append(test_data)
train_set = np.array(train_set)
test_set = np.array(test_set)

After creating the train & test images data we have to store them at a different place for future use.

# Create directory to store train and test folder
dir_create = “give dir path”
output_directory = os.mkdir(os.path.join(dir_create, ‘15-Scene_output’))

Now, let’s make the function to write the images to the folder which we created above.

# Function to create train and test folder in given directory
# data_dir = Original dataset path
# output_path = Path where we want to create folder
# output_directory = output directory name
# folder_name = folder name like. Train/Test
# dataset = train_set/test_set
def write_img_to_folder(data_dir, output_path, output_directory, folder_name,
 dataset):
new_folder_path = os.path.join(output_directory, folder_name)
 make_directory = os.mkdir(new_folder_path)
for data in dataset:
 class_name = data[0]
 data_files = data[1]
 data_path = os.path.join(
 output_path + ‘/’ + str(output_directory) + ‘/’ + str(folder_name),
 class_name)
 category_dire = os.mkdir(data_path)
 for filename in data_files:
 img = cv2.imread(data_dir + ‘/’ + class_name + ‘/’ + filename, 0)
 cv2.imwrite(os.path.join(data_path, filename), img)
output_path = os.getcwd()
# Train/Test folder using write_img_to_folder function
train_folder = write_img_to_folder(data_dir, output_path,'Train', train_set)
test_folder = write_img_to_folder(data_dir, output_path,'Test', test_set)

Load data

Now, our dataset is ready to use. We need to load the images from the stored path and convert those into an array to perform image classification. So, let’s get started with converting images into an array.

# Function to load image and convert into array
def convert_img_to_array(dir_path):
    X = []
    y = []
    one_hot_lookups = np.eye(15)
    for category in sorted(os.listdir(dir_path)):
        for file in os.listdir(dir_path + category):
            img = load_img(dir_path + category + '/' + file,
                       target_size=(224, 224))  # Load image
            img = img_to_array(img)  # Convert image to array
            img = preprocess_input(img)  # Preprocessing on the image
            X.append(img)  # Append image to X_train
            y.append(np.reshape(one_hot_lookups[int(category)],
                                  [15]))  # Append category to y_train
    return X,y

After creating the function of loading images from a given path and converting it into an array we need to generate our x_train, y_train and x_test,y_test data.

def load_data():
    train_dir_path = "Give train dir path"
    X_train,y_train = convert_img_to_array(train_dir_path)
    
    test_dir_path = "Give test dir path"
    X_test, y_test = convert_img_to_array(test_dir_path)
    
    X_train = np.asarray(X_train)
    X_test = np.asarray(X_test)
    y_train = np.asarray(y_train)
    y_test = np.asarray(y_test)
    
    X_train /= np.std(X_train, axis = 0)
    X_test /= np.std(X_test, axis = 0)
    
    return ((X_train, y_train), (X_test, y_test))
(x_train, y_train), (x_test, y_test) = load_data()

Creating the transfer learning model

Before generating the model, there are few things that need to be clear. The first one is what is transfer learning, in simple words transfer learning is a technique that uses the weights of other pre-trained models in order to get more accuracy. Here, we are using ‘ImageNet” pre-trained weights and this dataset has 1000 classes. And second thing is, we need to add some extra layers in order to get the required output. We have only 15 classes so, to settle down the final node we are adding some extra layers to the model. Here, we are performing transfer learning with the ResNet152V2 pre-trained model. Let’s begin the procedure.

def transfer_model(weights, input_shape, num_output):
    transfer_model = ResNet152V2(weights=weights, include_top=False,                              input_shape=input_shape,pooling='max',classifier_activation='softmax')
    inp = transfer_model.input
new_classification_layer = Dense(num_output, activation='softmax') # make a new softmax layer with 257 neurons
out = new_classification_layer(transfer_model.layers[-1].output) # connect our new layer to the second to last layer in ResNet152V2, and make a reference to it
final_model = keras.Model(inp, out) # create a new network between inp and out
    
    return final_model
transfer_model = transfer_model(weights='imagenet',input_shape=(224,224,3), num_output=15)
transfer_model.summary()

From the above code, we will get the below output.

Note: This is only the end of the model with 15 output nodes.

Compilation of model

After creating the model, it needs to compile before fitting the dataset. So, let’s first compile the model.

def compile_model(model):
    lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=0.01, decay_steps=1000, decay_rate=0.9)
# Optimizer for the model
    optimizer = keras.optimizers.SGD(learning_rate=lr_schedule)
# Creating the log file
    model.compile(loss='categorical_crossentropy',
              optimizer=optimizer,
              metrics=['accuracy'])
    return model
final_model = compile_model(final_model)

Fitting the data

After compiling the model, we need to fit the model with train and test data.

accuracy = []
losses = []
for i in range(3):
    keras.backend.clear_session()
    
    model = transfer_model
    
    model = compile_model(model)
    model_history = train_model(model, x_train, y_train, x_test, y_test)
    loss, acc = model.evaluate(x_test, y_test, verbose=1)
    
    plot_history(model_history.history)
    plt.show()
    
    print(f"\n")
    print(f"------------------------------------------------------")
    print(f"\n")
    accuracy.append(acc)
    losses.append(loss)

Here, I’ve run the entire model 3 times in order to achieve the best average accuracy. By implementing this code I’m able to achieve 89.53% accuracy.

Thank you.

Feel free to reach me at jivaniutsav007@gmail.com.