Generation of a concatenated Confusion Matrix in cross-validation

Photo by Petalz_R on Unsplash

After we pre-process the data and develop our model, then we need to evaluate the effectiveness of our model. Considering that, the Confusion Matrix an evaluation metric where we can assess the performance of our model in machine learning. However, when we are executing cross-validation (ex: Leave-one-out cross-validation, k-fold cross-validation) to generalize our dataset to independent data, we may need to generate an average accuracy or a confusion matrix. So, this blog mainly aims to generate a concatenated confusion matrix while using cross-validation.

IRIS dataset

We will use the IRIS dataset for our implementation. This is a simple dataset about three species of flower types as Iris Setosa, Iris Versicolour and Iris Virginica.

We will first load the dataset and investigate the properties.

import itertools

import matplotlib.pyplot as plt
import numpy as np
from sklearn import svm, datasets
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import KFold
# import IRIS dataset to play with
iris = datasets.load_iris()
data = iris.data
target = iris.target
class_names = iris.target_names
class_names
labels, counts = np.unique(target, return_counts=True)

We can retrieve the statistical information of the dataset by the following snippet.

print(iris.DESCR)

We will evaluate our model by K-fold cross-validation with 10 folds. The following code will split our dataset into training and test folds and will evaluate our model performance 10 times.

Trick !!!
In each fold, we will get the actual test labels (test_y) and predicted labels (predicted_labels) in each fold and append each of them to an array of actual_targets and predicted_targets respectively.

Here we will fit our dataset by a Support Vector Machine.

def evaluate_model(data_x, data_y):
    k_fold = KFold(10, shuffle=True, random_state=1)

    predicted_targets = np.array([])
    actual_targets = np.array([])

    for train_ix, test_ix in k_fold.split(data_x):
        train_x, train_y, test_x, test_y = data_x[train_ix], data_y[train_ix], data_x[test_ix], data_y[test_ix]

        # Fit the classifier
        classifier = svm.SVC().fit(train_x, train_y)

        # Predict the labels of the test set samples
        predicted_labels = classifier.predict(test_x)

        predicted_targets = np.append(predicted_targets, predicted_labels)
        actual_targets = np.append(actual_targets, test_y)

    return predicted_targets, actual_targets

In the following code, we will feed our generated actual_targets and predicted_targets arrays to the plot_confusion_matrix() function.

def plot_confusion_matrix(predicted_labels_list, y_test_list):
    cnf_matrix = confusion_matrix(y_test_list, predicted_labels_list)
    np.set_printoptions(precision=2)

    # Plot non-normalized confusion matrix
    plt.figure()
    generate_confusion_matrix(cnf_matrix, classes=class_names, title='Confusion matrix, without normalization')
    plt.show()

    # Plot normalized confusion matrix
    plt.figure()
    generate_confusion_matrix(cnf_matrix, classes=class_names, normalize=True, title='Normalized confusion matrix')
    plt.show()

Our following code is responsible for generating normalized or non-normalized confusion matrix.

def generate_confusion_matrix(cnf_matrix, classes, normalize=False, title='Confusion matrix'):
    if normalize:
        cnf_matrix = cnf_matrix.astype('float') / cnf_matrix.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    plt.imshow(cnf_matrix, interpolation='nearest', cmap=plt.get_cmap('Blues'))
    plt.title(title)
    plt.colorbar()

    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cnf_matrix.max() / 2.

    for i, j in itertools.product(range(cnf_matrix.shape[0]), range(cnf_matrix.shape[1])):
        plt.text(j, i, format(cnf_matrix[i, j], fmt), horizontalalignment="center",
                 color="white" if cnf_matrix[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

    return cnf_matrix

Code execution is as follows:

predicted_target, actual_target = evaluate_model(data, target)
plot_confusion_matrix(predicted_target, actual_target)

Our code will output the following concatenated confusion matrixes by considering the predictions for test data in each fold by cross-validation.

Confusion matrix without normalization

Confusion matrix with normalization

I hope this post provides you with some basic implementation of a concatenated confusion matrix.

If you believe it valuable, feel free to share this blog on any platform.

You can find my complete code here.

Happy Learning!!

You can reach me at:

LinkedIn: https://www.linkedin.com/in/rukshanisomarathna/

GitHub: https://github.com/Rukshani