What is ROC AUC and how to visualize it in python

Reciever Operating Characteristic or ROC curve is often utilised as a visualisation plot to measure the performance of a binary classifier. It’s not a metric of the model, per se, rather the graphical representation of the True Positive Rate (TPR) versus False Positive rate (FPR) at various classification threshold from 0 to 1. It can also be thought of as the power of Type I Error (False Positives) decision rule function, where:

The power of a binary hypothesis test is the probability that the test rejects the null hypothesis (H0) when a specific alternative hypothesis (H1) is true.

Wikipedia

First import the libraries

import pandas as pd
import numpy as np
from sklearn.metrics import confusion_matrix
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

We’ll use built in scikit-learn breast cancer dataset and basic logistic regression is sufficient for this case

data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, stratify=y)
lr = LogisticRegression()
lr.fit(X_train, y_train)
preds = lr.predict(X_test)

The confusion matrix looks like this

pd.DataFrame(confusion_matrix(y_test, preds), columns=['Predicted Benign', "Predicted Malignant"], index=['Actual Benign', 'Actual Malignant'])

Any binary classifier classifies the results in terms of positive aka 1 or negative aka 0 class, hence the name. So the results that the classifier predicts to be positive and are actually positive are the True Positives, the results that were falsely classified as positive but are in fact negative are False Positives. True Negatives, yeah, you guessed it, are actual negative class predicted as negative and False Negatives are the falsely predicted negative but are in fact positive

So in case with the breast cancer dataset:

tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()
print(f'True Positives: {tp}')
print(f'False Positives: {fp}')
print(f'True Negatives: {tn}')
print(f'False Negatives: {fn}')

TPR (aka Recall aka Sensitivity) measures the proportion of the actual positives that are correctly identified.

False Positive Rate measure the ratio between False Positives and the total number of actual negatives regardless of classification.

Since *.predict_proba* estimates probabilities for negative and positive classes, we need to select the probabilities for the positive class only

probas = lr.predict_proba(X_test)[:, 1]

We also need a function that will convert our predictions accroding to various threshold rates.

def get_preds(threshold, probabilities):
    return [1 if prob > threshold else 0 for prob in probabilities]

Next, we’ll create a lists of True Positive Rates and False Positive Rates at varying threshold rates

roc_values = []
for thresh in np.linspace(0, 1, 100):
    preds = get_preds(thresh, probas)
    tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()
    tpr = tp/(tp+fn)
    fpr = fp/(fp+tn)
    roc_values.append([tpr, fpr])
tpr_values, fpr_values = zip(*roc_values)

Finally, we can plot our results:

fig, ax = plt.subplots(figsize=(10,7))
ax.plot(fpr_values, tpr_values)
ax.plot(np.linspace(0, 1, 100),
         np.linspace(0, 1, 100),
         label='baseline',
         linestyle='--')
plt.title('Receiver Operating Characteristic Curve', fontsize=18)
plt.ylabel('TPR', fontsize=16)
plt.xlabel('FPR', fontsize=16)
plt.legend(fontsize=12);

This ROC visualization plot should aid at understanding the trade-off between the rates. We can also qunatify area under the curve also know as AUC using scikit-learn’s roc_auc_score metric, in order to assess the performance of the model. The closer the score to 1 the better the model distinguishes the clasess, if it’s closer to 0.5 then your model performs just as badly as the coin flip.

roc_auc_score(y_test, preds)

As you can see our basic Logistic Regression is not that bad. You can also use ROC AUC to compare different models or the same models with different parameters. I hope this will help you in visualising ROC curve and in your projects. Please give your react below, I’d be happy to hear your thoughts on this as well, so comment down below. Thank you for reading!