Model Evaluation Metrics — KS Score

What is KS Score? How’s it computed and used?

Photo by Pietro Jeng on Unsplash

What are evaluation metrics & why do we need to evaluate a model?

• Evaluation metrics are those which are used to quantify the performance of an ML model.
• Evaluating a model allows us to assess the quality of the predictions and determine if the model is generalizing well. Evaluation metrics serves the model after it has been trained by providing a score.
• Model evaluation helps to identify problems with the model and the data. It provides an opportunity to optimize the model, for example, by tuning hyperparameters or selecting different features.
• Finally, quantifying the model performance will build trust in the model and the stakeholders who rely on the model for decision-making will be able to get a measure of the model’s predictive power.

What is KS Score?

• Kolmogorov-Smirnov (KS) score is a way of comparing the cumulative sum of the positive and negative classes. It measures the maximum difference between the two over the range of predicted probabilities.
• The population of the entire dataset is divided into deciles based on the prediction probability. Each decile is generally known as a Tier.
• A high KS score indicates that the model has a better separation between the positive (goods) and negative classes (bads) at the respective tier. This metric is used widely in the finance and credit risk industry.
• KS Score provides a single scalar value that is easy to interpret and compare across different models.

Ideally, the max separation should be present in the riskiest 3 tiers (Top 3 tiers if DV = 0 is bad or bottom 3 tiers if DV = 1 is bad).

Computing KS Score

class ModelEvaluation:
    
    """
    A class to compute the gains table for a given predictions DataFrame.

    Attributes
    ----------
    predictions : pd.DataFrame
        A DataFrame containing 'PD', 'DV' columns.


    Methods
    -------
    compute_gains_table_ks():
        Computes the gains table from the provided predictions DataFrame.
        Returns the gains table & KS value from the gains table.
        
    """
    
    def __init__(self, predictions_df):
        self.predictions = predictions_df

    def compute_gains_table_ks(self):
        
        self.predictions['Bins'] = pd.qcut(self.predictions['PD'],q = 10)
        self.predictions['Total'] = 1
        self.predictions.rename(columns={'DV':'Bads'},inplace=True)
        
        gains_table = self.predictions[['Bins', 'Bads', 'Total']].groupby(self.predictions['Bins']).sum()

        gains_table['Goods'] = gains_table['Total'] - gains_table['Bads']
        gains_table['Cumulative_bads'] = gains_table['Bads'].cumsum()
        gains_table['Cumulative_goods'] = gains_table['Goods'].cumsum()
        gains_table['Cumulative_Bad_Rate'] = (gains_table['Cumulative_bads'] / gains_table['Cumulative_bads'].max() * 100).round(2)
        gains_table['Cumulative_Good_Rate'] = (gains_table['Cumulative_goods'] / gains_table['Cumulative_goods'].max() * 100).round(2)
        gains_table['KS'] = abs(gains_table["Cumulative_Bad_Rate"] - gains_table["Cumulative_Good_Rate"])
        gains_table = gains_table[['Bads', 'Goods', 'Total', 'Cumulative_bads', 'Cumulative_goods','Cumulative_Bad_Rate', 'Cumulative_Good_Rate', 'KS']]
        model_ks = gains_table['KS'].max()
        
       
        def identify_max(series):
            max_value = series.max()
            return ['<---' if i == max_value else '' for i in series]

        def highlight_max(series):
            max_value = series.max()
            return ['background-color: aquamarine' if i == max_value else '' for i in series]

        # Apply the identify_max & highlight_max functions to the DataFrame
        gains_table['Max_KS'] = identify_max(gains_table['KS'])
        gains_table = gains_table.style.apply(highlight_max,subset=['KS'])
        
        return gains_table, model_ks

Sample predictions data frame (where Label 0 — Class Good & Label 1 — Class Bad )

model_score = ModelEvaluation(predictions_df)
gains_table,ks = model_score.compute_gains_table_ks()

Gains Table

We can see that the KS Score for this dataset based on the predictions obtained from the ML model is 33.69.

Conclusion

To summarise, the KS Score serves as a powerful tool for quantifying the discriminatory power of predictive models, particularly in the realm of binary classification problems. KS Score is very helpful in identifying the separation between goods and bads at each tier defined by the probability ranges.

As a data scientist, it is crucial to understand and effectively utilize various model evaluation metrics, including the KS Score, to ensure that the models are not only accurate but also robust and reliable in practice. By incorporating the KS Score into one's evaluation toolkit, we can make more informed decisions about model selection, refinement, and deployment, ultimately driving better outcomes for the organization and stakeholders.

Thanks for reading the blog. If there are any questions or if you want to share any valuable feedback, feel free to reach out to me on Linkedin.