Decoding closed box Models with SHAP

9 min readJun 20, 2024

In the previous article (XAI), we delved into the world of explainable AI (XAI) and its growing importance in making machine learning models more transparent and interpretable. Among the various XAI techniques, SHAP (SHapley Additive exPlanations) has established itself as one of the most powerful and widely adopted methods. In this deep dive, we’ll explore the technical underpinnings of SHAP, its real-world applications, and the substantial business benefits it brings to the table.

So, what is SHAP?

At its core, SHAP leverages the concept of Shapley values, a game-theoretic approach to ensure fair distribution of payouts among players. In the context of machine learning, these “players” are features within a model, and the “payout” is the prediction output. SHAP values measure the impact and contribution of each feature to a prediction, providing a granular understanding of how each feature influences the model’s output.

https://www.britannica.com/science/game-theory

SHAP employs game theory similarly to this matrix, where strategies (support, oppose, evade) determine outcomes. In SHAP, features are like strategies, and their contributions to the prediction are fairly distributed, ensuring each feature’s impact is clearly and equitably evaluated, much like payoffs in the matrix.

How SHAP Works

Feature Permutations: SHAP evaluates all possible combinations of features to determine the effect on the prediction when a feature is included or excluded.
Contribution Calculation: It calculates the average contribution of each feature across all permutations.
Feature Attribution: SHAP assigns a value to each feature, indicating its impact on the prediction. Positive values increase the prediction, while negative values decrease it.

SHAP Explainer Methods

SHAP presents itself in several explainer methods tailored to different types of models and data, so each of them related to the type of model you are using for Machine learning:

Sampling Explainer

Purpose: This explainer is particularly useful when dealing with large datasets where computing exact Shapley values is computationally expensive.
How it works: It samples various subsets (or coalitions) of features and calculates their contributions to the prediction. By aggregating these contributions, it provides an estimate of the Shapley values.
Advantages: More scalable and faster for large datasets compared to exact methods.

Permutation Explainer

Purpose: It approximates Shapley values by iterating through permutations of features.
How it works: For each feature, it permutes the feature values and measures the change in the model’s prediction. This change helps in estimating the importance of the feature.
Advantages: Provides a straightforward way to estimate feature importance by looking at the impact of feature permutation.

Partition Explainer

Purpose: This explainer is designed for structured data and leverages the hierarchical nature of the data to compute Shapley values more efficiently.
How it works: It partitions the features into groups and computes the Shapley values by considering these groups. This hierarchical approach reduces the computational complexity.
Advantages: More efficient for structured data, especially when there are logical groupings of features.

Tree Explainer

Purpose: Specifically tailored for tree-based models like decision trees, random forests, and gradient boosted trees.
How it works: It exploits the structure of the tree models to compute exact Shapley values efficiently.
Advantages: Very efficient for tree-based models, often providing exact Shapley values without the need for approximations.

Gradient Explainer

Purpose: Designed for neural networks, this explainer uses gradients to compute the contribution of each input feature to the model’s output.
How it works: It calculates the gradients of the outputs with respect to the inputs, which indicate how changes in input features affect the predictions.
Advantages: Tailored for neural networks, providing insights into feature importance by analyzing gradient information.

UIB(Understand it better)

This visual explanation helps to clarify which features are most influential in the model’s decision-making process and how each one’s contribution shifts the prediction from a baseline of 0.1 to 0.4. Such insights are crucial for understanding, validating, and potentially improving the model.

https://shap.readthedocs.io/en/latest/index.html

Model Inputs and Output: The model takes inputs such as Age, Sex, Blood Pressure (BP), and Body Mass Index (BMI). The base model prediction (base rate) is 0.1, and the final output is 0.4.

The arrow shows the transformation from the base model output to the final prediction through the addition of SHAP values. Each feature’s contribution is quantified, illustrating how it drives the model’s decision from the baseline prediction to the final outcome.

Deep Dive into 2 Examples:

Let's see 2examples to understand SHAP in action in real world applications.

Tabular Example: Real Estate Pricing example using LightGBM

Flat predictions: Consider a typical scenario: a real estate investor uses a predictive model to determine the value of a property. Traditionally, the output might be a single figure — an estimated price. This number alone, without context, offers limited insight into which factors contributed to the assessment and how.

# Make predictions with the LightGBM model
predicted_prices = model.predict(X_test)

# Convert predictions to a DataFrame for easier comparison
predictions_df = pd.DataFrame({
    'True Prices': y_test.values,
    'Predicted Prices': predicted_prices
})
predictions_df.head()

Shap Values: Implementing SHAP changes this dynamic dramatically. It allows the investor to see precisely how different features, such as median income, house age, and geographical location, each contribute to the final valuation. SHAP reveals that a high median income in the area and a newer property age significantly increase the property’s predicted value, whereas the location’s specific characteristics might have a smaller, though still notable, effect. This granular insight not only fosters greater confidence among stakeholders but also empowers them with the ability to make more informed decisions. They can pinpoint investment opportunities based on features that SHAP identifies as value-enhancing, optimize property prices for the market, or even refine their business models to focus on properties with characteristics most aligned with profitable outcomes.

import shap

# Initialize a SHAP TreeExplainer with the model
explainer = shap.TreeExplainer(model)

# Calculate SHAP values for the test set
shap_values = explainer.shap_values(X_test)

# Visualize the SHAP values with a summary plot
shap.summary_plot(shap_values, X_test, feature_names=housing.feature_names)

Shap Force Plot: Implementing SHAP changes this dynamic significantly. It allows the investor to see clearly how different features, such as median income, latitude, and average occupancy, each contribute to the final valuation. SHAP reveals that a low median income in the area decreases the property’s predicted value, while factors like latitude and longitude also have a negative, though smaller, impact. This detailed insight not only builds confidence among stakeholders but also helps them make more informed decisions. They can identify investment opportunities based on features that SHAP highlights as important, adjust property prices for the market, or refine their strategies to focus on properties with characteristics that align with profitable outcomes.

import shap

# Generate a force plot for the first instance in the test dataset
force_plot = shap.force_plot(
    explainer.expected_value,      # The base value (average model output over the dataset)
    shap_values[0],                # SHAP values for the first prediction
    feature_names=housing.feature_names,  # Feature names
    matplotlib=True                # Optionally use matplotlib for static plots if you are not in a Jupyter environment
)

force_plot

Textual Example: Sentiment Prediction in customer review

Flat predictions: Consider a typical scenario: a product manager uses a predictive model to determine the sentiment of customer reviews. Traditionally, the output might be a single figure — an estimated sentiment score. This number alone, without context, offers limited insight into which factors contributed to the assessment and how.

# Load a more complex dataset (20 newsgroups dataset)
data = fetch_20newsgroups(subset='train', categories=['rec.sport.baseball', 'rec.sport.hockey'])
reviews = data.data
target = data.target  # 0 for 'rec.sport.baseball', 1 for 'rec.sport.hockey'

# Vectorize the text data
vectorizer = TfidfVectorizer(max_features=1000)  # Limit to 1000 features for simplicity
X = vectorizer.fit_transform(reviews)
y = target

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train the XGBoost model
model = xgb.XGBClassifier()
model.fit(X_train, y_train)

# Make predictions with the model
predicted_sentiments = model.predict(X_test)

# Convert predictions to a DataFrame for easier comparison
predictions_df = pd.DataFrame({
    'True Sentiments': y_test,
    'Predicted Sentiments': predicted_sentiments
})
print(predictions_df.head())

SHAP Values: The SHAP summary plot allows the user to see precisely how different features, such as specific words or phrases in the reviews, each contribute to the final sentiment score. SHAP reveals that certain words have a significant impact on the model’s predictions. For instance, words like “hockey” and “nhl” are strongly associated with one sentiment class, while words like “baseball” and “playoff” are associated with another.

# Generate a force plot for the first instance in the test dataset
shap.initjs()
force_plot = shap.force_plot(
    explainer.expected_value,      # The base value (average model output over the dataset)
    shap_values[0],                # SHAP values for the first prediction
    feature_names=vectorizer.get_feature_names_out()  # Feature names
)
force_plot

SHAP Force Plot: This enables the user to see clearly how different features, such as specific words or phrases in the reviews, each contribute to the final sentiment score. SHAP reveals that negative words in the reviews decrease the predicted sentiment score, while positive words have a positive impact. This detailed insight not only builds confidence among stakeholders but also helps them make more informed decisions. They can identify areas for improvement based on features that SHAP highlights as important, adjust product descriptions for better customer perception, or refine their customer service strategies to address common complaints.

force_plot = shap.force_plot(
    explainer.expected_value,
    shap_values[0],
    feature_names=vectorizer.get_feature_names_out(),
    matplotlib=True
)

Real-World Applications

SHAP explanations have found their way into various real-world scenarios, enhancing decision-making processes and fostering trust in AI systems:

Healthcare: SHAP helps doctors understand the key factors driving a model’s diagnosis or risk assessment, enabling more informed treatment decisions.
Finance: In credit lending models, SHAP values highlight the features that influence loan approval or rejection, ensuring fair and unbiased decisions.
Fraud Detection: SHAP pinpoints the attributes that trigger fraud alerts, allowing investigators to quickly validate the legitimacy of flagged transactions.
Customer Churn: By identifying the factors contributing to customer attrition, SHAP insights help businesses devise targeted retention strategies.

Business Benefits

The adoption of SHAP-powered explainability brings a host of business benefits:

Increased Trust and Transparency: SHAP explanations increase the transparency of AI systems, fostering trust among users and stakeholders by elucidating the decision-making process of models.
Bias Detection and Mitigation: SHAP reveals discriminatory feature attributions, helping identify and mitigate potential biases in models, which is essential for ethical AI practices.
Model Debugging and Improvement: By highlighting influential features, SHAP aids data scientists in refining models, leading to improved performance and more accurate predictions.
Regulatory Compliance: SHAP assists in meeting stringent regulations in sectors like finance and healthcare by providing transparent explanations of model decisions.
Business Insights and Strategy: Understanding feature impacts allows businesses to optimize strategies and replicate successful outcomes based on predictive insights.

Conclusion

SHAP has emerged as a game-changer in the realm of explainable AI, enabling organizations to unlock the black box of machine learning models. By providing clear and interpretable explanations for model predictions, SHAP not only enhances trust and transparency but also facilitates model debugging, bias mitigation, and regulatory compliance.

As AI continues to permeate various industries and decision-making processes, the importance of explainable AI techniques like SHAP cannot be overstated. By embracing SHAP and integrating it into their AI workflows, businesses can harness the full potential of their models while ensuring fairness, accountability, and transparency.

In the next article, we’ll explore another prominent XAI technique, LIME (Local Interpretable Model-agnostic Explanations), and compare its strengths and use cases with SHAP. Stay tuned for more insights into the exciting world of explainable AI!