The Critical Role of Explainable Data in AI: Why Interpretable Models Aren’t Enough
In the rapidly evolving field of artificial intelligence, much attention has been given to explainable AI and interpretable machine learning models. However, a crucial aspect often overlooked is the explainability of the data itself. This article explores why explainable AI without explainable data is meaningless and how this concept is particularly vital for startups and knowledge discovery activities.
What is Data Explainability?
Data explainability is the ability to understand and justify why specific data should be used in training an AI model. It involves answering three key questions:
- To which function does this data sample belong?
- What data samples can replace this data block while maintaining performance?
- What is the smallest set that represents all of this data?
Understanding these aspects is crucial for building reliable and trustworthy AI systems.
Predictions Interpretability vs. Data Explainability
While predictions interpretability focuses on understanding model outcomes, data explainability delves into the nature of the input data itself. Let’s explore the distinction:
Predictions Interpretability:
- Focuses on explaining model outputs
- Utilizes tools like PCP, ALE, LIME, or SHAP
- Limited to the specific model being interpreted
Data Explainability:
- Examines the quality and relevance of input data
- Ensures data validity for model training
- Impacts overall model generalizability and performance
What would Explainable AI without Explainable Data look like?
Data explainability is the research topic of explainable data, while predictive interpretability is the research topic of interpretable machine learning. Although many tools have been developed to help form predictive interpretation capabilities for AI models, they all have one thing in common: they are limited to the model they serve, regardless of whether they are PCP, ALE, LIME, or SHAP. On the other hand, the validity of an AI model depends on the training data. Let’s consider the following example to clarify this statement.
The Pitfall of Unexplainable Data: A Simulation Study
The source code can be found in this Github repro.
To illustrate the importance of data explainability and the limitations of model interpretability alone, let’s consider a simulated example:
One day, a data scientist conducts data analysis of the data produced by two sinusoidal generators. The input data of these two machines has the same format of seven independent random variables distributed according to a uniform probability distribution U(0, 2).
Uncensored Data
Function of the Generator I
Y = sin((X1+..+X5)*1e-3)*1e3
Function of the Generator II
Y = sin((X3+..+X7)*1e-3+π)*1e3
Visible Data (Format of Training Data)
The data scientist proceeds to train an AI model using the visible data.
Result
Before training a particular model, let’s predict the result of the training together. Because the data column used to distinguish the generators (Generator_Id column) is missing, the result of training will be the same as predicting for a single generator. Thus, the natural of the result will be overfitting based on the input data with very high error in the test set. Therefore, any interpretation based on such a model is invalid!
In this experiment, the results mirrored the prediction. The difference between the training loss (2.7054) and the validation loss (5.5929) represents the overfitting of the model.
In other similar experiments, the results are not always so obvious. Interested readers can see more in this discussion.
Validity of the experiment
Although this is a simulated scenario, it reflects real-world challenges in data science and AI development. In practice, missing or unexplained variables can similarly impact model performance and interpretability, especially when dealing with complex, multi-dimensional datasets. This raises questions about the generalizability of the AI model to different types of data.
The Three Pillars of Explainable Data
While there’s no universally agreed-upon definition of explainable data, we can rely on the above definition of data explainability to solve the problem of AI model validity. Accordingly, Explainable Data will carry three key properties:
- Referentiality: Data points should be able to reference each other, indicating they belong to a larger entity.
For example: In the experiment, data points generated by the generator I can refer to each other through the distance between them on the generator I’s generating function. Accordingly, each data point in the training set will belong to one of two data clusters representing generators I and II. - Generality: Focus on explaining data generation rules rather than overfitting to specific data points.
For example: Find the smallest set that represents a data set. - Validity: Ensuring that explanations based on the data set at hand will still be valid with respect to the test data.
For example: Clustering the experimental data set into two parts corresponds to each generator so that every data point in the test set can be completely interpreted based on the training results.
By embedding the coherence sampling algorithm Nguyen (2024) in the training process, the original data can be divided into two clusters as shown below:
To see clusters, let’s look at the labeled-version of the chart based on the Generator_Id column of the uncensored dataset:
It should be noted that this article will only provide enough information for readers to get an easy-to-understand qualitative view of the data without delving into effective clustering algorithms to address related questions. Accordingly, with the existence of such a clustering algorithm, the problem of validity error of the AI model mentioned above would be resolved. Furthermore, this approach will help us avoid the pitfall of relying too heavily on model interpretability tools without critically examining the underlying data. True explainable AI requires both interpretable models and explainable data working in tandem.
For readers unfamiliar with Cohesion-Convergence Groups (CCGs): CCGs are clusters of data points in a dataset that exhibit synchronized behavior during the optimization of a neural network using a training set belonging to that dataset. The loss function values corresponding to data points in these clusters tend to move in sync with each other as the neural network optimizes toward the minimum value of its objective function, providing information about the feeding data and the dynamics of the neural network.
In fact, during optimization, this behavior is exhibited long before the neural network fully converges, as reflected in the output of coherence sampling algorithms Nguyen (2024). This makes CGGs a suitable tool for studying explainable data.
Conclusion
Explainable data is an important category of explainable AI. The abundance of data today has unintentionally overshadowed its importance in data science projects driven by big data. However, not every project can be lucky enough to avoid being forced to pay attention to explainable data. This is especially true for entrepreneurship and knowledge discovery activities where startups and organizations engaged in knowledge discovery often face unique challenges:
- Limited historical data: Unlike established companies, startups rarely have data maturity based on decades of traditional data analytics.
- Continuous data mining: They must constantly seek new data properties to enhance their databases.
- Addressing missing data: Startups often grapple with incomplete datasets, making explainable data even more crucial.
These factors will make explainable data poised to become a critical focus in AI development, particularly for innovative and agile organizations. As we advance in AI development, it’s clear that explainable models alone are not sufficient. The quality, relevance, and explainability of our data are equally, if not more, important. By focusing on data explainability, we can build more robust, reliable, and trustworthy AI systems that truly serve their intended purposes.
References
Thien An L. Nguyen. Evidence, definitions and algorithms regarding the existence of cohesive-convergence groups in neural network optimization, 2024. URL https://arxiv.org/abs/2403.05610.
