Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
This blog post summarizes a paper on “Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making”. This paper will be presented at the 26th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW) and will be published in the journal Proceedings of the ACM (PACM).
Issue of AI Overreliance
Advanced artificial intelligence (AI) and machine learning (ML) models are increasingly being considered to increase efficiency and reduce the cost of performing decision-making tasks from various types of organizations and domains (e.g. health, bail decisions, child welfare services, etc.). However, users might place too much trust in the AI/ML system and even agree with ‘wrong’ AI outputs and they achieve worse performance than humans or AI/ML models alone.
What did we do?
In this work, we contribute to an empirical study that analyzes the effect of AI explanations on users’ trust and reliance on AI during clinical decision-making. Specifically, we focus on the task of assessing post-stroke survivors’ quality of motion. We conducted a within-subject experiment with seven therapists and ten laypersons to compare the effect of counterfactual explanations with one of the widely used AI explanations, feature importance explanations.
- Feature importance: describes the contribution/importance of each input feature (e.g. kinematic variables — joint angle, distance between joints for the context of the study)
- Counterfactual explanations: describe how the inputs can be modified to achieve an AI output in a certain way (e.g. how does a patient’s incorrect/abnormal motion need to be changed to become a normal motion?)
One potential reason for overreliance on AI might be that humans rarely involve analytical thinking on AI outputs. This work hypothesizes that reviewing counterfactual explanations will allow a user to critically think about how to change AI inputs to update an AI output and improve the user’s analytical review of an AI output to reduce overreliance on AI.
(b) the predicted assessment score by AI along with salient feature-based explanations that compare a post-
stroke survivor’s unaffected and affected side using the top three most important features and counterfactual,
what-if explanations that describe how input features need to be changed to flip an AI output (e.g. ‘correct’
to ‘incorrect’ ROM).
What did we learn?
- When ‘right’ AI outputs were presented, human+AI performance with both feature importance and counterfactual explanations increased than humans alone
- When ‘wrong’ AI outputs were presented, human+AI performance with both feature importance and counterfactual explanations decreased than humans alone
- Counterfactual explanations reduced overreliance on ‘wrong’ AI outputs by 21% compared to feature importance
- Domain experts (i.e. therapists) had lower performance degradation and overreliance on ‘wrong’ AI outputs than laypersons while using both feature importance and counterfactual explanations
- Both experts and laypersons expressed higher subjective usability scores of ‘usefulness’, ‘less effort & frustration’, ‘trust’, and ‘usage intent’ on feature importance than counterfactual explanations.
Features and Counterfactuals for (1) all cases, (2) cases with ‘right’ AI outputs, and (3) cases with ‘wrong’ AI
outputs. Although participants’ performances improved after reviewing ‘right’ AI outputs with both salient
feature analysis and counterfactual explanations, their performance reduced after reviewing ‘wrong’ AI
outputs. Counterfactual explanations assisted participants to have lower degraded performance than salient
feature analysis. *, **, and *** indicate 90%, 95%, and 99% statistical significance levels.
Implications: Overall, our work brings to light that providing AI explanations does not necessarily indicate improved human-AI collaborative decision-making. This work provides new insights into:
1) the potential of counterfactual explanations to improve analytical reviews on AI outputs and reduce overreliance on ‘wrong’ AI outputs with the cost of cognitive burdens.
2) a gap between users’ perceived benefits and actual trustworthiness/usefulness of an AI system (e.g. improving performance while relying on ‘right’ outcomes)
Please check our presentation at CSCW 2023 conference on in the Human AI Collaboration I session and our paper for the details of this work (link). If you are interested in further discussing this work or collaborating in this space, feel free to reach out to Min Lee (link).
Citation Format:
Min Hun Lee and Chong Jun Chew. 2023. Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making. Proc. ACM Hum.-Comput. Interact. 7, CSCW2, Article 369 (October 2023), 22 pages. https://doi.org/10.1145/3610218
