Explainable AI Reloaded: Do we need to Rethink our XAI Expectations in the Era of Large Language Models like ChatGPT?

by Upol Ehsan (Georgia Tech, US) and Mark Riedl (Georgia Tech, US)

Provocation

With the advent of Foundation Models & Large Language Models like ChatGPT, is “opening the black box” still a reasonable and achievable goal for XAI? Do we need to shift our perspectives?

In February of 2023, Navid (pseudonym), a relative of the first author of this article, asked if ChatGPT could be used to do homework. Navid is a high-schooler who shared that he was always under a tight deadline and needed to be more “efficient” with his homework. He heard from his friends that ChatGPT can help summarize papers or books, which can make Navid more “productive” with his homework.

Fig 1: Screenshot from ChatGPT’s response to summarizing a paper showcasing its hallucinated or confabulated content

Before responding to Navid, ChatGPT was taken for a test drive. ChatGPT was prompted to summarize an academic paper similar to how Navid might prepare to write a summary or essay about it — as someone who wasn’t an AI researcher or experienced in prompting Large Language Models (LLMs). Fortunately, ChatGPT generated a coherent response. ChatGPT gave the names of the authors of the paper and details about the paper’s publication at ACM FAccT 2020. Unfortunately, the names of the authors and publication details were made up! The details were extremely plausible — the paper very well could have appeared at that conference, but did not. The first author of this paper almost missed another detail in ChatGPT’s summary. The original paper described a framework with two dimensions: social and technical. The generated summary claimed the framework described three dimensions: social, technical, and cultural, which, while wrong, was plausible enough that even the very author of the paper almost missed that crucial inaccuracy!

Separating Fact from Fiction

The true story above demonstrates the effortful process required to disentangle fact from fiction in GPT’s output, even from someone knowledgeable of the source material. Even more notably, there was no way for our protagonist, an expert in Explainable AI, to “open” the black box of ChatGPT and understand why it produced what it produced or where it might be faithful to the facts or prone to confabulation (also called hallucination). On the one hand, he lacked access to the internal details such as the parameters of the model. On the other hand, even if one did have access to the internal parameters of the model, given the scale and complexity of the neural architecture of such a large language model, interpreting it is unlikely to produce human-understandable and actionable information.

Tensions: XAI and Large Language Models

The field of Explainable AI (XAI) is concerned with developing techniques, concepts, and processes that can help stakeholders understand the reasons behind the AI system’s decision-making. Explanations can be used for developers to identify and correct the causes of flaws or inaccuracies in an AI model. More importantly, explanations can help (direct and indirect) stakeholders to calibrate their reliance on the AI’s decision so that one does not overrely on a particular AI output or behavior when the system produces an unreliable result.

When we consider a service such as ChatGPT, GPT-4, Bing Chat, Bard, LaMDA, or PaLM, what prospects are there for explainable AI? These models have hundreds of billions of parameters, all acting in conjunction to generate a distribution over possible words to choose from to build a response, word by word. If we had access to all the weights, could we interpret and explain the model? If we had access to the parameters of a model and the activation values for an input could we interpret and explain the model? In the case of the above large language models the point is moot. All these models run on servers behind APIs that do not allow inspection of the neuron activations and weights. However, even if we could access this information, the raw values of weights and activations are meaningless to most people without synthesizing some visualization or text summarization that provides a lay-understandable analysis of the internal operations of the system and how the results were generated by the system. Consider OpenAI’s work on interpreting the patterns that cause individual neurons to activate (Bills et al. 2023). How would knowing what causes neuron #2142 to activate have helped Navid, a non-AI expert, know how to better use ChatGPT to complete his homework? What actionable information from this neural activation pattern can a non-AI expert use meaningfully?

Is Explainable AI doomed to fail?

As Large Language Models (LLMs) become prominent, is Explainable AI — a research area in flux and its infancy — doomed to fail? We do not think so. There is hope.

Before we throw in the towel, there are a few things to consider.

AI systems are Human-AI assemblages

First, we need to appreciate the reality of AI systems. When we say “AI systems,” what we very often mean to say is “Human-AI assemblages,” where the “human” part of the Human-AI assemblage is often implicit. No real-world AI systems work in a vacuum. Black boxes by themselves do not do the work — humans with black boxes do the work. Even if the human contribution to the work is to just provide an input, this is a significant contribution because AI systems are useful to people as tools. Thus, the explainability of AI systems entails explainability of the Human-AI assemblage, which has at least two components: the human (or humans) and the AI. Thus, how can we achieve the explainability of the Human-AI assemblage by just focusing on the explainability of the AI model? XAI is therefore not just technical, it is sociotechnical. It requires more than just algorithmic transparency — more than being able to open the black box.

Fig 2: Illustrating how the explainability of the Human-AI assemblage is more than just technical (algorithmic) transparency

Second, AI is evolving. Compared to AI and machine learning systems even five years ago, the deep learning systems in the foundation model era, such as large language models, are much more complex, have orders of magnitude more parameters, and are running at unprecedented scales. Thus, like any other design material that is shaped and molded by humans, the very design material of AI itself has evolved (Yang et al. 2020, Dove et al. 2017, Ehsan et al. 2023). Our understanding and expectations of what it means for AI-as-design-material to be explainable should also evolve. Further, XAI techniques that focus solely on the algorithm or the model face a new challenge: it is getting increasingly hard to open the black box! As AI systems are increasingly end-user facing, those that need the explanations the most are on the other side of an AI or user interface. This is the case for the most popular Large Language Models and chatbots, and it is also the case for other types of consumer-facing systems. When the initial vision of XAI was articulated, a popular framing was to “open” the (proverbial) “black box” of AI (Castelvecchi 2016, Nott 2017, Ehsan 2022b), so that we could see inside of it, figure out what it was doing, why it was doing it, and if it was doing it correctly. With the advent of large language models, that ability to open the black box is increasingly limited due to the sheer complexity of the models and the increased prevalence of models behind restrictive APIs. And even if we did manage to “open” it, we will not understand what we see.

Enter Human-Centered Explainable AI

As algorithm and model transparency face new obstacles due to scale and the walled gardens of models being hosted behind restrictive APIs, a new front has emerged: Human-Centered Explainable AI (HCXAI) (Ehsan & Riedl, 2020). HCXAI is a holistic vision of AI explainability, one that is human-centered and sociotechnical in nature. It draws its conceptual DNA from critical AI studies and HCI including critical technical practice (Agre 1997), reflective design (Dourish et al. 2004), and value-sensitive design (Friedman et al. 2013). HCXAI encourages us to critically reflect and question dominant assumptions and practices of a field, such as algorithm-centered XAI. It also adopts a value-sensitive approach to both users and designers in the development of technology that challenges the status quo of a field. HCXAI encapsulates the philosophy that not everything that is important lies inside the black box of AI. Critical insights can lie outside it. Why? Because that’s where the humans are.

Thinking outside the black box of AI can help us meet our goals of helping people understand and calibrate their trust in AI systems. Even if we cannot meaningfully open the black box or interpret its complexities, there are a lot of things we can do to understand and explain the system around the black box. Increasing the aperture of XAI can help us focus on the most important part: who the human(s) is (are), what they are trying to achieve in seeking an explanation, and how to design XAI techniques that meet those needs. Indeed, explanations of the sociotechnical system can offer us an important affordance: actionability (Singh et al. 2021, Joshi et al. 2019, Ehsan et al. 2021b, Mansi & Riedl 2023).

At its core, actionability is about what a user can do with the information in an explanation. (Ehsan et al. 2021b). An actionable XAI system empowers the user by increasing the space of possible informed actions to achieve their end goals. This could be understanding how to change the inputs, contesting a decision, or learning when and how to use the system more appropriately. Actionability also addresses another important question: how do we know if an XAI system is useful? There are an increasing number of reports of XAI systems that are deployed and fail to have any measurable impact on their users (e.g., Alqaraawi et al. 2020, Stumpf et al. 2016). Many of these systems failed because the XAI systems were not designed with user needs in mind, such as by providing users with information they could already intuit themselves, by providing information that was onerous to verify, or by providing information that users could not use. In other words, the explanations generated by the systems were not actionable.

The way forward

With the reframing around human-AI assemblages and XAI systems that place the human as the central concern, and armed with actionability as the metric for success, we now lay out three possible paths forward. This list is not meant to be exhaustive or prescriptive. It is meant to be generative by providing emerging evidence for how Human-Centered XAI (HCXAI) can address our growing needs for understanding our increasingly AI-infused world.

Explainability outside the black box: Social Transparency

Most consequential AI systems are embedded in organizational environments where groups of humans interact with it. These real-world AI systems, as well as the explanations they produce, are socially-situated (Ehsan 2021a). Therefore, the socio-organizational context in which these systems are used is key. Why are we not incorporating socio-organizational contexts into how we think about explainability in AI? How can we tackle the explainability of Human-AI assemblages?

Enter Social Transparency (ST) a sociotechnically-informed perspective that incorporates the socio-organizational context into explaining AI-mediated decision-making (Ehsan et al. 2021a). Social transparency allows us to augment the explainability of a human-AI assemblage without necessarily changing anything about the AI model. Social transparency allows one to annotate an output or behavior from an AI system with who did what, when and why. These annotations are shared between others using the system. They allow users to see whether and why others have accepted or rejected an AI’s output. Social transparency does two important things: first, it challenges the dominant narrative of algorithm-centered notions of XAI; second, it expands our understanding of XAI beyond technical transparency by illustrating how adding social context can help people make better, more actionable decisions with AI systems.

Imagine the following scenario: Aziz is a software seller trying to use a powerful AI-based pricing tool to do something consequential: offer the right price to a client company. The AI suggests a price. Moreover, its suggestion has technical transparency — it explains its recommendation by showing Aziz the top features it considered, such as sales quota goals, comparative pricing with other clients, and costs. Confident with the AI’s recommendation, Aziz makes a bid, but the client finds the price too high and walks out.

Fig 3: Aziz, a software seller, using an AI-powered pricing recommender that also shares top 3 features it considered for that pricing recommendation

Despite an accurate AI model and the presence of technical transparency, why did the bid fail? There could be algorithmic reasons for it. But might also be relevant contextual factors outside the box that can help explain why the bid failed. Perhaps the history between Aziz and the client that wasn’t honored? Or maybe there were external events that happened since the model was trained, such as a pandemic-induced budgetary crisis.

Now imagine that Aziz could see that 65% of his peers rejected the AI’s pricing recommendation.

Fig 4: Adding historical agreement rates with AI from peers

Or, what if Aziz knew that Jess, a director in the company, sold the product at a loss due to pandemic-related budgetary cuts?

Fig 5: Illustrating the 4W — who did what, when, and why — encoding Social Transparency

This peripheral vision of who did what, when and why — called the 4W — are the constitutive design elements of Social Transparency that can encode relevant socio-organizational context.

The benefit of taking a holistic approach to explainability is clear: a study of real-world AI users in sales, cybersecurity, and healthcare found that social transparency, in the form of the 4W, helped people calibrate their trust in the AI’s performance, provide actionable information for AI contestability and robust decision-making, and the organizational context made visible enabled better collective actions in the organization and strengthened the human-AI assemblages (Ehsan et al. 2021a).

By incorporating the socio-organizational context, Social Transparency makes our understanding of XAI more holistic, representing the Human-AI assemblage more realistically than a purely algorithm-centered XAI view. We should note that Social Transparency is agnostic to whether an AI system is black-boxed or not. As long as there is an AI-based recommendation or decision, we can attach 4W — the socio-organizational context — to it. In a completely black-boxed AI system, there will not be any technical transparency. However, the 4W can add transparency to the social side of the Human-AI assemblage.

Explainability around the Edges of the Black Box: Rationale Generation & Scrutability

If the black box cannot be cracked open in any meaningful sense, there is another possibility: incorporate explainability around the edges of the black box to foster a better functional understanding in the user (Paez 2019) such that it fosters actionability. One of the original formulations of rationale generation (Ehsan et al. 2019) was that we didn’t need to know how a black box worked as long as we could learn how to give actionable advice about the black box by looking at its inputs and outputs. It was philosophically grounded in Fodor’s work on Language of Thought: how is it that, despite not having a 1–1 neural correlate of thought, humans can effectively communicate by translating their thoughts into words? For Human-AI interaction, even if the exact mechanism of the (artificial) neural correlate of AI’s thought was not known to the human, as long as actionable information is present in the explanation from an AI agent, the Human-AI interaction can proceed. In short, explanations that don’t directly access the model can still generate actionable information.

In the case of large language models, the actionable information is whether any particular input is likely to produce a reliable response that can be trusted. Large language models might be generally capable at many tasks such as question-answering, they are not infallible, and it is always possible for a user to ask a question that results in a confabulation (also called a “hallucination”) that the user is unable to vet. In this case, we can directly use the API to probe how it responds to particular stimuli (Xie et al. 2022). It is proposed that an XAI system can decompose the original, human authored question into a series of more fine-grained, related questions that provide more opportunities for the model to confabulate responses if it isn’t competent at the original question. These sub-questions can be selected to be easier for the user to vet. Generating questions to challenge an LLM has been demonstrated to increase users’ ability to determine whether the answer should be trusted or not.

Explainability by Leveraging Infrastructural Seams: Seamful XAI

No AI system is perfect. Mistakes are inevitable. Breakdowns in AI systems often occur when the assumptions we make in design and development do not hold true when they are deployed in the real-world. For example, an AI system can fail when it is trained on data from North America but deployed in South Asia, especially when the end user is unaware of this infrastructural mismatch. We call these mismatches between design assumptions and real-world usage the seams. Handling the mistakes from AI systems is hard, especially when the AI’s decision-making is hidden or black-boxed. Although black-boxing AI systems can make the user experience seamless and easy to use, concealing the seams can lead to downstream harms for end-users, such as uncritical AI acceptance. What can we do differently? How do we move beyond seamless AI? And what can we gain by doing so?

Seamful XAI is a design lens that incorporates the principles of seamful design (Chalmers et al. 2003) to augment explainability and user agency. A classic example of seamful design is a “seamful map” of WiFi coverage in your home (Chalmers et al. 2003). If you know the WiFi’s dead zones in your home, you will be able to best use it because you can then avoid. Without revealing the seams, users can have reasonable expectations of perfect WiFi. The map makes the seams in the WiFi’s infrastructure visible to users, which allows them to recalibrate their expectations and behavior. A seamful design principle asks us to leverage the weakness in opportunistic ways (Gaver et al. 2003).

Unlike seamlessness, seamful design doesn’t aim to hide the infrastructure. Rather, it puts the infrastructure and all its imperfections front and center. Seamful design helps us recognize and grapple with the complex infrastructures systems reside in. Conversely, seamless design ideals risks making the labor it takes to make the system work invisible (e.g., datawork, ghostwork, maintenance work). And, as invisible work is invariably unaccounted for and unappreciated, workers who conduct this work will feel undervalued or invisible. Seamfulness embraces the imperfect reality of spaces we inhabit and makes the most out of it.

In the context of AI, seams can be conceptualized as mismatches, gaps, or cracks in assumptions between the world of how AI systems are designed and the world of how AI systems are used in practice. Seamful XAI seeks to empower users with information that augments their agency by strategically identifying gaps between ideal design assumptions and reality.

At the heart of Seamful XAI are four observations:

1. Seams are inevitable, arising from the integration of heterogeneous sociotechnical components during technology deployments.
2. Seams are revealed through system breakdowns.
3. Instead of treating seams as problematic negatives to be erased, they can be used strategically to calibrate users’ reliance and understanding of an AI system.
4. The goal of this strategic revelation (and concealment) is to support user agency (actionability, contestability, and appropriation).

Let’s take a look at the Seamful XAI design process through an example of a study participant working through it.

Fig 6: An overview of the Seamful XAI design process with key questions relevant to each step

The first step of the process begins with generating “breakdowns.” Breakdowns are answers to the question, “what could go wrong when this technology gets deployed?” Answers could include technology failures, unfair treatment of groups, inducing over-reliance, or deskilling.

The second step is around anticipating and crafting seams, which is done in three parts. First (2A in the diagram), we ask: “what might we (as developers, designers, researchers, etc.) do to make the breakdown happen?” While this question might seem counter-intuitive, it allows us to systematically prevent breakdowns by understanding their causes. This step inverts the problem and makes it a goal directed task, which is important to generate concrete outcomes instead of open-ended problems. Next (2B), we try to anticipate the reasons for the breakdown (the seams) in the appropriate stage in the AI’s lifecycle (the colored boxes numbered 1–6 in Fig. 7). Finally (2C), we craft the seam by thinking about the gap between the ideal expectation and the reality of use.

Fig 7: A screenshot of the virtual whiteboard used for the seamful XAI design activity in the study, with zoomed-in examples. Area 1: Envisioning breakdown (Step 1). In the study, we provided sample breakdowns (A), which participants could either use directly or get inspiration for their own envisioning. Area 2: Anticipating & crafting seams (Step 2). We provided guiding prompts (B) for effectively crafting the seams. We also shared exemplary seams © for each stage of the AI lifecycle framework. Area 3: Designing with seams (Step 3). We asked participants to articulate their reasoning for choosing a seam and tag which user goals the selected seam (E) can support for augmenting user agency. For details see Ehsan et al. 2022.

The final step involves using the seams generated in step 2 in a way to empower user agency and explainability. Here (3A), we ask: given our end goal, which seams do we show and which do we hide (e.g. strategic revelation and concealment)? The revealed seams (3B) should empower users through better explainability. This step of the Seamful XAI process is a major differentiator from other Responsible AI processes: unlike most processes that stop at identifying gaps, this one goes beyond. It not only uncovers the gaps but also utilizes them as avenues to support users. For further details, check out our paper on Seamful XAI (Ehsan et al. 2022a).

In a co-designing study with 43 real-world AI practitioners and researchers, we found that a Seamful XAI approach is beneficial in three ways:

• It enhances explainability by helping stakeholders reveal the AI’s blind spots, highlight its fallibility, and showcase the strengths and weaknesses of the system. These outcomes can facilitate calibration of trust and reliance in the AI.
• It augments user agency by providing peripheral vision of the AI’s blind spots. Seamful information expands the action space of what users can do. Information in seams can convert “unknown unknowns” to “known unknowns,” which can empower users to know “where” to start an investigation.
• It is a resourceful way to not just reveal seams but also anticipate and mitigate harms from AI systems.

Seamful XAI promotes a shift towards a proactive design approach in (X)AI, which can mitigate the impact of harms from a “move fast and break things” philosophy endemic in AI design.

Takeaways

We began with the provocation: With the advent of Foundation Models & Large Language Models like ChatGPT, is “opening the black box” still a reasonable and achievable goal for XAI? Do we need to shift our perspectives?

We believe so.

The proverbial “black box” of AI has evolved, and so should our expectations on how to make it explainable. As the box becomes more opaque and harder to “open,” the human side of the Human-AI assemblage remains as a fruitful space to explore. In the most extreme case, the human side may be all there is left to explore. Even if we can open the black box, it is unclear what actionable outcomes would become available.

There are four important lessons coming out of our research on Human-centered XAI (HCXAI).

• First, the human-centered XAI perspective takes a pragmatic and resourceful view of explainability, especially if black boxes are expected to persist. By considering the actions afforded to the user by the explanations, HCXAI centers the focus on the human, ensuring AI augments human abilities rather than replace them.
• Second, explainability isn’t only achieved by looking inside the black box through mechanistic descriptions of how an algorithm works. Actionability can be achieved by exploring explainability outside and around the edges of the black box because human-centered XAI takes a more expansive view of what it means to provide insights into a black box that can afford a wider range of actions.
• Third, explicitly treating AI systems as human-AI assemblages means focusing on explainability of the assemblage, not just the AI. This widened perspective opens up avenues for not just factoring in who is interacting with the black box, but also how human teams can work together — directly or indirectly — to contextualize a dynamically changing real-world AI behavior.
• Fourth, seamful XAI turns the disadvantages and weaknesses of an AI system into advantages. The gaps between user expectations and AI capabilities are exactly the gaps that explanations address. Instead of hiding those gaps to create seamless experiences, seamful XAI leverages these gaps in an opportunistic manner to augment explainability and user agency.

As we reload our expectations on XAI, we invite you to do what HCXAI asks us to do: centering the design and evaluation around the human. This positioning can reveal unmet needs that must be addressed while avoiding the costly mistake of building XAI systems that do not make a difference. While there have been many examples of XAI systems that have failed to have the intended impact of users, it is often the case that these tenets of HCXAI were overlooked. XAI is a relatively young field of research that has yet to find its footing, even as the landscape of black box AI systems is rapidly evolving. It is not yet time to give up hope on XAI. Instead, we invite you to adopt critical reflection and value-sensitivity into XAI research and evaluation, making it human-centered.

Will Human-centered XAI solve all our problems? No, but it will help us ask the right questions.

Get Involved

• 📚 Check out this accessible primer on Human-centered Explainable AI (HCXAI).
• 💬 Join the growing HCXAI community by checking out the proceedings from the HCXAI workshop held in conjunction with CHI, the flagship conference in Human-Computer Interaction. This workshop is the flagship gathering for researchers, practitioners, and policy makers in HCXAI. Keynotes from leaders and research papers from 2021, 2022, and 2023 editions can be found at our website.
• 👩‍🔬 How does one’s AI background (or lack thereof) impact one’s interpretations of AI explanations? The Who in XAI (tweetorial here) has answers.
• 😱 Explanations don’t always lead to good outcomes. There can be dark patterns where users are deceived. But what happens when things go wrong despite best intentions from AI creators? The concept of Explainability Pitfalls can help (tweetorial here).
• 🤔 Given some XAI systems can lead to harm, is destroying it enough to address its harms? The Algorithmic Imprint can help you answer that question (tweetorial here)
• 👑 Do we need one definition of explainability to rule them all? Answer this question using Social Construction of XAI (tweetorial here).
• ⛰️ Every XAI system has a silent killer. It’s called the sociotechnical gap. How do you detect & diagnose it? Check out this framework on Charting the Sociotechnical Gap in XAI that helps you do that (tweetorial here).

About the Authors

Upol Ehsan is a Researcher and Doctoral Candidate in the School of Interactive Computing at Georgia Tech and an affiliate at the Data & Society Research Institute. Combining AI, HCI, and philosophy, his work in Explainable AI (XAI) and Responsible AI aims to foster a future where anyone, regardless of their background, can use AI-powered technology with dignity. His work has pioneered the area of Human-centered Explainable AI (a sub-field of XAI), receiving multiple awards at ACM CHI, FAccT, and HCII and been covered in major media outlets. By promoting equity and ethics in AI, he wants to ensure stakeholders who aren’t at the table do not end up on the menu. He graduated summa cum laude, Phi Beta Kappa from Washington & Lee University with dual-degrees in Philosophy (B.A.) and Engineering (B.S.) followed by a MS in Computer Science from Georgia Tech. Outside research, he is a founder and advisor for Aalor Asha, an educational institute he started for underprivileged children subjected to child labor. He is also a social entrepreneur and has co-founded DeshLabs, a social innovation lab focused on fostering grassroots innovations in emerging markets.

Mark Riedl is a Professor in the Georgia Tech School of Interactive Computing and Associate Director of the Georgia Tech Machine Learning Center. Dr. Riedl’s research focuses on human-centered artificial intelligence — the development of artificial intelligence and machine learning technologies that understand and interact with human users in more natural ways. Dr. Riedl’s recent work has focused on story understanding and generation, computational creativity, explainable AI, and teaching virtual agents to behave safely. His research is supported by the NSF, DARPA, ONR, the U.S. Army, U.S. Health and Human Services, Disney, Google, Meta, and Amazon. He is the recipient of a DARPA Young Faculty Award and an NSF CAREER Award.

References

• Agre, P. (1997). Toward a critical technical practice: Lessons learned in trying to reform AI in Bowker. G., Star, S., Turner, W., and Gasser, L., eds, Social Science, Technical Systems and Cooperative Work: Beyond the Great Divide, Erlbaum.
• Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020, March). Evaluating saliency map explanations for convolutional neural networks: a user study. In Proceedings of the 25th International Conference on Intelligent User Interfaces (pp. 275–285).
• Bills, S. Cammarata, N. Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike,, J., Wu, J., Saunders. W. (May 9, 2023). Language models can explain neurons in language models. OpenAI Research Blog.https://openai.com/research/language-models-can-explain-neurons-in-language-models
• Castelvecchi, D. (2016). Can we open the black box of AI?. Nature News, 538(7623), 20.
• Chalmers, M., MacColl, I., & Bell, M. (2003, September). Seamful design: Showing the seams in wearable computing. In 2003 IEE Eurowearable (pp. 11–16). IET.
• Dove, Graham, Kim Halskov, Jodi Forlizzi, and John Zimmerman. “UX design innovation: Challenges for working with machine learning as a design material.” In Proceedings of the 2017 chi conference on human factors in computing systems, pp. 278–288. 2017.
• Dourish, P., Finlay, J., Sengers, P., & Wright, P. (2004, April). Reflective HCI: Towards a critical technical practice. In CHI’04 extended abstracts on Human factors in computing systems (pp. 1727–1728).
• Ehsan, U., & Riedl, M. O. (2020). Human-centered explainable ai: Towards a reflective sociotechnical approach. In HCI International 2020-Late Breaking Papers: Multimodality and Intelligence: 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings 22 (pp. 449–466). Springer International Publishing.
• Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021a, May). Expanding explainability: Towards social transparency in ai systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–19).
• Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daume III, H. (2022a). Seamful XAI: Operationalizing Seamful Design in Explainable AI. arXiv preprint arXiv:2211.06753.
• Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I., Muller, M., & Riedl, M. O. (2021b). The who in explainable ai: How ai background shapes perceptions of ai explanations. arXiv preprint arXiv:2107.13509.
• Ehsan, U., Saha, K., De Choudhury, M., & Riedl, M. O. (2023). Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), 1–32.
• Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O. (2019, March). Automated rationale generation: a technique for explainable AI and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces (pp. 263–274).
• Ehsan, U., Wintersberger, P., Liao, Q. V., Watkins, E. A., Manger, C., Daumé III, H., … & Riedl, M. O. (2022b, April). Human-Centered Explainable AI (HCXAI): beyond opening the black-box of AI. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (pp. 1–7).
• Friedman, B., Kahn, P. H., Borning, A., & Huldtgren, A. (2013). Value sensitive design and information systems. Early engagement and new technologies: Opening up the laboratory, 55–95.
• Gaver, W. W., Beaver, J., & Benford, S. (2003, April). Ambiguity as a resource for design. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 233–240).
• Joshi, S., Koyejo, O., Vijitbenjaronk, W., Kim, B., & Ghosh, J. (2019). Towards realistic individual recourse and actionable explanations in black-box decision making systems. arXiv preprint arXiv:1907.09615.
• Mansi, G., & Riedl, M. (2023). Why Don’t You Do Something About It? Outlining Connections between AI Explanations and User Actions. arXiv preprint arXiv:2305.06297.
• Nott, G. (2017). Explainable artificial intelligence: Cracking open the black box of AI. Computer world, 4.
• Páez, A. (2019). The pragmatic turn in explainable artificial intelligence (XAI). Minds and Machines, 29(3), 441–459.
• Singh, R., Miller, T., Lyons, H., Sonenberg, L., Velloso, E., Vetere, F., … & Dourish, P. (2021). Directive explanations for actionable explainability in machine learning applications. ACM Transactions on Interactive Intelligent Systems.
• Stumpf, S., Bussone, A., & O’sullivan, D. (2016). Explanations considered harmful? user interactions with machine learning systems. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI).
• Xie, K., Wiegreffe, S., Riedl, M.O. (2022). Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes. In Proceedings of EMNLP 2022 Findings.
• Wolchover, N., & Reading, L. (2017). New theory cracks open the black box of deep learning. Quanta Magazine, 3, 4.
• Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020, April). Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. In Proceedings of the 2020 chi conference on human factors in computing systems (pp. 1–13).