Trends in Explainable AI (XAI) Literature

11 min readJan 16, 2023

This is a report on XAI-Scholar, a collection of XAI* papers by me, and some interesting trends and insights I derived from it. This is an informal version of the Arxiv report here. The reproducing code and data are available at this github repository. The collection is based on the SemanticScholar database using this Python API.

*By XAI I’m referring to a relatively inclusive definition for research articles that discuss the development, implementation, or practice, of explanations/interpretations in modern AI systems (whether they refer to themselves as such or not). This definition is aligned with the most common definition that I’ve observed in various curated collections of XAI papers, workshops and journal issues on XAI.

In recent years XAI research has started to reach a size that makes it (1) difficult to grasp with manual surveying; (2) large enough that it’s possible to see overall empirical and statistical trends. My goal was to collect a large and well-formatted set of XAI papers to make this empirical analysis possible. The analysis below can serve as a proof of concept for what’s possible to do with this. My personal motivation was to look at the multidisciplinarity of XAI, but there’s many other possible uses.

XAI research has multiple properties that make it difficult to observe in its entirety, compared to similar fields:

It’s very multidisciplinary, with non-negligible communities in many different fields that don’t often interact or share venues.
The terminology that papers use to self-identify as XAI research is not unique to XAI (E.g., “xai” and “Xai Xai” are names with multiple senses which appear in research), and this terminology is much more recent than the actual history of XAI.
The most prevalent definitions of “XAI research” papers often include papers that don’t self-identify as XAI, as long as they attempt to explain AI technology.

Some findings:

XAI research has had its biggest “expansion” growth spikes outside of Computer Science in 2016, 2018 and 2021 (in particular 2021).
There is clear growth over time in the relative proportion of papers that are authored by authors which traditionally publish in two or more distinct fields of study.
CS has different citing relationships with different XAI fields. For example, XAI-CS cites XAI-Psychology more often than the vice versa, but the relationship flips for XAI-CS and XAI-Medicine. This “direction” of influence shows which fields often inform/are informed by which other fields in the current literature.
There is a difference across XAI fields by how often they inform non-XAI research. The highest proportion being in XAI-Biology, XAI-Engineering and XAI-Law — -while the lowest proportion being in XAI-Psychology, XAI-Business and XAI-Philosophy, whose influence more often carries to other XAI literature.
Citation behavior across papers is significantly different between fields. For example, the top-cited Philosophy papers cited by XAI-Philosophy are significantly different from those cited by XAI-CS, and so on. Unsurprisingly, citations outside of a field tend to focus on a smaller variety of papers in that field, but the papers that “break out of” the traditional boundaries of their field are not always the most cited papers in that field.
The collection can serve as a paper discovery engine by seeing which XAI papers, for example, are the most influential to papers outside of their field, or outside of XAI; or which non-XAI papers of a particular field are the most informative to XAI in another field. The arxiv report and github repository have a lot of examples, but I will also include some here.

Collection Method

Step 1- Keyword-based search

I searched for the following keywords in a SemanticScholar query and filter them using exact-match using papers’ title + abstract.

The keywords are: “ xai “, “(xai)”, “hcxai”, “explainability”, “interpretability”, “explainable ai”, “explainable artificial intelligence”, “interpretable ml”, “interpretable machine learning”, “interpretable model”, “feature attribution”, “feature importance”, “global explanation”, “local explanation”, “local interpretation”, “global interpretation”, “model explanation”, “model interpretation”, “saliency”, “counterfactual explanation”.

This set of keywords is actually pretty conservative (initially I started with a bigger list), but even with this conservative set, 1-keyword-match filtering unfortunately resulted in many non-XAI papers being retrieved for various reasons — so I only include papers match 2 unique keywords or more. A random sample of 100 papers yielded 99% precision with a manual check (one paper mentioned keywords in the abstract simply as motivation for adjacent research).

The keyword-based search retrieved 3101 papers.

Step 2- Manually curated collections

I took the papers from the following curated lists:

Then searched for them by title using the SemanticScholar API with fuzzy matching.

This added 766 papers to 3867 papers total.

Step 3- Citation tree expansion with manual filtering

I took the top 2000 papers that were the most cited by the set of papers from steps 1–2 and manually selected the papers about XAI from them myself.

This added 648 papers to 4515 papers total.

Step 4- Citation tree expansion with automatic filtering

After every step 1–3 above, I retrieved all citations and references of the collected papers and filtered them via the 2-keyword-match method. I did this recursively until no more new papers were added.

This added 709 papers overall to 5224 papers total.

Step 5- Manual quality check

Finally, I skimmed over the dataset heuristically and found 25 incorrectly-attributed papers. Removing them brought the final number down to 5199 papers.

Summary:

Steps 1 and 4 were based on automatic filtering with keywords, with the role of retrieving papers that self-identify as XAI according to recent popular terminology. Steps 2 and 3 were based on manual filtering which was not constrained to specific keywords, with the role of retrieving papers that were published before XAI terminology converged, and papers that don’t self-identify as XAI, but are considered as such by the community.

The collection

The final collection has 5199 papers. It is available here.

Each paper has:

The SemanticScholar ID and URL
Title
Abstract
Authors
Number of citations
Number of references
Year
Venue
Field of study
SemanticScholar’s “tldr” summary

The following data for each paper can also be retrieved from SemanticScholar separately (the github repo has the necessary code):

References list
Citations list
SemanticScholar’s embedding vector

The final size of the data is 19 MB. The size of the full data, after retrieving refs and cites from SemanticScholar, would be around 950 MB.

This is the distribution across fields of study:

And without CS for readability:

Limitations:

It goes without saying that the retrieval here is biased: Steps 1 and 4 are biased towards specific terminology, and steps 2 and 3 are biased towards influential/highly-cited papers in CS and Mathematics.
SemanticScholar is a bit noisy — don’t expect perfect formatting. Some papers are missing their abstracts; some papers’ fields are wrong; venue names are inconsistent (i.e., there’s multiple strings that refer to the same venue); missing citations/references, authors, and so on. This is the minority, but it’s not negligible either.
<100% precision — I did my best, but some non-XAI papers likely slipped through. From the sampling I did, you can expect very high precision, but not 100%.
Obviously, this collection is far from 100% recall. From my very biased and anecdotal observations, I estimate that the “true” body of XAI literature is around x1.5 to x2 as big as this collection, as of 31/Dec/2022 — so around 8k to 10k papers total. The missed papers likely either predate the time when terminology started converging, or with a low amount of citations due to steps 2–3 prioritizing highly-cited papers.
The collection was retrieved over a period of time in December 2022, but some venues seem to have a delay from publication date to proceedings/Semantic Scholar appearance — so observations of growth trends may want to consider 2022 as a partial year (I don’t have any evidence on how big or small this issue is).

When using the collection, rely on your own judgment to decide if the observed trends can overcome the margin of error based on these limitations.

Growth trends

First, XAI generally shows relative yearly growth — -but this growth is largely controlled by Computer Science (which shows the same growth trend).

Controlling for non-CS papers reveals different growth trends. For example, XAI-Medicine shows exponential growth, in particular with large relative growth in 2016, 2018 and 2021.

These trends also hold when controlling for non-Medicine and non-CS papers.

Overall, it appears that XAI has had the biggest expansions into “non-central” fields in 2016, 2018 and 2021.

Collaboration trends

We can look at the interaction between the fields of study variable and the author-paper graph by defining the “field” of an author as the field of the majority of their papers (retrieved separately via SemanticScholar).

Then we can count the number of collaborations. We can visualize this network as a weighted undirected graph. The edge weights are the % of collaborations out of all papers with at least one collaboration (edges lower than 5% were omitted).

Common field pairings for papers that are authored by authors that publish in different fields.

As before, since CS skews the whole scale, we can look at the same graph sans CS for a fine-grained view at other field pairings (edges lower than 3% were omitted):

Common field pairings (sans CS) for papers that are authored by authors that publish in different fields.

And of course we can check for growth trends in collaborations. This plot shows the yearly number of papers with a collaboration, out of all yearly papers:

It’s hard to see in this graph, but if we plot the percentage of papers with collaborations, we can see the proportional growth over time:

If we consider 2016 as an outlier, there is a gradual increase in the proportion of papers with authors that traditionally publish in different fields working together. Pretty cool!

Citation trends

Finally we can check the interaction between the field of study variable and the citation-reference graph.

The easiest trends to analyze are around CS, since all fields have a significant citing relationship with CS.

For example, which XAI fields are the most cited by XAI-CS (top 5)?

Being informed by XAI-Mathematics makes sense, of course, but we can also see that XAI-CS literature is non-negligibly informed by XAI-Engineering, XAI-Psychology, XAI-Medicine and XAI-Business.

What is the “direction” of the citation relationship between CS and other fields?

We can visualize this as a weighted directed graph: below we can see the citation vs. reference relationships between XAI fields around CS — every pair of edges between two fields is normalized to sum to 100%. Relationships that span less than 100 cites across both edges are omitted.

The graph basically shows which of two fields is more informed by the other: We can see that XAI papers in Psychology and Mathematics are cited by XAI-CS more often than the other way around. XAI-Engineering/XAI-Medicine are roughly equal, and for all other fields, the relationship flips — meaning that they’re more informed by XAI-CS than XAI-CS is informed by them.

What are the citation relationships in XAI between non-CS fields?

As before we can omit CS to look at other pairings. This graph shows the top-3 cited XAI fields for every XAI field. All outgoing edges from a node are normalized to sum to 100%. Fields with less than 20 references to other fields are omitted.

Which XAI fields proportionally inform non-XAI literature the most?

If we define papers outside of the collection as non-XAI papers, this plot shows the ranking of XAI fields by the percentage of non-XAI citations out of all citations to that XAI field. It basically shows which fields most often inform non-XAI literature. (fields with less than 100 outgoing citations were omitted).

For example, XAI-Biology is relatively often cited by non-XAI literature, and the opposite is true for XAI-Philosophy, which seems to be comparatively more often cited by XAI.

Paper-level Citation Trends and Paper Discovery

Finally we can look at citation behavior at the level of individual papers.

For example, when XAI-CS cites Philosophy, are they citing a wide variety of papers, or a select minority of papers?

The top-10 cited Philosophy papers by XAI-CS control 32% of the citations between these fields. We can check this more generally by looking at the entropy of the paper distribution for paper citations by XAI-CS for each field:

We can see here that Philosophy has a relatively peaky distribution as we expected. If you are curious, by the way — “Law” citations by XAI-CS are overwhelmingly controlled by one paper in particular (“Accountable Algorithms”) with 44% of the citations :)

The Philosophy papers as cited by XAI-CS are very different from those cited by XAI-Philosophy!

For illustration, here is another scenario with Psychology and CS:

Conclusions

XAI research is converging around specific terminology at scale that makes it possible to observe trends empirically. While the retrieval process has some limitations and biases, it’s possible to account for them on some level (for example, by acknowledging that the retrieval is biased towards CS, and seeing trends that overcome this bias in the opposite direction). The analysis I did here is mostly focused on the field of study variable, but it’s also possible to look at many other trends — check the github repository if you would like to do this. Thanks for reading :)