Explainable AI, but Explainable to Whom? An Exploratory Case Study of xAI in Healthcare

Julie Gerlings*, Millie Søndergaard Jensen, Arisa Shollo

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingBook chapterResearchpeer-review


Advances in AI technologies have resulted in superior levels of AI-based model performance. However, this has also led to a greater degree of model complexity, resulting in “black box” models. In response to the AI black box problem, the field of explainable AI (xAI) has emerged with the aim of providing explanations catered to human understanding, trust, and transparency. Yet, we still have a limited understanding of how xAI addresses the need for explainable AI in the context of healthcare. Our research explores the differing explanation needs amongst stakeholders during the development of an AI-system for classifying COVID-19 patients for the ICU. We demonstrate that there is a constellation of stakeholders who have different explanation needs, not just the “user”. Further, the findings demonstrate how the need for xAI emerges through concerns associated with specific stakeholder groups i.e., the development team, subject matter experts, decision makers, and the audience. Our findings contribute to the expansion of xAI by highlighting that different stakeholders have different explanation needs. From a practical perspective, the study provides insights on how AI systems can be adjusted to support different stakeholders’ needs, ensuring better implementation and operation in a healthcare context.
Original languageEnglish
Title of host publicationHandbook of Artificial Intelligence in Healthcare: Vol 2 : Practicalities and Prospects
EditorsChee-Peng Lim, Yen-Wei Chen , Ashlesha Vaidya, Charu Mahorkar , Lakhmi C. Jain
Number of pages30
Place of PublicationCham
Publication date2022
ISBN (Print)9783030836191
ISBN (Electronic)9783030836207
Publication statusPublished - 2022
SeriesIntelligent Systems Reference Library

Bibliographical note

Published online: 27 November 2021.


  • Artificial intelligence
  • X-ray
  • COVID-19
  • xAI
  • Explainable AI
  • Decision making support
  • Stakeholder concerns

Cite this