Academia.eduAcademia.edu

Outline

Title
Abstract
Summary
References
All Topics
Computer Science
Artificial Intelligence

Principles of Explanation in Human-AI Systems

Profile image of Shane MuellerShane Mueller

2021, arXiv (Cornell University)

Abstract

Explainable Artificial Intelligence (XAI) has re-emerged in response to the development of modern AI and ML systems. These systems are complex and sometimes biased, but they nevertheless make decisions that impact our lives. XAI systems are frequently algorithm-focused; starting and ending with an algorithm that implements a basic untested idea about explainability. These systems are often not tested to determine whether the algorithm helps users accomplish any goals, and so their explainability remains unproven. We propose an alternative: to start with human-focused principles for the design, testing, and implementation of XAI systems, and implement algorithms to serve that purpose. In this paper, we review some of the basic concepts that have been used for user-centered XAI systems over the past 40 years of research. Based on these, we describe the "Self-Explanation Scorecard", which can help developers understand how they can empower users to by enabling self-explanation. Finally, we present a set of empiricallygrounded, user-centered design principles that may guide developers to create successful explainable systems. User-Centered Explanation in AI Although usability testing is a cornerstone of user-centered design, evaluation often comes too late to provide guidance about implementing a usable system. In response, researchers and designers have proposed guidelines that codify research on human users and advocate for the involvement of users in system development from the beginning (e.g., Greenbaum and Kyng 1991; Hoffman et al. 2010). The most famous and detailed set of guidelines may be Apple's Human Interface Guidelines (cf. Mountford 1998), but others have proposed simpler principles such as Neilson's (1994) interface design heuristics or Karat's (1998) "User's Bill of Rights". With the advent of new, powerful AI systems that are complex and difficult to understand, the field of Explainable AI (XAI) has re-emerged as an important area of human-machine interaction. Much of the interest in XAI has focused on deep learning systems. Consequently, most explanations have concentrated on technologies to visualize or otherwise expose deep networks structures, features, or

Related papers

Considerations on explainable AI and users’ mental models
Wijnand IJsselsteijn

2019

As the aim of explaining is understanding, XAI is successful when the user has a good understanding of the AI system. This paper shows, using theories from the social sciences and HCI, that appropriately capturing and accounting for the user’s mental model while explaining is key to successful XAI.

View PDFchevron_right
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI
Shane Mueller

ArXiv, 2019

This is an integrative review that address the question, "What makes for a good explanation?" with reference to AI systems. Pertinent literatures are vast. Thus, this review is necessarily selective. That said, most of the key concepts and issues are expressed in this Report. The Report encapsulates the history of computer science efforts to create systems that explain and instruct (intelligent tutoring systems and expert systems). The Report expresses the explainability issues and challenges in modern AI, and presents capsule views of the leading psychological theories of explanation. Certain articles stand out by virtue of their particular relevance to XAI, and their methods, results, and key points are highlighted. It is recommended that AI/XAI researchers be encouraged to include in their research reports fuller details on their empirical or experimental methods, in the fashion of experimental psychology research reports: details on Participants, Instructions, Procedur...

View PDFchevron_right
Innovations in Explainable AI : Bridging the Gap Between Complexity and Understanding
Keshav Jena, International Journal of Scientific Research in Computer Science, Engineering and Information Technology IJSRCSEIT

International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2023

The integration of Artificial Intelligence (AI) into various domains has witnessed remarkable advancements, yet the opacity of complex AI models poses challenges for widespread acceptance and application. This research paper delves into the field of Explainable AI (XAI) and explores innovative strategies aimed at bridging the gap between the intricacies of advanced AI algorithms and the imperative for human comprehension. We investigate key developments, including interpretable model architectures, local and visual explanation techniques, natural language explanations, and model-agnostic approaches. Emphasis is placed on ethical considerations to ensure transparency and fairness in algorithmic decision-making. By surveying and analyzing these innovations, this research contributes to the ongoing discourse on making AI systems more accessible, accountable, and trustworthy, ultimately fostering a harmonious collaboration between humans and intelligent machines in an increasingly AI-driven world.

View PDFchevron_right
What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research
Kevin Baum

Artificial Intelligence

Previous research in Explainable Artificial Intelligence (XAI) suggests that a main aim of explainability approaches is to satisfy specific interests, goals, expectations, needs, and demands regarding artificial systems (we call these 'stakeholders' desiderata') in a variety of contexts. However, the literature on XAI is vast, spreads out across multiple largely disconnected disciplines, and it often remains unclear how explainability approaches are supposed to achieve the goal of satisfying stakeholders' desiderata. This paper discusses the main classes of stakeholders calling for explainability of artificial systems and reviews their desiderata.

View PDFchevron_right
Examination of Current AI Systems within the Scope of Right to Explanation and Designing Explainable AI Systems
Orhan Gazi Yalcin

DC JURIX, 2019

This research aims to explore explainable artificial intelligence , a new sub field of artificial intelligence that is gaining importance in academic and business literature due to increased use of intelligent systems in our daily lives. As part of the research project, first of all, the necessity of the explainability in AI systems will be explained in terms of accountability, transparency, liability, and fundamental rights & freedoms. The latest explainable AI algorithms introduced by the AI researchers will be examined firstly from technical and then, from legal perspectives. Their statistical and legal competencies will be analyzed. After detecting the deficiencies of the current solutions, a comprehensive and technical AI system design will be proposed which satisfies not only the statistical requisites; but also the legal, ethical, and logical requisites.

View PDFchevron_right
Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction
Andrew Silva

International Journal of Human–Computer Interaction

Figure A1.: Reduced xAI mean scores and sub-scale mean scores for all methods. The trends for the reduced scale match the full xAI scores, with the same ordering of conditions. While sub-scale scores for usability do not present much variance across conditions, the sub-scales of transparency and simulatability offer more variation across conditions. Statistical analysis of the aggregated reduced xAI scores reveal that counterfactual explanations score higher than probability scores (p < 0.05). Appendix A. Reduced 14-Question xAI Survey Echoing the results of our primary investigation with the full survey, here we present results according to our reduced xAI survey. An ANCOVA showed that certain conditions in our experiment were rated as significantly more explainable than others (F (7, 277) = 3.14, p = 0.003). Our independent variable is the explainability method and our dependent variable is the explainability score. We include as a covariate the participant's baseline explainability score. A Shapiro-Wilk test revealed that our data were not normally distributed, but we proceed with an ANCOVA due to a lack of non-parametric alternative and the robustness of the F-test (Cochran, 1947; Glass, Peckham, & Sanders, 1972; Hack, 1958; Pearson, 1931). A Tukey's HSD post-hoc analysis reveals that Counterfactual was rated as more explainable than Probability Scores (p = 0.002), as shown in Figure A1 The reduced questionnaire, after a factor analysis and verification is given in Table A1.

View PDFchevron_right
Bridging AI Developers and End Users: an End-User-Centred Explainable AI Taxonomy and Visual Vocabularies
Diane Gromala

2019

Researchers in the re-emerging field of explainable/interpretable artificial intelligence (XAI) have not paid enough attention to the end users of AI, who may be lay persons or domain experts such as doctors, drivers, and judges. We took an end-user-centric lens and conducted a literature review of 59 technique papers on XAI algorithms and/or visualizations. We grouped the existing explanatory forms in the literature into the end-user-friendly XAI taxonomy. It consists of three forms that explain AI’s decisions: feature attribute, instance, and decision rules/trees. We also analyzed the visual representations for each explanatory form, and summarized them as the XAI visual vocabularies. Our work is a synergy of XAI algorithm, visualization, and user-centred design. It provides a practical toolkit for AI developers to define the explanation problem from a user-centred perspective, and expand the visualization space of explanations to develop more end-user-friendly XAI systems.

View PDFchevron_right
Explainable AI
Britta Wrede

KI - Künstliche Intelligenz

View PDFchevron_right
Metrics for Explainable AI: Challenges and Prospects
Shane Mueller

arXiv (Cornell University), 2018

The question addressed in this paper is: If we present to a user an AI system that explains how it works, how do we know whether the explanation works and the user has achieved a pragmatic understanding of the AI? In other words, how do we know that an explanainable AI system (XAI) is any good? Our focus is on the key concepts of measurement. We discuss specific methods for evaluating: (1) the goodness of explanations, (2) whether users are satisfied by explanations, (3) how well users understand the AI systems, (4) how curiosity motivates the search for explanations, (5) whether the user's trust and reliance on the AI are appropriate, and finally, (6) how the human-XAI work system performs. The recommendations we present derive from our integration of extensive research literatures and our own psychometric evaluations.

View PDFchevron_right
Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI
Sergio Gil-Lopez

Information Fusion, 2020

In the last few years, Artificial Intelligence (AI) has achieved a notable momentum that, if harnessed appropriately, may deliver the best of expectations over many application sectors across the field. For this to occur shortly in Machine Learning, the entire community stands in front of the barrier of explainability, an inherent problem of the latest techniques brought by sub-symbolism (e.g. ensembles or Deep Neural Networks) that were not present in the last hype of AI (namely, expert systems and rule based models). Paradigms underlying this problem fall within the so-called eXplainable AI (XAI) field, which is widely acknowledged as a crucial feature for the practical deployment of AI models. The overview presented in this article examines the existing literature and contributions already done in the field of XAI, including a prospect toward what is yet to be reached. For this purpose we summarize previous efforts made to define explainability in Machine Learning, establishing a novel definition of explainable Machine Learning that covers such prior conceptual propositions with a major focus on the audience for which the explainability is sought. Departing from this definition, we propose and discuss about a taxonomy of recent contributions related to the explainability of different Machine Learning models, including those aimed at explaining Deep Learning methods for which a second dedicated taxonomy is built and examined in detail. This critical literature analysis serves as the motivating background for a series of challenges faced by XAI, such as the interesting crossroads of data fusion and explainability. Our prospects lead toward the concept of Responsible Artificial Intelligence, namely, a methodology for the large-scale implementation of AI methods in real organizations with fairness, model explainability and accountability at its core. Our ultimate goal is to provide newcomers to the field of XAI with a thorough taxonomy that can serve as reference material in order to stimulate future research advances, but also to encourage experts and professionals from other disciplines to embrace the benefits of AI in their activity sectors, without any prior bias for its lack of interpretability.

View PDFchevron_right

References (66)

  1. Adams, B. D., Bruyn, L. E., Houde, S., Angelopoulos, P., Iwasa- Madge, K., and McCann, C. 2003. Trust in automated systems. Canada Ministry of National Defence.
  2. Alam, L. 2020. Investigating the Impact of Explanation on Re- pairing Trust in Ai Diagnostic Systems for Re-Diagnosis (Publi- cation No. 28088930) [Master's Thesis, Michigan Technological University].
  3. Aloimonos, J., Weiss, I., and Bandyopadhyay, A. 1988. Active vision. International journal of computer vision, 1(4), 333-356.
  4. Brézillon, P. 1994. Context needs in cooperative building of ex- planations. In First European Conference on Cognitive Science in Industry (pp. 443-450).
  5. Clancey, W. J. 1986. From GUIDON to NEOMYCIN and HER- ACLES in twenty short lessons. AI Magazine, 7(3), 40. Clancey, W. J. 2020. Designing agents for people: Case studies of the Brahms work practice simulation framework. [Kindle].
  6. Chari, S., Seneviratne, O., Gruen, D. M., Foreman, M. A., Das, A. K., & McGuinness, D. L. (2020, November). Explanation On- tology: A Model of Explanations for User-Centered AI. In Inter- national Semantic Web Conference (pp. 228-243). Springer.
  7. Chi, M. T., Bassok, M., Lewis, M. W., Reimann, P., and Glaser, R. 1989. Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145-182.
  8. Conant, R. C., and Ashby, W R. 1970. Every good regulator of a system must be a model of that system. International Journal of Systems Science, 1(2), 89-97.
  9. Deal, S.V. and Hoffman, R.R. (2010, September/October). The Practitioner's Cycles Part 3: Implementation problems. IEEE In- telligent Systems, pp. 77-81.
  10. Doshi-Velez, F., and Kim, B. 2017. A roadmap for a rigorous sci- ence of interpretability. ArXiv Preprint ArXiv:1702.08608. https://arxiv.org/abs/1702.08608
  11. Doyle, J. K., and Ford, D., 1998, Mental models concepts for sys- tem dynamics research, System Dynamics Review, 14(1), 3-29.
  12. Doyle, J., Radzicki, M., & Trees, W. 2008. Measuring change in mental models of complex dynamic systems. Complex Decision Making, 269-294.
  13. Doyle, D., Tsymbal, A., & Cunningham, P. 2003. A review of ex- planation and explanation in case-based reasoning. Trinity Col- lege Dublin, Department of Computer Science Technical Report TCD-CS-2003-41. http://www.tara.tcd.ie/handle/2262/12919
  14. Fallon, C. K., and Blaha, L. M. 2018. Improving Automation Transparency: Addressing Some of Machine Learning's Unique Challenges. International Conference on Augmented Cognition, 245-254. Springer.
  15. Forrester, J. 1961. Industrial Dynamics. Cambridge, MA: Pro- ductivity Press
  16. Findlay, J. M., Findlay, J. M., and Gilchrist, I. D. 2003. Active vi- sion: The psychology of looking and seeing (No. 37). Oxford University Press.
  17. Goodman, B., & Flaxman, S. 2017. European Union regulations on algorithmic decision-making and a "right to explanation". AI magazine, 38(3), 50-57.
  18. Greenbaum, J. & Kyng, M. (Eds.). 1991. Design at work: Coop- erative design of computer systems. Hillsdale, NJ: Lawrence Erl- baum Associates.
  19. Grice. H. P. 1975. Logic and Conversation. In Syntax and seman- tics 3: Speech arts. 41-58.
  20. Guerlain, S. 1995. Using the critiquing approach to cope with brittle expert systems. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 39, pp. 233-237). SAGE Publications Sage CA: Los Angeles, CA.
  21. Greenbaum, J. and Kyng, M. (Eds.) 1991. Design at work: Coop- erative design of computer systems. Hillsdale, NJ: Lawrence Erl- baum Associates.
  22. Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., and Darrell, T. 2016. Generating visual explanations. Euro- pean Conference on Computer Vision, 3-19. http://link.springer.- com/chapter/10.1007/978-3-319-46493-0_1
  23. Hoffman, R. R., Deal, S. V., Potter, S., and Roth, E. M. 2010. The practitioner's cycles, part 2: Solving envisioned world problems. IEEE Intelligent Systems, 25(3), 6-11.
  24. Hoffman, R.R. (ed) 2012. Collected Essays on Human-Centered Computing, 2001-2011. New York: IEEE Computer Soc. Press.
  25. Hoffman, R.R. 2017. A taxonomy of emergent trusting in the hu- man-machine relationship. In P. Smith and R.R. Hoffman, Eds., Cognitive systems engineering: The future for a changing world (137-164). Boca Raton, FL: Taylor & Francis.
  26. Hoffman, R. R., Klein, G., and Mueller, S. T. 2018b. Explaining explanation for "Explainable AI". Proceedings of the 2018 con- ference of the Human Factors and Ergonomic Society (HFES), Philadelphia PA, October 2018.
  27. Hoffman, R. R., Mueller, S. T., and Klein, G. 2017. Explaining Explanation, Part 2: Empirical Foundations. IEEE Intelligent Sys- tems, 32(4), 78-86.
  28. Hoffman, R., Miller, T., Mueller, S. T., Klein, G. and Clancey, W. J. 2018c. Explaining Explanation, Part 4: A Deep Dive on Deep Nets. IEEE Intelligent Systems, 33(3), 87-95.
  29. Hoffman, R. R., Mueller, S. T., Klein, G., and Litman, J. 2018a. Metrics for explainable AI: Challenges and prospects. arXiv:1812.04608.
  30. Hoffman, R. R., Clancey, W. J., and Mueller, S. T. 2020. Ex- plaining AI as an Exploratory Process: The Peircean Abduction Model. arXiv:2009.14795.
  31. Karat, C. M. 1998. Guaranteeing rights for the user. Communica- tions of the ACM, 41(12), 29-31.
  32. Karsenty, L., and Brezillon, P. J. 1995. Cooperative problem solving and explanation. Expert Systems with Applications, 8(4), 445-462.
  33. Kass, R., and Finin, T. 1988. The need for user models in generat- ing expert system explanation. International Journal of Expert Systems, 1(4), 345-375.
  34. Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. 2020. Interpreting Interpretability: Under- standing Data Scientists' Use of Interpretability Tools for Ma- chine Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-14).
  35. Kim, B., Khanna, R., and Koyejo, O. O. 2016. Examples are not enough, learn to criticize! Criticism for interpretability. In Ad- vances in Neural Information Processing Systems (2280-2288).
  36. Klein, G. and Hoffman, R. R. 2008. Macrocognition, mental models, and cognitive task analysis methodology. In J. M. Schraagen, L. G. Militello, T. Ormerod and R. Lipshitz (Eds.), Naturalistic decision making and macrocognition (pp. 57-80). Aldershot, England: Ashgate.
  37. Klein, G. Hoffman, R. R., and Mueller. S. T. 2019. Scorecard for self-explaining capabilities of AI Systems. Technical Report Pre- pared by Task Area 2, DARPA XAI Program.
  38. Kulesza, T., Burnett, M., Wong, W.-K., and Stumpf, S. 2015. Principles of explanatory debugging to personalize interactive machine learning. Proceedings of the 20th International Confer- ence on Intelligent User Interfaces, 126-137.
  39. Lakkaraju, H., Kamar, E., Caruana, R., and Leskovec, J. 2017. In- terpretable & explorable approximations of black box models. ArXiv Preprint ArXiv:1707.01154.
  40. Langlotz, C. P., and Shortliffe, E. H. 1989. The critiquing ap- proach to automated advice and explanation: Rationale and exam- ples. In Expert Knowledge an Explanation: The Knowledge-Lan- guage Interface, Charlie Ellis (Ed). Ellis Horwood Limited.
  41. Leake, D. B. 1995. Abduction, experience, and goals: A model of everyday abductive explanation. Journal of Experimental & The- oretical Artificial Intelligence, 7(4), 407-428.
  42. Lim, B. Y., Dey, A. K., and Avrahami, D. 2009. Why and why not explanations improve the intelligibility of context-aware intel- ligent systems. In Proceedings of the SIGCHI Conference on Hu- man Factors in Computing Systems (pp. 2119-2128). ACM. Meyers, C., and Jones, T. B. 1993. Promoting Active Learning. Strategies for the College Classroom. Jossey-Bass Inc., Publish- ers, 350 Sansome Street, San Francisco, CA 94104.
  43. Miller, T. 2017. Explanation in artificial intelligence: Insights from the social sciences. ArXiv:1706.07269 [Cs]. http://arxiv.org/ abs/1706.07269
  44. Moore, J. D., and Swartout, W. R. 1988. Explanation in expert systems: A survey. University of Southern California Marina del rey Information Sciences Institute. DTIC #ADA206283.
  45. Mountford, S. J. 1998. A history of the Apple human interface group. ACM SIGCHI Bulletin, 30(2), 144-146.
  46. Mueller, S. T., Agarwal, P. Linja, A, Dave, N., and Alam, L. 2020. The unreasonable ineptitude of deep image classification networks. Proceedings of the 64 th annual meeting of the Human Factors Society.
  47. Mueller, S.T., Hoffman, R.R., Clancey, W, Emrey, A., and Klein, G. 2019. "Explanation in Human-AI Systems: A Literature Meta- Review, Synopsis of Key Ideas and Publications, and Bibliogra- phy for Explainable AI." Report on Award No. FA8650-17-2- 7711, DARPA XAI Program. DTIC AD1073994.
  48. Mueller, S.T. and Klein, G. 2011. Improving Users' Mental Mod- els of Intelligent Software Tools," Intelligent Systems, IEEE, 26(2), 77-83.
  49. Muir, B. M. 1994. Trust in automation: Part I. Theoretical issues in the study of trust and human intervention in automated sys- tems. Ergonomics, 37(11), 1905-1922.
  50. Muir, B. M., and Moray, N. 1996. Trust in automation. Part II. Experimental studies of trust and human intervention in a process control simulation. Ergonomics, 39(3), 429-460.
  51. Nielson, J. 1994. 10 Usability heuristics for user interface design. https://www.nngroup.com/articles/ten-usability-heuristics/ Naiseh, M., Jiang, N., Ma, J., & Ali, R. 2020. Personalising Ex- plainable Recommendations: Literature and Conceptualisation. In World Conference on Information Systems and Technologies (518-533). Springer, Cham.
  52. Ribera, M., and Lapedriza, A. 2019. Can we do better explana- tions? A proposal of user-centered explainable AI. In IUI Work- shops.
  53. Ribeiro, M. T., Singh, S., and Guestrin, C. 2016. Why Should I Trust You?: Explaining the Predictions of Any Classifier. In Pro- ceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135-1144). ACM. Rozenblit, L., and Keil, F. 2002. The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive science, 26(5), 521-562.
  54. Rouse, W. B., and Morris, N. M. 1986. On looking into the black box: Prospects and limits in the search for mental models. Psy- chological Bulletin, 100(3), 349.
  55. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. 2017. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (pp. 618- 626).
  56. Settles, B. 2009. Active learning literature survey. University of Wisconsin-Madison Department of Computer Sciences.
  57. Sørmo, F., Cassens, J., and Aamodt, A. 2005. Explanation in case-based reasoning-perspectives and goals. Artificial Intelli- gence Review, 24(2), 109-143.
  58. Swartout, W. R. 1977. A digitalis therapy advisor with explana- tions. Proceedings of the 5th International Joint Conference on Artificial Intelligence-Volume 2, 819-825. http://dl.acm.org/cita- tion.cfm?id=1623009
  59. Swartout, W. R., and Moore, J. D. 1993. Explanation in second generation expert systems. Second Generation Expert Systems, 543-585.
  60. Tate, D., Grier, R., A. Martin, C., L. Moses, F., and Sparrow, D. 2016. A Framework for evidence based licensure of adaptive au- tonomous systems. IDA Paper P-5325. 10.13140/ RG.2.2.11845.86247
  61. Veinott, E., Klein, G., and Wiggins, S. 2010. Evaluating the ef- fect of the Premortem Method on Plan confidence. Paper pre- sented at the 7th International ISCRAM Conference 2010.
  62. Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. 2019. Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1-15).110111
  63. Wick, M. R., and Thompson, W. B. 1992. Reconstructive expert system explanation. Artificial Intelligence, 54(1-2), 33-70.
  64. Woolf, B. 2007. Building Intelligent Interactive Tutors: Student- centered strategies for revolutionizing e-learning. Morgan Kauf- mann Publishers Inc.
  65. Yang, S. C. H., and Shafto, P. 2017. Explainable artificial intelli- gence via Bayesian Teaching. In NIPS 2017 workshop on Teach- ing Machines, Robots, and Humans (pp. 127-137).
  66. Zeiler, M. D., and Fergus, R. 2014. Visualizing and understand- ing convolutional networks. In European conference on computer vision (pp. 818-833). Springer, Cham.

Related papers

Explainability as a User Requirement for Artificial Intelligence Systems
Mia Schmitz

Computer

View PDFchevron_right
Can we do better explanations? A proposal of user-centered explainable AI
Mireia Ribera

2019

Artificial Intelligence systems are spreading to multiple applications and they are used by a more diverse audience. With this change of the use scenario, AI users will increasingly require explanations. The first part of this paper makes a review of the state of the art of Explainable AI and highlights how the current research is not paying enough attention to whom the explanations are targeted. In the second part of the paper, it is suggested a new explainability pipeline, where users are classified in three main groups (developers or AI researchers, domain experts and lay users). Inspired by the cooperative principles of conversations, it is discussed how creating different explanations for each of the targeted groups can overcome some of the difficulties related to creating good explanations and evaluating them.

View PDFchevron_right
Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goals
Diane Gromala

arXiv (Cornell University), 2023

Non-technical end-users are silent and invisible users of the state-of-the-art explainable artificial intelligence (XAI) technologies. Their demands and requirements for AI explainability are not incorporated into the design and evaluation of XAI techniques, which are developed to explain the rationales of AI decisions to end-users and assist their critical decisions. This makes XAI techniques ineffective or even harmful in high-stakes applications, such as healthcare, criminal justice, finance, and autonomous driving systems. To systematically understand end-users' requirements to support the technical development of XAI, we conducted the EUCA user study with 32 layperson participants in four AI-assisted critical tasks. The study identified comprehensive user requirements for feature-, example-, and rulebased XAI techniques (manifested by the enduser-friendly explanation forms) and XAI evaluation objectives (manifested by the explanation goals), which were shown to be helpful to directly inspire the proposal of new XAI algorithms and evaluation metrics. The EUCA study findings, the identified explanation forms and goals for technical specification, and the EUCA study dataset support the design and evaluation of end-user-centered XAI techniques for accessible, safe, and accountable AI.

View PDFchevron_right
EUCA: the End-User-Centered Explainable AI Framework
Diane Gromala

arXiv (Cornell University), 2021

The ability to explain decisions to end-users is a necessity to deploy AI as critical decision support. Yet making AI explainable to non-technical end-users is a relatively ignored and challenging problem. To bridge the gap, we first identify twelve end-userfriendly explanatory forms that do not require technical knowledge to comprehend, including feature-, example-, and rule-based explanations. We then instantiate the explanatory forms as prototyping cards in four AI-assisted critical decision-making tasks, and conduct a user study to co-design low-fidelity prototypes with 32 layperson participants. The results confirm the relevance of using explanatory forms as building blocks of explanations, and identify their proprieties-pros, cons, applicable explanation goals, and design implications. The explanatory forms, their proprieties, and prototyping supports (including a suggested prototyping process, design templates and exemplars, and associated algorithms to actualize explanatory forms) constitute the End-User-Centered explainable AI framework EUCA, and is available at http://weinajin.github.io/end-user-xai. It serves as a practical prototyping toolkit for HCI/AI practitioners and researchers to understand user requirements and build end-user-centered explainable AI. CCS Concepts: • Computing methodologies → Artificial intelligence; • Human-centered computing → User studies.

View PDFchevron_right
Reviewing the Need for Explainable Artificial Intelligence (xAI)
Arisa Shollo

Proceedings of the Annual Hawaii International Conference on System Sciences, 2021

The diffusion of artificial intelligence (AI) applications in organizations and society has fueled research on explaining AI decisions. The explainable AI (xAI) field is rapidly expanding with numerous ways of extracting information and visualizing the output of AI technologies (e.g. deep neural networks). Yet, we have a limited understanding of how xAI research addresses the need for explainable AI. We conduct a systematic review of xAI literature on the topic and identify four thematic debates central to how xAI addresses the black-box problem. Based on this critical analysis of the xAI scholarship we synthesize the findings into a future research agenda to further the xAI body of knowledge.

View PDFchevron_right

Related topics

  • Computer Science
  • Artificial Intelligence
  • Balanced Scorecard
  • Artificial Intelligence and Robo...
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved