Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
An Overview of Ai Opacity in Policy and Regulation
Use Case: Ai-Based Student Proctoring
Use Case Description
References

The role of explainable AI in the context of the AI Act

Profile image of Gabriele MazziniGabriele Mazzini

2023 ACM Conference on Fairness, Accountability, and Transparency

https://doi.org/10.1145/3593013.3594069
visibility

…

description

12 pages

link

1 file

Abstract

The proposed EU regulation for Artificial Intelligence (AI), the AI Act, has sparked some debate about the role of explainable AI (XAI) in high-risk AI systems. Some argue that black-box AI models will have to be replaced with transparent ones, others argue that using XAI techniques might help in achieving compliance. This work aims to bring some clarity as regards XAI in the context of the AI Act and focuses in particular on the AI Act requirements for transparency and human oversight. After outlining key points of the debate and describing the current limitations of XAI techniques, this paper carries out an interdisciplinary analysis of how the AI Act addresses the issue of opaque AI systems. In particular, we argue that neither does the AI Act mandate a requirement for XAI, which is the subject of intense scientific research and is not without technical limitations, nor does it ban the use of black-box AI systems. Instead, the AI Act aims to achieve its stated policy objectives with the focus on transparency (including documentation) and human oversight. Finally, in order to concretely illustrate our findings and conclusions, a use case on AI-based proctoring is presented.

Figures (3)
The proposed EU regulation for Artificial Intelligence (AI), the AI Act, has sparked some debate about the role of explainable AI (XAI) in high-risk AI systems. Some argue that black-box AI models will have to be replaced with transparent ones, others argue that using XAI techniques might help in achieving compliance. This work aims to bring some clarity as regards XAI in the context of the AI Act and focuses in particular on the AI Act requirements for transparency and human oversight. After outlining key points of the debate and describing the current limitations of XAI techniques, this paper carries out an interdisciplinary analysis of how the AI Act addresses the issue of opaque AI systems. In particular, we argue that neither does the AI Act mandate a requirement for XAI, which is the subject of intense scientific research and is not without technical limitations, nor does it ban the use of black-box Al systems. Instead, the AI Act aims to achieve its stated policy objectives with the focus on transparency (including documentation) and human oversight. Finally, in order to concretely illustrate our findings and conclusions, a use case on Al-based proctoring is presented. - Computing methodologies — Artificial intelligence; + Ap. plied computing — Law. The role of explainable Al in the context of the Al Act KEYWORDS explainable artificial intelligence, XAI, AI Act, EU regulation, trust- worthy AI, transparency, human oversight
The proposed EU regulation for Artificial Intelligence (AI), the AI Act, has sparked some debate about the role of explainable AI (XAI) in high-risk AI systems. Some argue that black-box AI models will have to be replaced with transparent ones, others argue that using XAI techniques might help in achieving compliance. This work aims to bring some clarity as regards XAI in the context of the AI Act and focuses in particular on the AI Act requirements for transparency and human oversight. After outlining key points of the debate and describing the current limitations of XAI techniques, this paper carries out an interdisciplinary analysis of how the AI Act addresses the issue of opaque AI systems. In particular, we argue that neither does the AI Act mandate a requirement for XAI, which is the subject of intense scientific research and is not without technical limitations, nor does it ban the use of black-box Al systems. Instead, the AI Act aims to achieve its stated policy objectives with the focus on transparency (including documentation) and human oversight. Finally, in order to concretely illustrate our findings and conclusions, a use case on Al-based proctoring is presented. - Computing methodologies — Artificial intelligence; + Ap. plied computing — Law. The role of explainable Al in the context of the Al Act KEYWORDS explainable artificial intelligence, XAI, AI Act, EU regulation, trust- worthy AI, transparency, human oversight
videos as input data and perform some AJ-driven processing to in- fer high-level information of a person’s facial behaviour including micro-expressions, emotions, gaze direction or head pose. Facial behaviour analysis algorithms may be used in a wide range of appli- cations areas [58]. An example of these systems is OpenFace2.0 [14], which provides facial landmark detection (FL), head pose estimation (HP), gaze direction analysis (GA), and the intensity of activation of a set of 18 facial micro-expressions called Action Units (AUs). AUs encode the movements of facial muscles and their intensity according to the Facial Action Coding System (FACS [35]). Examples of AUs include: brow lowerer (AU4), cheek raiser (AU6) or lip corner puller (AU12). Thus, AUs provide an objective and fine-grained description of a person’s facial behaviour. This toolkit combines a pipeline of deep learning and other machine learning techniques with real-time video processing capabilities, and is able to run from a simple webcam and in any type of hardware. The tool provides two different outputs: a video with the extracted facial informa- tion superimposed over the original input video (Figure 1-top), and a CSV file with detailed logs including confidence values on the predictions made (Figure 1-bottom 1°). Figure 1: Outputs provided by the OpenFace 2.0 toolkit [14]. Top: visualisation of extracted facial information superim- posed over the original input video. The blue box represents head pose orientation, red dots are extracted facial landmark positions, green lines indicate gaze direction and the his- tograms on the left correspond to the intensity of activation of each AU. Bottom: excerpt of the log detailing the value of each of these data values per frame.
videos as input data and perform some AJ-driven processing to in- fer high-level information of a person’s facial behaviour including micro-expressions, emotions, gaze direction or head pose. Facial behaviour analysis algorithms may be used in a wide range of appli- cations areas [58]. An example of these systems is OpenFace2.0 [14], which provides facial landmark detection (FL), head pose estimation (HP), gaze direction analysis (GA), and the intensity of activation of a set of 18 facial micro-expressions called Action Units (AUs). AUs encode the movements of facial muscles and their intensity according to the Facial Action Coding System (FACS [35]). Examples of AUs include: brow lowerer (AU4), cheek raiser (AU6) or lip corner puller (AU12). Thus, AUs provide an objective and fine-grained description of a person’s facial behaviour. This toolkit combines a pipeline of deep learning and other machine learning techniques with real-time video processing capabilities, and is able to run from a simple webcam and in any type of hardware. The tool provides two different outputs: a video with the extracted facial informa- tion superimposed over the original input video (Figure 1-top), and a CSV file with detailed logs including confidence values on the predictions made (Figure 1-bottom 1°). Figure 1: Outputs provided by the OpenFace 2.0 toolkit [14]. Top: visualisation of extracted facial information superim- posed over the original input video. The blue box represents head pose orientation, red dots are extracted facial landmark positions, green lines indicate gaze direction and the his- tograms on the left correspond to the intensity of activation of each AU. Bottom: excerpt of the log detailing the value of each of these data values per frame.
Figure 2: From AI computational tasks to real-world oper- ational AI systems conceived for a specific context and in- tended purpose: AI-based proctoring audio. That is, we can consider a system for online proctoring as containing multiple components and computational tasks, some of them AlI-based and some of them not (see Figure 2). Furthermore, for the purpose of this paper, we can consider that Al-based com- putational tasks can be based on either interpretable AI models or opaque models, regardless of whether XAI techniques are ap- plied or not. Keeping this in mind, in the next section we aim to show what elements can be brought into play to meet the AI Act requirements of transparency and human oversight — to the extent they were legally applicable - without the need for the models to be interpretable or for XAI techniques to be used.
Figure 2: From AI computational tasks to real-world oper- ational AI systems conceived for a specific context and in- tended purpose: AI-based proctoring audio. That is, we can consider a system for online proctoring as containing multiple components and computational tasks, some of them AlI-based and some of them not (see Figure 2). Furthermore, for the purpose of this paper, we can consider that Al-based com- putational tasks can be based on either interpretable AI models or opaque models, regardless of whether XAI techniques are ap- plied or not. Keeping this in mind, in the next section we aim to show what elements can be brought into play to meet the AI Act requirements of transparency and human oversight — to the extent they were legally applicable - without the need for the models to be interpretable or for XAI techniques to be used.

Related papers

Explainability and Transparency in Artificial Intelligence: Ethical Imperatives and Practical Challenges
IJAERS Journal

Artificial Intelligence (AI) is increasingly embedded in high-stakes domains such as healthcare, finance, and law enforcement, where opaque decision-making raises significant ethical concerns. Among the core challenges in AI ethics are explainability and transparency—key to fostering trust, accountability, and fairness in algorithmic systems. This review explores the ethical foundations of explainable AI (XAI), surveys leading technical approaches such as model-agnostic interpretability techniques and post-hoc explanation methods and examines their inherent limitations and trade-offs. A real-world case study from the healthcare sector highlights the critical consequences of deploying non-transparent AI models in clinical decision-making. The article also discusses emerging regulatory frameworks and underscores the need for interdisciplinary collaboration to address the evolving ethical landscape. The review concludes with recommendations for aligning technical innovation with ethical imperatives through responsible design and governance.

View PDFchevron_right
Examination of Current AI Systems within the Scope of Right to Explanation and Designing Explainable AI Systems
Orhan Gazi Yalcin

DC JURIX, 2019

This research aims to explore explainable artificial intelligence , a new sub field of artificial intelligence that is gaining importance in academic and business literature due to increased use of intelligent systems in our daily lives. As part of the research project, first of all, the necessity of the explainability in AI systems will be explained in terms of accountability, transparency, liability, and fundamental rights & freedoms. The latest explainable AI algorithms introduced by the AI researchers will be examined firstly from technical and then, from legal perspectives. Their statistical and legal competencies will be analyzed. After detecting the deficiencies of the current solutions, a comprehensive and technical AI system design will be proposed which satisfies not only the statistical requisites; but also the legal, ethical, and logical requisites.

View PDFchevron_right
Transparency and Explainability of AI Systems: Ethical Guidelines in Practice
Sari Kujala

Lecture Notes in Computer Science, 2022

Recent studies have highlighted transparency and explainability as important quality requirements of AI systems. However, there are still relatively few case studies that describe the current state of defining these quality requirements in practice. [Question] The goal of our study was to explore what ethical guidelines organizations have defined for the development of transparent and explainable AI systems. We analyzed the ethical guidelines in 16 organizations representing different industries and public sector. [Results] In the ethical guidelines, the importance of transparency was highlighted by almost all of the organizations, and explainability was considered as an integral part of transparency. Building trust in AI systems was one of the key reasons for developing transparency and explainability, and customers and users were raised as the main target groups of the explanations. The organizations also mentioned developers, partners, and stakeholders as important groups needing explanations. The ethical guidelines contained the following aspects of the AI system that should be explained: the purpose, role of AI, inputs, behavior, data utilized, outputs, and limitations. The guidelines also pointed out that transparency and explainability relate to several other quality requirements, such as trustworthiness, understandability, traceability, privacy, auditability, and fairness. [Contribution] For researchers, this paper provides insights into what organizations consider important in the transparency and, in particular, explainability of AI systems. For practitioners, this study suggests a structured way to define explainability requirements of AI systems.

View PDFchevron_right
The Ethics of Understanding: Exploring Moral Implications of Explainable AI
Balkrishna Yadav

International Journal of Science and Research (IJSR), 2024

Explainable AI (XAI) refers to a specific kind of artificial intelligence systems that are intentionally built to ensure that their operations and results can be comprehended by humans. The main objective is to enhance the transparency of AI systems' decisionmaking processes, allowing users to understand the rationale behind certain judgements. Important elements of XAI include transparency, interpretability, reasoning, traceability, and user-friendliness. The advantages of Explainable Artificial Intelligence (XAI) include trust and confidence in the system's outputs, ensuring accountability and compliance with regulations, facilitating debugging and refinement of the model, promoting greater cooperation between humans and AI systems, and enabling informed decision-making based on transparent explanations. Examples of XAI applications include healthcare, banking, legal systems, and autonomous systems. Healthcare guarantees that AI-powered diagnosis and treatment suggestions are presented in a straightforward and comprehensible manner, while finance offers explicit elucidations for credit score, loan approvals, and fraud detection. Legal frameworks promote transparency in the implementation of AI applications, therefore assuring equity and mitigating the risk of biases. As artificial intelligence becomes more embedded in society, the significance of explainability will persistently increase, guaranteeing responsible and efficient utilization of these systems. The study of explainable AI is essential as it tackles the ethical, sociological, and technical difficulties presented by the growing use of AI systems. The level of transparency in AI decision-making processes has a direct influence on accountability, since systems that are not transparent might hide the reasoning behind the judgements. Explainability is crucial for detecting and reducing biases in AI systems, so preventing them from perpetuating or worsening social injustices. The objective of the study is to ascertain significant ethical concerns, comprehend the viewpoints of stakeholders, establish an ethical framework, and provide suggestions for policies. The incorporation of Explainable AI into different industries has a significant and far-reaching effect on both technology and society. This includes potential benefits such as increased trust and acceptance, adherence to regulations, improved AI development and troubleshooting, ethical AI design, empowerment and equal access, advancements in education and collaboration, changes in skill requirements, and the establishment of new ethical guidelines.

View PDFchevron_right
Explainable AI is indispensable in areas where liability is an issue
Nelson Brochado

2019

Explainable (or interpretable) artificial intelligence (XAI) refers to a collection of techniques used to explain (to humans) or interpret the inner workings and outcomes (e.g. predictions) of black box artificial intelligence (AI) or machine learning (ML) models. XAI can also refer to white box AI or ML models whose inner workings and outcomes (or outputs) can be more easily (with respect to black box ones) understood and trusted by humans .

View PDFchevron_right
Unlocking the Black Box: Explainable Artificial Intelligence (XAI) for Trust and Transparency in AI Systems
Nipuna Thalpage

Journal of digital art & humanities, 2023

Explainable Artificial Intelligence (XAI) has emerged as a critical field in AI research, addressing the lack of transparency and interpretability in complex AI models. This conceptual review explores the significance of XAI in promoting trust and transparency in AI systems. The paper analyzes existing literature on XAI, identifies patterns and gaps, and presents a coherent conceptual framework. Various XAI techniques, such as saliency maps, attention mechanisms, rule-based explanations, and model-agnostic approaches, are discussed to enhance interpretability. The paper highlights the challenges posed by black-box AI models, explores the role of XAI in enhancing trust and transparency, and examines the ethical considerations and responsible deployment of XAI. By promoting transparency and interpretability, this review aims to build trust, encourage accountable AI systems, and contribute to the ongoing discourse on XAI.

View PDFchevron_right
Explainable AI and the philosophy and practice of explanation
Kieron O'Hara

Computer Law & Security Review, 2020

Considerations of the nature of explanation and the law are brought together to argue that computed accounts of AI systems' outputs cannot function on their own as explanations of decisions informed by AI. The important context for this inquiry is set by Article 22(3) of GDPR. The paper looks at the question of what an explanation is from the point of view of the philosophy of science -i.e. it asks not what counts as explanatory in legal terms, or what an AI system might compute using provenance metadata, but rather what explanation as a social practice consists in, arguing that explanation is an illocutionary act, and that it should be considered as a process, not a text. It cannot therefore be computed, although computed accounts of AI systems are likely to be important inputs to the explanatory process.

View PDFchevron_right
Reflections on the data protection compliance of AI systems under the EU AI Act
Dr. Balázs Hohmann

Cogent Social Sciences, 2025

The European Union’s Artificial Intelligence Act (AI Act) establishes a novel regulatory framework for AI systems, with far-reaching implications for data protection compliance. This study critically analyses the regulatory environment created by the Act and its interaction with the General Data Protection Regulation (GDPR), addressing whether it provides clearer compliance pathways or introduces additional burdens for developers. Using comparative legal analysis, the research finds that the AI Act supplements the GDPR through a risk-based approach that subjects high-risk AI systems to specific obligations. Key concerns include algorithmic opacity, bias, and legal uncertainty in profiling and automated decision-making. The analysis shows that the AI Act attempts to mitigate these risks by strengthening requirements on risk assessment, human oversight, and data governance. While the Act broadly aligns with GDPR principles such as transparency, fairness, and accountability, it also introduces new procedural and documentation duties that may increase compliance complexity. Overall, the study concludes that the AI Act constitutes a complementary yet stricter regulatory layer for AI-driven data processing, balancing innovation with fundamental rights protection and requiring joint interpretation with the GDPR to ensure coherent application.

View PDFchevron_right
Explainable AI: Making Machine Learning Decisions Transparent
Adelola Ikumapayi Nathan

Explainable AI (XAI) has emerged as a critical field in artificial intelligence, addressing the "black box" nature of complex machine learning models. This article explores the importance of transparency in AI decision-making, the techniques used to achieve explainability, and the implications for various sectors. We examine the current state of XAI, its applications in healthcare, finance, and other critical areas, and discuss the ethical and regulatory considerations surrounding transparent AI. The paper concludes with an analysis of XAI's contribution to the broader field of AI and its potential future developments.

View PDFchevron_right
Building Trust to AI Systems Through Explainability. Technical and legal perspectives
Michał Araszkiewicz

2019

In this position paper we discuss two perspectives on explainability of AI systems: technical and legal one, and we investigate how the two perspectives should be integrated to develop trust in the AI systems. We consider trust building as a process that should reflected in the design process of AI systems.

View PDFchevron_right

References (124)

  1. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Pro- cessing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation).
  2. 2022. Draft standardisation request to the European Standardisation Organ- isations in support of safe and trustworthy artificial intelligence. https: //ec.europa.eu/docsroom/documents/52376
  3. Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y Lim, and Mohan Kankan- halli. 2018. Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1-18.
  4. Alessandro Achille and Stefano Soatto. 2018. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research 19, 1 (2018), 1947-1980.
  5. Amina Adadi and Mohammed Berrada. 2018. Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE access 6 (2018), 52138- 52160.
  6. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. Advances in neural information processing systems 31 (2018).
  7. Ulrich Aïvodji, Hiromi Arai, Sébastien Gambs, and Satoshi Hara. 2021. Char- acterizing the risk of fairwashing. Advances in Neural Information Processing Systems 34 (2021), 14822-14834.
  8. David Alvarez-Melis and Tommi S Jaakkola. 2018. On the robustness of inter- pretability methods. arXiv preprint arXiv:1806.08049 (2018).
  9. José P Amorim, Pedro H Abreu, João Santos, Marc Cortes, and Victor Vila. 2023. Evaluating the faithfulness of saliency maps in explaining deep learning models using realistic perturbations. Information Processing & Management 60, 2 (2023), 103225.
  10. Elvio Amparore, Alan Perotti, and Paolo Bajardi. 2021. To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods. PeerJ Computer Science 7 (2021), e479.
  11. Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel. 2020. Fairwashing explanations with off-manifold detergent. In International Conference on Machine Learning. PMLR, 314-323.
  12. Daniel W Apley and Jingyu Zhu. 2020. Visualizing the effects of predictor variables in black box supervised learning models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82, 4 (2020), 1059-1086.
  13. Nishanth Arun, Nathan Gaw, Praveer Singh, Ken Chang, Mehak Aggarwal, Bryan Chen, Katharina Hoebel, Sharut Gupta, Jay Patel, Mishka Gidwani, et al. 2020. Assessing the (un) trustworthiness of saliency maps for localizing abnor- malities in medical imaging. arXiv preprint arXiv:2008.02766 (2020).
  14. Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE inter- national conference on automatic face & gesture recognition (FG 2018). IEEE, 59-66.
  15. David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. 2020. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences 117, 48 (2020), 30071-30078.
  16. Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yun- han Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. 2020. Explainable machine learning in deployment. In Proceedings of the 2020 confer- ence on fairness, accountability, and transparency. 648-657.
  17. Sebastian Bordt, Michèle Finck, Eric Raidl, and Ulrike von Luxburg. 2022. Post- Hoc Explanations Fail to Achieve Their Purpose in Adversarial Contexts. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 891-905. https://doi.org/10.1145/3531146.3533153
  18. Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. 2020. Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. In Proceedings of the 25th international conference on intelligent user interfaces. 454-464.
  19. Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1-21.
  20. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accu- racy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77-91.
  21. Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD inter- national conference on knowledge discovery and data mining. 1721-1730.
  22. Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. 2019. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems 32 (2019).
  23. European Commission. 2020. White Paper: On Artificial Intelli- gence -A European Approach to Excellence and Trust. https: //commission.europa.eu/publications/white-paper-artificial-intelligence- european-approach-excellence-and-trust_en
  24. Roberto Confalonieri, Ludovik Coba, Benedikt Wagner, and Tarek R Besold. 2021. A historical perspective of explainable Artificial Intelligence. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 11, 1 (2021), e1391.
  25. US Congress. 2022. Algorithmic Accountability Act of 2022. https://www. congress.gov/bill/117th-congress/senate-bill/3572/text
  26. Rogier Creemers, Graham Webster, and Helen Toner. 2022. Translation: Internet Information Service Algorithmic Recommendation Management Provisions -Effective March 1, 2022. https://digichina.stanford.edu/work/translation- internet-information-service-algorithmic-recommendation-management- provisions-effective-march-1-2022/
  27. Francesco Croce and Matthias Hein. 2019. Sparse and imperceivable adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4724-4732.
  28. Botty Dimanov, Umang Bhatt, Mateja Jamnik, and Adrian Weller. 2020. You Shouldn't Trust Me: Learning Models Which Conceal Unfairness From Multiple Explanation Methods.. In SafeAI@ AAAI.
  29. Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. 2019. Explanations can be manipulated and geometry is to blame. Advances in Neural Information Processing Systems 32 (2019).
  30. Ann-Kathrin Dombrowski, Christopher J Anders, Klaus-Robert Müller, and Pan Kessel. 2022. Towards robust explanations for deep neural networks. Pattern Recognition 121 (2022), 108194.
  31. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of inter- pretable machine learning. arXiv preprint arXiv:1702.08608 (2017).
  32. Upol Ehsan, Samir Passi, Q Vera Liao, Larry Chan, I Lee, Michael Muller, Mark O Riedl, et al. 2021. The who in explainable ai: How ai background shapes percep- tions of ai explanations. arXiv preprint arXiv:2107.13509 (2021).
  33. Malin Eiband, Daniel Buschek, Alexander Kremer, and Heinrich Hussmann. 2019. The impact of placebic explanations on trust in intelligent systems. In Extended abstracts of the 2019 CHI conference on human factors in computing systems. 1-6.
  34. Fabian Eitel, Kerstin Ritter, Alzheimer's Disease Neuroimaging Initiative (ADNI, et al. 2019. Testing the robustness of attribution methods for convolutional neu- ral networks in MRI-based Alzheimer's disease classification. In Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support. Springer, 3-11.
  35. Rosenberg Ekman. 1997. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA.
  36. European Commission. 2018. Artificial Intelligence for Europe. Technical Report COM(2018) 137 final. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/ ?uri=CELEX:52018DC0237&from=EN
  37. European Commission. 2018. Coordinated Plan on Artificial Intelligence. Techni- cal Report COM(2018) 795 final.
  38. European Commission. 2019. Building Trust in Human-Centric Artificial Intelli- gence. Technical Report COM(2019) 168 final. https://eur-lex.europa.eu/legal- content/EN/TXT/HTML/?uri=CELEX:52019DC0168&from=EN
  39. European Commission. 2021. Proposal for a Regulation laying down harmonised rules on Artificial Intelligence and amending certain union legislative acts. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206
  40. Heike Felzmann, Eduard Fosch-Villaronga, Christoph Lutz, and Aurelia Tamò- Larrieux. 2020. Towards transparency by design for artificial intelligence. Science and Engineering Ethics 26, 6 (2020), 3333-3361.
  41. M. Fink and M. Finck. 2022. Reasoned A(I)Administration: Explanation Require- ments in EU Law and the Automation of Public Administration. European Law Review 47, 3 (2022), 376-392.
  42. Samuel G Finlayson, Hyung Won Chung, Isaac S Kohane, and Andrew L Beam. 2018. Adversarial attacks against medical deep learning systems. arXiv preprint arXiv:1804.05296 (2018).
  43. Aaron Fisher, Cynthia Rudin, and Francesca Dominici. 2019. All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously. J. Mach. Learn. Res. 20, 177 (2019), 1-81.
  44. G20. 2019. G20 Ministerial Statement on Trade and Digital Economy. https: //wp.oecd.ai/app/uploads/2021/06/G20-AI-Principles.pdf
  45. Amirata Ghorbani, Abubakar Abid, and James Zou. 2019. Interpretation of neural networks is fragile. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 3681-3688.
  46. Kate Goddard, Abdul Roudsari, and Jeremy C Wyatt. 2012. Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association 19, 1 (2012), 121-127.
  47. Riccardo Guidotti. 2021. Evaluating local explanation methods on ground truth. Artificial Intelligence 291 (2021), 103428.
  48. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2018), 1-42.
  49. David Gunning. 2017. Explainable artificial intelligence (xai). Defense Advanced Research Projects Agency (DARPA), nd Web 2 (2017).
  50. Ronan Hamon, Henrik Junklewitz, Ignacio Sanchez, Gianclaudio Malgieri, and Paul De Hert. 2022. Bridging the Gap Between AI and Explainability in the GDPR: Towards Trustworthiness-by-Design in Automated Decision-Making. IEEE Computational Intelligence Magazine 17, 1 (2022), 72-85. https://doi.org/ 10.1109/MCI.2021.3129960
  51. Peter Hase, Harry Xie, and Mohit Bansal. 2021. The out-of-distribution prob- lem in explainability and search methods for feature importance explanations. Advances in Neural Information Processing Systems 34 (2021), 3650-3666.
  52. Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural net- work robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 (2019).
  53. Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018. Met- rics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608 (2018).
  54. Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M Drucker. 2019. Gamut: A design probe to understand how data scientists un- derstand machine learning models. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1-13.
  55. Andreas Holzinger, Anna Saranti, Christoph Molnar, Przemyslaw Biecek, and Wojciech Samek. 2022. Explainable AI methods-a brief overview. In Interna- tional Workshop on Extending Explainable AI Beyond Deep Models and Classifiers. Springer, 13-38.
  56. The White House. 2022. Blueprint for an AI Bill of Rights. https://www. whitehouse.gov/ostp/ai-bill-of-rights/
  57. Xiyang Hu, Cynthia Rudin, and Margo Seltzer. 2019. Optimal sparse decision trees. Advances in Neural Information Processing Systems 32 (2019).
  58. Isabelle Hupont, Songül Tolan, Hatice Gunes, and Emilia Gómez. 2022. The Landscape of Facial Processing Applications in the Context of the European AI Act and the Development of Trustworthy Systems. Nature Scientific Reports (2022).
  59. Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. arXiv preprint arXiv:1902.10186 (2019).
  60. Yunzhe Jia, James Bailey, Kotagiri Ramamohanarao, Christopher Leckie, and Xingjun Ma. 2020. Exploiting patterns to explain individual predictions. Knowl- edge and Information Systems 62, 3 (2020), 927-950.
  61. Margot E. Kaminski. 2019. The Right to Explanation, Explained. Berkeley Technology Law Journal 34 (2019), 189. https://doi.org/10.15779/Z38TD9N83H
  62. Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. 2020. Interpreting interpretability: under- standing data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1-14.
  63. Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. 2019. The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, 267-280.
  64. Anastasiya Kiseleva. 2021. Making AI's Transparency Transparent: notes on the EU Proposal for the AI Act. https://europeanlawblog.eu/2021/07/29/making- ais-transparency-transparent-notes-on-the-eu-proposal-for-the-ai-act/
  65. Erwan Le Merrer and Gilles Trédan. 2020. Remote explainability faces the bouncer problem. Nature Machine Intelligence 2, 9 (2020), 529-539.
  66. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appro- priate reliance. Human factors 46, 1 (2004), 50-80.
  67. Benjamin Letham, Cynthia Rudin, Tyler H McCormick, and David Madigan. 2015. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. The Annals of Applied Statistics 9, 3 (2015), 1350-1371.
  68. Ariel Levy, Monica Agrawal, Arvind Satyanarayan, and David Sontag. 2021. Assessing the impact of automated suggestions on decision making: Domain experts mediate model errors but take less initiative. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-13.
  69. Gabriel Lima, Nina Grgić-Hlača, Jin Keun Jeong, and Meeyoung Cha. 2022. The Conflict Between Explainable and Accountable Decision-Making Algorithms. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 2103-2113. https://doi.org/10.1145/3531146.3534628
  70. Zachary C Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (2018), 31-57.
  71. Jiachang Liu, Chudi Zhong, Margo Seltzer, and Cynthia Rudin. 2022. Fast Sparse Classification for Generalized Linear and Additive Models. Proceedings of machine learning research 151 (2022), 9304.
  72. Pedro Lopes, Eduardo Silva, Cristiana Braga, Tiago Oliveira, and Luís Rosado. 2022. XAI Systems Evaluation: A Review of Human and Computer-Centred Methods. Applied Sciences 12, 19 (2022), 9423.
  73. Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017).
  74. Gianclaudio Malgieri and Giovanni Comandé. 2017. Why a right to legibility of automated decision-making exists in the general data protection regulation. International Data Privacy Law 7, 4 (2017), 243-265.
  75. Gabriele Mazzini and Salvatore Scalzo. 2022. The Proposal for the Artificial Intelligence Act: Considerations around Some Key Concepts. La via europea per l'Intelligenza artificiale. Atti del Convegno del Progetto Dottorale di Alta Formazione in Scienze Giuridiche-Ca'Foscari Venezia, 25-26 novembre 2021 (2022). https://doi.org/10.2139/ssrn.4098809
  76. Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence 267 (2019), 1-38.
  77. Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547 (2017).
  78. Sina Mohseni, Haotao Wang, Chaowei Xiao, Zhiding Yu, Zhangyang Wang, and Jay Yadawa. 2021. Taxonomy of Machine Learning Safety: A Survey and Primer. ACM Computing Surveys (CSUR) (2021).
  79. Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. 2020. Interpretable machine learning-a brief history, state-of-the-art and challenges. In Joint Eu- ropean Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 417-431.
  80. Johanna D Moore and William R Swartout. 1988. Explanation in expert sys- temss: A survey. Technical Report. UNIVERSITY OF SOUTHERN CALIFORNIA MARINA DEL REY INFORMATION SCIENCES INST.
  81. Jan Nikolas Morshuis, Sergios Gatidis, Matthias Hein, and Christian F Baumgart- ner. 2022. Adversarial Robustness of MR Image Reconstruction Under Realistic Perturbations. In International Workshop on Machine Learning for Medical Image Reconstruction. Springer, 24-33.
  82. W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. 2019. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences 116, 44 (2019), 22071-22080.
  83. Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yas- min Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. 2022. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. arXiv preprint arXiv:2201.08164 (2022).
  84. Ian E Nielsen, Dimah Dera, Ghulam Rasool, Ravi P Ramachandran, and Nid- hal Carla Bouaynaya. 2022. Robust explainability: A tutorial on gradient-based attribution methods for deep neural networks. IEEE Signal Processing Magazine 39, 4 (2022), 73-84.
  85. Mahsan Nourani, Joanie King, and Eric Ragan. 2020. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 112-121.
  86. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447-453.
  87. OECD. 2019. Recommendation of the Council on Artificial Intelligence OECD/LEGAL/0449.
  88. High Level Expert Group on Artificial Intelligence. 2019. Ethics Guidelines for Trustworthy AI.
  89. Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pe- dreschi, Alan Perotti, and Salvatore Rinzivillo. 2023. Co-design of human- centered, explainable AI for clinical decision support. ACM Transactions on Interactive Intelligent Systems (2023).
  90. Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi. 2022. Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems. In CHI Conference on Human Factors in Computing Systems. 1-9.
  91. Andrea Papenmeier, Dagmar Kern, Gwenn Englebienne, and Christin Seifert. 2022. It's Complicated: The Relationship between User Trust, Model Accuracy and Explanations in AI. ACM Transactions on Computer-Human Interaction (TOCHI) 29, 4 (2022), 1-33.
  92. Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. 2019. Learning to deceive with attention-based explanations. arXiv preprint arXiv:1909.07913 (2019).
  93. Luyu Qiu, Yi Yang, Caleb Chen Cao, Jing Liu, Yueyuan Zheng, Hilary Hei Ting Ngai, Janet Hsiao, and Lei Chen. 2021. Resisting out-of-distribution data problem in perturbation of xai. arXiv preprint arXiv:2107.14000 (2021).
  94. Luyu Qiu, Yi Yang, Caleb Chen Cao, Yueyuan Zheng, Hilary Ngai, Janet Hsiao, and Lei Chen. 2022. Generating Perturbation-based Explanations with Robust- ness to Out-of-Distribution Data. In Proceedings of the ACM Web Conference 2022. 3594-3605.
  95. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135-1144.
  96. Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. 2022. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys 16 (2022), 1-85.
  97. Mirka Saarela and Lilia Geogieva. 2022. Robustness, Stability, and Fidelity of Explanations for a Deep Skin Cancer Classification Model. Applied Sciences 12, 19 (2022), 9545.
  98. Rabia Saleem, Bo Yuan, Fatih Kurugollu, Ashiq Anjum, and Lu Liu. 2022. Ex- plaining Deep Neural Networks: A Survey on the Global Interpretation Methods. Neurocomputing (2022).
  99. Philipp Schmidt and Felix Biessmann. 2020. Calibrating human-ai collaboration: Impact of risk, ambiguity and transparency on algorithmic bias. In Interna- tional Cross-Domain Conference for Machine Learning and Knowledge Extraction. Springer, 431-449.
  100. Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan- tam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE inter- national conference on computer vision. 618-626.
  101. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In International conference on machine learning. PMLR, 3145-3153.
  102. Dylan Slack, Anna Hilgard, Himabindu Lakkaraju, and Sameer Singh. 2021. Counterfactual explanations can be manipulated. Advances in Neural Information Processing Systems 34 (2021), 62-75.
  103. Dylan Slack, Anna Hilgard, Sameer Singh, and Himabindu Lakkaraju. 2021. Reliable post hoc explanations: Modeling uncertainty in explainability. Advances in Neural Information Processing Systems 34 (2021), 9391-9404.
  104. Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 180-186.
  105. Francesco Sovrano, Salvatore Sapienza, Monica Palmirani, and Fabio Vitali. 2022. Metrics, Explainability and the European AI Act Proposal. J 5, 1 (March 2022), 126-138. https://doi.org/10.3390/j5010010
  106. Timo Speith. 2022. A review of taxonomies of explainable artificial intelli- gence (XAI) methods. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 2239-2250.
  107. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning. PMLR, 3319- 3328.
  108. Harini Suresh, Steven R Gomez, Kevin K Nam, and Arvind Satyanarayan. 2021. Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-16.
  109. Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. Advances in Neural Information Processing Systems 32 (2019).
  110. Kaya ter Burg and Heysem Kaya. 2022. Comparing Approaches for Explaining DNN-Based Facial Expression Classifications. Algorithms 15, 10 (2022), 367.
  111. Media UK Secretary of State for Digital, Culture and Sport. 2022. AI Regulation Policy Paper. https://www.gov.uk/government/publications/establishing-a- pro-innovation-approach-to-regulating-ai/establishing-a-pro-innovation- approach-to-regulating-ai-policy-statement
  112. UNESCO. 2022. Recommendation on the Ethics of Artificial Intelligence. https: //unesdoc.unesco.org/ark:/48223/pf0000381137
  113. Van Roy V, Rossetti F, Perset K, and Galindo-Romero L. 2021. AI Watch -National strategies on Artificial Intelligence: A European perspective, 2021 edition. Scien- tific analysis or review, Policy assessment, Country report KJ-NA-30745-EN-N (online). Luxembourg (Luxembourg). https://doi.org/10.2760/069178(online)
  114. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
  115. Giorgio Visani, Enrico Bagli, Federico Chesani, Alessandro Poluzzi, and Davide Capuzzo. 2022. Statistical stability indices for LIME: Obtaining reliable explana- tions for machine learning models. Journal of the Operational Research Society 73, 1 (2022), 91-101.
  116. Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. 2017. Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation. International Data Privacy Law 7, 2 (2017), 776-99.
  117. Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces. 318-328.
  118. David S Watson, Limor Gultchin, Ankur Taly, and Luciano Floridi. 2021. Local explanations via necessity and sufficiency: Unifying theory and practice. In Uncertainty in Artificial Intelligence. PMLR, 1382-1392.
  119. Magdalena Wischnewski, Nicole Krämer, and Emmanuel Müller. 2023. Measur- ing and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 755, 16 pages. https://doi.org/10.1145/3544548.3581197
  120. Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convo- lutional networks. In European conference on computer vision. Springer, 818-833.
  121. Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020. Effect of Confi- dence and Explanation on Accuracy and Trust Calibration in AI-Assisted Deci- sion Making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* '20). Association for Computing Ma- chinery, New York, NY, USA, 295-305. https://doi.org/10.1145/3351095.3372852
  122. Yujia Zhang, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell. 2019. " Why Should You Trust My Explanation?" Understanding Uncertainty in LIME Explanations. arXiv preprint arXiv:1904.12991 (2019).
  123. Jianlong Zhou, Amir H Gandomi, Fang Chen, and Andreas Holzinger. 2021. Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics 10, 5 (2021), 593.
  124. Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. 2022. Do feature attribution methods correctly attribute features?. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 9623-9633.

Related papers

Unlocking the Black Box: Analysing the EU Artificial Intelligence Act's Framework for Explainability in AI
Georgios Pavlidis

Law Innovation and Technology, Vol. 16 No. 1, pp. 293-308, 2024

The lack of explainability of Artificial Intelligence (AI) is one of the first obstacles that the industry and regulators must overcome to mitigate the risks associated with the technology. The need for 'eXplainable AI' (XAI) is evident in fields where accountability, ethics and fairness are critical, such as healthcare, credit scoring, policing and the criminal justice system. At the EU level, the notion of explainability is one of the fundamental principles that underpin the AI Act, though the exact XAI techniques and requirements are still to be determined and tested in practice. This paper explores various approaches and techniques that promise to advance XAI, as well as the challenges of implementing the principle of explainability in AI governance and policies. Finally, the paper examines the integration of XAI into EU law, emphasising the issues of standard setting, oversight, and enforcement.

View PDFchevron_right
Explainability in AI Policies: A Critical Review of Communications, Reports, Regulations, and Standards in the EU, US, and UK
Luca Nannini

2023 ACM Conference on Fairness, Accountability, and Transparency

Public attention towards explainability of artificial intelligence (AI) systems has been rising in recent years to offer methodologies for human oversight. This has translated into the proliferation of research outputs, such as from Explainable AI, to enhance transparency and control for system debugging and monitoring, and intelligibility of system process and output for user services. Yet, such outputs are difficult to adopt on a practical level due to a lack of a common regulatory baseline, and the contextual nature of explanations. Governmental policies are now attempting to tackle such exigence, however it remains unclear to what extent published communications, regulations, and standards adopt an informed perspective to support research, industry, and civil interests. In this study, we perform the first thematic and gap analysis of this plethora of policies and standards on explainability in the EU, US, and UK. Through a rigorous survey of policy documents, we first contribute an overview of governmental regulatory trajectories within AI explainability and its sociotechnical impacts. We find that policies are often informed by coarse notions and requirements for explanations. This might be due to the willingness to conciliate explanations foremost as a risk management tool for AI oversight, but also due to the lack of a consensus on what constitutes a valid algorithmic explanation, and how feasible the implementation and deployment of such explanations are across stakeholders of an organization. Informed by AI explainability research, we then conduct a gap analysis of existing policies, which leads us to formulate a set of recommendations on how to address explainability in regulations for AI systems, especially discussing the definition, feasibility, and usability of explanations, as well as allocating accountability to explanation providers.

View PDFchevron_right
Accountability of AI Under the Law: The Role of Explanation
David Weinberger

SSRN Electronic Journal

The ubiquity of systems using artificial intelligence or "AI" has brought increasing attention to how those systems should be regulated. The choice of how to regulate AI systems will require care. AI systems have the potential to synthesize large amounts of data, allowing for greater levels of personalization and precision than ever before-applications range from clinical decision support to autonomous driving and predictive policing. That said, our AIs continue to lag in common sense reasoning [McCarthy, 1960], and thus there exist legitimate concerns about the intentional and unintentional negative consequences of AI systems [Bostrom, 2003, Amodei et al., 2016, Sculley et al., 2014]. How can we take advantage of what AI systems have to o↵er, while also holding them accountable? In this work, we focus on one tool: explanation. Questions about a legal right to explanation from AI systems was recently debated in the EU General Data Protection Regulation [Goodman and Flaxman, 2016, Wachter et al., 2017a], and thus thinking carefully about when and how explanation from AI systems might improve accountability is timely. Good choices about when to demand explanation can help prevent negative consequences from AI systems, while poor choices may not only fail to hold AI systems accountable but also hamper the development of much-needed beneficial AI systems. Below, we briefly review current societal, moral, and legal norms around explanation, and then focus on the di↵erent contexts under which explanation is currently required under the law. We find that there exists great variation around when explanation is demanded, but there also exist important consistencies: when demanding explanation from humans, what we typically want to know is whether and how certain input factors a↵ected the final decision or outcome. These consistencies allow us to list the technical considerations that must be considered if we desired AI systems that could provide kinds of explanations that are currently required of humans under the law. Contrary to popular wisdom of AI systems as indecipherable black boxes, we find that this level of explanation should generally be technically feasible but may sometimes be practically onerous-there are certain aspects of explanation that may be simple for humans to provide but challenging for AI systems, and vice versa. As an interdisciplinary team of legal scholars, computer scientists, and cognitive scientists, we recommend that for the present, AI systems can and should be held to a similar standard of explanation as humans currently are; in the future we may wish to hold an AI to a di↵erent standard.

View PDFchevron_right
The AI Act proposal a new right to technical interpretability
Chiara Gallese

2023

The debate about the concept of the so called right to explanation in AI is the subject of a wealth of literature. It has focused, in the legal scholarship, on art. 22 GDPR and, in the technical scholarship, on techniques that help explain the output of a certain model (XAI). The purpose of this work is to investigate if the new provisions introduced by the proposal for a Regulation laying down harmonised rules on artificial intelligence (AI Act), in combination with Convention 108 plus and GDPR, are enough to indicate the existence of a right to technical explainability in the EU legal framework and, if not, whether the EU should include it in its current legislation. This is a preliminary work submitted to the online event organised by the Information Society Law Center and it will be later developed into a full paper.

View PDFchevron_right
Operationalizing Explainable AI in the EU Regulatory Ecosystem
Luca Nannini

IEEE intelligent systems, 2024

The European Union's regulatory ecosystem presents challenges balancing legal and sociotechnical drivers for explainable AI systems. Core tensions emerge on dimensions of oversight, user needs and litigation. This paper maps provisions on algorithmic transparency and explainability across major EU data, AI, and platform policies using qualitative analysis. We characterize involved stakeholders and organizational implementation targets. Constraints become visible between useful transparency for accountability versus confidentiality protections. Through an AI hiring system example, we explore complications operationalizing explainability. Customization is required satisfying explainability desires within confidentiality and proportionality bounds. Findings advise technologists on prudent eXplainable AI technique selection given multi-dimensional tensions. Outcomes recommend policymakers balance worthy transparency goals with cohesive legislation enabling equitable dispute resolution.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved