Human-Centered Explainability

Overview

Chapter 77 asked whether an explanation tells the truth about a model. This chapter asks whether it helps the person who receives it. The two questions are independent. A perfectly faithful explanation can be incomprehensible, irrelevant to the decision at hand or misread; a comprehensible explanation can lead people to accept advice they should have rejected. Because explanations in high-stakes settings are given to people who act on them (a loan officer, a clinician, a caseworker, an applicant), the value of an explanation lies in what it does to their understanding and their decisions.

Human-centered explainability places the people who use and are affected by an AI system at the center of explanation design. It starts from their questions and goals, designs explanations for the context in which they work, and evaluates explanations by their effect on understanding, reliance and outcomes rather than by their technical properties alone. The chapter first explains why the algorithm-centered view is insufficient, then presents methods for eliciting explanation needs, the empirical evidence on how explanations affect human decisions, designs that promote appropriate reliance, the relation between explanations and fairness judgments, and a method for designing and evaluating explanations. It builds on the audiences and evaluation levels of Chapter 70; Chapter 79 treats explanation as a legal right.

Learning objectives:

• Explain why explainability must be designed and evaluated for specific people, tasks and contexts.
• Use the categories of the XAI question bank to elicit and structure explanation needs.
• Define appropriate reliance, overreliance and underreliance and compute them from decision records.
• Summarize the empirical evidence on how explanations affect trust, reliance, team performance and fairness judgments.
• Describe design interventions that reduce overreliance and their limitations, including unequal benefits.
• Apply a human-centered method to design and evaluate explanations, with reliance measured by group.

Key questions

• What do the people who use or are affected by an AI system actually want to know about it, and how can that be found out?
• Do explanations make human-AI decisions better, and under which conditions do they make them worse?
• What does it mean for a person to rely on AI advice appropriately, and how can it be measured?
• How do explanations shape people's judgments of whether a system is fair, and their own fairness as decision makers?
• How should explanations be evaluated so that a claim that they "help users" is evidence rather than assumption?

Definitions and terminology

Human-centered explainable AI. An approach that puts the human at the center of technology design and develops an understanding of who the human is by considering values, interpersonal dynamics and the socially situated nature of AI systems (Ehsan and Riedl, 2020).

Explainability need. A question a user wants answered about an AI system, its data, its outputs or a specific prediction, together with the goal behind it (Liao, Gruen and Miller, 2020).

AI-assisted decision making. A setting in which an AI system makes a recommendation and a human makes the final decision, accepting or overriding the recommendation (Zhang, Liao and Bellamy, 2020).

Reliance. A person's behavior of following AI advice. Overreliance is accepting AI advice when it is wrong; underreliance is rejecting it when it is right; appropriate reliance is following correct advice and overriding incorrect advice. Lee and See (2004) frame the problem of automation as people failing to rely upon it appropriately.

Automation bias. In the wording of the EU AI Act, the possible tendency of automatically relying or over-relying on the output produced by an AI system (Article 14(4)(b) of Regulation (EU) 2024/1689).

Complementary performance. Team performance that is higher than that of either the human or the AI working alone (Bansal et al., 2021).

Cognitive forcing function. An intervention that compels people to engage analytically with a decision, for example by asking for their own judgment before showing the AI's recommendation (Buçinca, Malaya and Gajos, 2021).

Social transparency. A perspective that incorporates the socio-organizational context, such as how other people used and judged the system's outputs, into explaining AI-mediated decisions (Ehsan et al., 2021).

From algorithm-centered to human-centered explanation

Why the algorithm-centered view is insufficient

Much of the explainability literature expresses an algorithm-centric view, relying, in a phrase Liao, Gruen and Miller (2020) quote from earlier work, on researchers' intuition of what constitutes a good explanation. The authors note that explanations are often demanded by lay users who lack deep technical understanding of AI but hold preconceptions of what constitutes a useful explanation for decisions in a familiar domain. A list of the features with the highest weights, the most common output of explanation algorithms, may not be what such users need. Chapter 70 summarizes the social-science findings behind this concern: people want contrastive, selected, causal and socially delivered explanations (Miller, 2019).

Liao and Varshney (2021) identify three roles that human-centered approaches play for explainability technologies: to drive technical choices by users' explainability needs, to uncover pitfalls of existing methods and inform new ones, and to provide conceptual frameworks for human-compatible explanation. Ehsan and Riedl (2020) go further and advocate a reflective sociotechnical approach, informed by critical technical practice and supplemented with value-sensitive and participatory design, that asks who the human is beyond one-to-one interaction with a computer and helps designers see their own intellectual blind spots.

Source: validant.ai.

Explainability needs

The XAI question bank

Liao, Gruen and Miller (2020) interviewed 20 user experience and design practitioners working on various AI products, using as a study probe an algorithm-informed XAI question bank in which explainability needs are represented as prototypical questions users might ask about an AI system. Building on intelligibility types from earlier work in context-aware computing (how, why, why not, what if), they added two question types that existing techniques can address, "how to be that" (what minimal changes would yield a different prediction) and "how to still be this" (what changes are permitted while keeping the same prediction), and three descriptive types about the model's input (training data), output and performance. The resulting nine categories, extended after the interviews with new questions and an "others" category, structure the needs that an explanation product may have to meet.

Category	Typical question	Typical answer
Input	What data was the system trained on?	Data documentation (Chapter 107)
Output	What does the output mean and how should it be used?	Output description, instructions for use
Performance	How accurate is the system, and for whom?	Performance and group metrics (Part IV)
How (global)	How does the system make decisions overall?	Global explanations (Chapter 72)
Why	Why is this instance given this prediction?	Feature attributions, examples (Chapter 73)
Why not	Why not a different prediction?	Contrastive features, counterfactual examples
What if	What would the system predict if the input changed?	Interactive what-if inspection
How to be that	What must change to get a different prediction?	Counterfactuals and recourse (Chapter 74)
How to still be this	What may change without changing the prediction?	Scoped rules, prototypes

Source: categories from Liao, Gruen and Miller (2020); example questions other than the leading "Why" question and the mapping to chapters are the compendium's synthesis.

Two observations from the table matter for fairness. First, three of the nine categories (input, output, performance) are answered by documentation rather than by an explanation algorithm, which means that "explainability" includes reporting performance by group. Second, the "how to be that" question is the question of recourse, whose distribution across groups is itself a fairness criterion (Chapters 74 and 76).

Why people want explanations

The interviews revealed several goals behind explanation requests (Liao, Gruen and Miller, 2020). In AI-assisted decision making, explanations were most often sought to gain further insights or evidence: when output is disagreeable, unexpected or unfamiliar, explanations help people assess the AI's judgment; when they agree with it, explanations can increase confidence or suggest causes for follow-up actions, and some users saw them as a way to mitigate their own decision biases. A second motivation was to evaluate the system's capability, both to decide on adoption and to stay aware of its limitations; the authors stress that appropriating trust should be distinguished from enhancing trust. A third was to adapt one's interaction to use the AI better, for example by focusing attention on cases where it is likely to err or by correcting its inputs. Finally, informants described an ethical responsibility to provide explanations. The authors conclude that the motivation, and the downstream action it serves, should guide the selection of explanation methods: if the goal is to mitigate biases, users may want to see both positive and negative evidence.

Tensions in practice

The same study documents why explainability is hard to deliver. Practitioners had to balance it against other product goals, such as protecting proprietary data and avoiding legal or marketing concerns from exposing the algorithm. Explanations could disrupt workflows (a clinician-facing product had to avoid breaking doctors' routines), or expose rationales that users found wrong, oversimplified or outdated, making a product, in one informant's words, a victim of trying to be too transparent. And users' questions often concerned the whole system and its place in their work (what to do with an output, how to incorporate it into other processes) rather than one model and one prediction. These tensions are not reasons to omit explanations, but they explain why explanation is a design problem that requires teamwork between designers, data scientists and the people who use the system.

What explanations do to human decisions

Trust and appropriate reliance

The goal of explanation in AI-assisted decision making is not more trust but better-calibrated trust. Lee and See (2004) review trust in automation from organizational, sociological, interpersonal, psychological and neurological perspectives and argue that trust guides reliance when complexity and unanticipated situations make a complete understanding of the automation impractical, which describes most uses of machine learning. A person who trusts a system too much accepts its errors; a person who trusts it too little discards its correct advice. The standard of success is therefore appropriate reliance, which can be measured from decision records if the correctness of the AI advice is known for each case.

Explanations raise acceptance, right or wrong

Controlled experiments repeatedly find that explanations change how often people follow AI advice without improving their ability to tell good advice from bad. Bansal et al. (2021) ran mixed-method user studies on three data sets (sentiment of beer and book reviews, and logical reasoning questions from the LSAT) in which an AI with accuracy comparable to humans assisted participants. They observed complementary improvements from AI assistance, but explanations did not increase them; rather, explanations increased the chance that people accepted the AI's recommendation regardless of its correctness.

Zhang, Liao and Bellamy (2020) found that showing a confidence score helped people calibrate their trust, but that calibration alone was not enough to improve joint decisions, which also depended on whether humans could bring unique knowledge to complement the AI's errors; they highlighted problems in using local explanations to calibrate trust. Wang and Yin (2021) compared four common model-agnostic explanation methods in recidivism prediction and forest cover prediction and found that the effects depended on participants' domain expertise: many explanations satisfied none of the desired properties (improving understanding, helping recognize uncertainty, supporting calibrated trust) in the task where people had little expertise, and counterfactual explanations, though considered to resemble how humans explain, did not seem to improve calibrated trust.

Transparency can overload

Making a model more transparent does not necessarily help. Poursabzi-Sangdeh et al. (2021) ran a sequence of pre-registered experiments with 3,800 participants who predicted apartment prices in a New York City neighborhood with the help of models that made the same predictions but were presented with two or eight features and as clear (internals shown) or black box. Participants who saw a clear model with few features could better simulate its predictions, but they did not follow its predictions more closely, and participants who saw a clear model were less able to detect and correct the model's sizable mistakes, seemingly because of information overload. The authors draw the lesson that interpretable models must be tested, not assumed, to achieve their intended effects.

Experts misread their own tools

Overreliance is not a problem of lay users only. Kaur et al. (2020) studied how data scientists use two interpretability tools, the InterpretML implementation of generalized additive models and the SHAP package, through a contextual inquiry with 11 participants and a survey of 197. Their results indicate that data scientists over-trust and misuse interpretability tools, and few participants could accurately describe the visualizations the tools produce. For fairness work, where the people who run explanation tools are usually data scientists, this finding means that an attribution plot in an assessment report is only as reliable as its reader's understanding of what it shows (Chapter 73).

Explanations can be designed to mislead

Because explanations shape trust, they can be used to shape it wrongly. Lakkaraju and Bastani (2020) propose a theoretical framework for understanding and generating misleading explanations and show in a user study with domain experts that user trust in black-box models can be manipulated through misleading explanations. Together with the technical manipulation results of Chapter 77, this means that an explanation can be unfaithful by construction and persuasive by design.

Worked example: measuring reliance

Constructed for illustration. A benefits agency pilots an AI system that recommends approval or refusal of applications; caseworkers make the final decision. On 500 pilot cases with known correct outcomes, the AI is right on 400 (accuracy 0.80). Three interface conditions are compared, each on the same cases: the recommendation alone, the recommendation with a feature-based explanation, and the explanation with a cognitive forcing design in which the caseworker records a provisional decision before seeing the AI's recommendation. Because the decision is binary, overriding a wrong recommendation yields the correct decision.

Overreliance is the share of wrong recommendations that are followed, underreliance the share of right recommendations that are overridden, and team accuracy the share of final decisions that are correct:

Condition	Followed when right	Followed when wrong	Overreliance	Underreliance	Team accuracy
Recommendation only	340	55	0.55	0.15	0.770
With explanation	372	78	0.78	0.07	0.788
Explanation and forced first judgment	360	35	0.35	0.10	0.850

Source: constructed example; the AI alone has accuracy 0.800.

Source: validant.ai.

Adding the explanation raises overall agreement with the AI from 0.79 to 0.90 and reduces underreliance, which a product team might read as success. But 23 of the 55 additional acceptances are wrong recommendations, and overreliance rises from 0.55 to 0.78: the caseworkers now accept 78 of the 100 wrong recommendations, and team accuracy (0.788) stays below that of the AI alone (0.800). Only the forcing design achieves complementary performance (0.850), by cutting the acceptance of wrong recommendations to less than half of the explanation condition (35 against 78) while keeping most correct ones. Agreement with the AI, the metric most often reported, cannot distinguish the second condition from a good one; the split by correctness can.

Designing for appropriate reliance

Cognitive forcing

Buçinca, Malaya and Gajos (2021) argue, drawing on dual-process theories of cognition, that people rarely engage analytically with each recommendation and explanation and instead develop general heuristics about when to follow the AI. Building on research in medical decision making, they designed three cognitive forcing interventions to compel more thoughtful engagement and compared them in an experiment with 199 participants against two simple explainable AI designs and a no-AI baseline. Cognitive forcing significantly reduced overreliance compared with the simple explainable AI designs, but participants gave the least favorable subjective ratings to the designs that reduced overreliance the most. The authors also audited their interventions for unequal effects and found that, on average, the interventions benefited participants higher in Need for Cognition (the motivation to engage in effortful thinking) more.

The last finding carries a fairness lesson of its own. An intervention that improves decisions on average can widen differences between the people who use it, and a design evaluated only on average effects can introduce an inequality it was not meant to create.

The cost of checking

Vasconcelos et al. (2023) argue that people strategically choose whether to engage with an explanation, weighing the costs and benefits of doing the task themselves against those of relying on the AI. In five studies with 731 participants who worked with a simulated AI on a maze task, costs such as task difficulty and explanation difficulty, and benefits such as monetary compensation, affected overreliance, and in some conditions explanations reduced it. They suggest that some null effects in the literature may arise because the explanations did not sufficiently reduce the cost of verifying the AI's prediction. For design, this shifts the question from "is there an explanation?" to "does the explanation make checking the recommendation cheaper than accepting it?"

Confidence and social context

Two further levers come from the studies above and from work on the organizational context of decisions. Confidence information helped people calibrate trust in the study of Zhang, Liao and Bellamy (2020), though it did not by itself improve decisions. Ehsan et al. (2021) introduce social transparency, which incorporates the socio-organizational context into explanations, for example how colleagues acted on similar outputs; from interviews with 29 AI users and practitioners they develop a framework in which social transparency can calibrate trust, improve decision making, support collective organizational action and contribute to a more holistic explainability.

Explanations, fairness perceptions and fair decisions

How explanations shape fairness judgments

Explanations also influence whether people judge a system to be fair. Dodge et al. (2019) studied four types of programmatically generated explanations with more than 160 crowd workers and found that certain explanations were considered inherently less fair while others enhanced confidence in the fairness of the algorithm; that different fairness problems, model-wide issues versus case-specific discrepancies, may be exposed more effectively by different explanation styles; and that individual differences, including prior positions on algorithmic fairness, affected how people reacted. Binns et al. (2018) conducted three experiments on perceptions of justice in algorithmic decisions such as loan approvals, insurance quotes and CV filtering. Dimensions of justice familiar from human decision making were engaged by algorithmic decisions too, with concerns about arbitrariness, generalization and dignity; explanation styles mattered to justice perceptions mainly when participants saw several styles, and the authors suggest there may be no single best way to explain algorithmic decisions.

Explanations and the fairness of human decisions

Whether explanations make human-AI decisions fairer is a different question from whether they change perceptions. Schoeffer, De-Arteaga and Kühl (2024) studied the effect of feature-based explanations on the distributive fairness of AI-assisted decisions in predicting occupations from short textual biographies. Explanations influenced fairness perceptions, which in turn related to people's tendency to follow AI recommendations, but did not enable them to discern correct from incorrect recommendations. Depending on which features were highlighted, explanations could foster or hinder distributive fairness: features that were task-irrelevant and evidently associated with gender prompted overrides of stereotype-aligned recommendations, while features that appeared task-relevant induced reliance that reinforced stereotype-aligned errors. The authors conclude that feature-based explanations are not a reliable mechanism to improve distributive fairness.

Worked example: reliance by group

Constructed for illustration, continuing the pilot. Splitting the explanation condition by applicant group shows where the accepted errors fall. Group a accounts for 320 cases (260 correct and 60 incorrect recommendations), group b for 180 cases (140 correct and 40 incorrect).

Applicant group	Cases	AI accuracy	Overreliance	Underreliance	Team accuracy
Group a	320	0.813	0.70 (42 of 60)	0.077 (20 of 260)	0.806
Group b	180	0.778	0.90 (36 of 40)	0.057 (8 of 140)	0.756

Source: constructed example; the group totals add up to the explanation condition of the previous table.

The AI is somewhat less accurate for group b, and the caseworkers correct its errors less often for group b: 9 of 10 wrong recommendations about group b applicants are accepted, against 7 of 10 for group a. The human-AI team therefore widens the accuracy gap between the groups from 0.035 (AI alone) to 0.051 (team). A fairness assessment that measured only the AI's outputs would miss the part of the disparity that arises in the human step; an assessment that measured only the final decisions would not see that the explanation design is where it arises. The small number of wrong recommendations per group (40 for group b) also means that the overreliance rates come with wide intervals and that a real pilot needs enough incorrect recommendations per group to estimate them (Chapter 43).

Evaluating human-centered explanation

Chapter 70 introduces the three levels of evaluation proposed by Doshi-Velez and Kim (2017): application-grounded (real users, real tasks), human-grounded (real users, simplified tasks) and functionally-grounded (no users, proxy tasks). Human-centered evaluation occupies the first two levels. Hoffman, Mueller, Klein and Litman (2018) discuss what such evaluations should measure and name six aspects: the goodness of explanations, whether users are satisfied by them, how well users understand the AI system, how curiosity motivates the search for explanations, whether users' trust and reliance are appropriate, and how the human-AI work system performs. In the Co-12 scheme of Nauta et al. (2023), the user-facing properties are context (relevance to the user's needs), coherence (accordance with prior knowledge) and controllability (whether the user can influence the explanation); their review found that only one in five papers introducing an explanation method evaluated it with users.

Question	Measure	Design note
Do users understand the system?	Forward and counterfactual simulation accuracy	Compare with a no-explanation control
Do users rely appropriately?	Overreliance, underreliance	Requires cases with known correct answers, including AI errors
Do decisions improve?	Team accuracy against AI alone and human alone	Complementary performance is the target
Is the effect equal across people?	All measures by user group and by subject group	Plan sample sizes per group in advance
Do users find it useful?	Satisfaction, perceived usefulness	Report, but do not treat as evidence of benefit
Does it hold in the workflow?	Field measures in deployment	Simplified tasks may not transfer

Source: compiled from Doshi-Velez and Kim (2017), Hoffman et al. (2018), Bansal et al. (2021) and Buçinca et al. (2021); the arrangement is the compendium's.

Three design requirements follow from the evidence. The evaluation needs AI errors in its test cases, since overreliance cannot be measured on cases where the AI is right. It needs a control condition without explanation, because many effects of explanations appear only by comparison. And it needs the realistic users and stakes of the application wherever possible: the experts in Kaur et al. (2020) and the lay participants in Poursabzi-Sangdeh et al. (2021) both failed in ways their designers did not foresee.

Legal context

The Act's language aligns closely with the empirical findings of this chapter. Article 14(4)(b) names automation bias as a risk that oversight must address, which is the overreliance documented in the studies above; an explanation that increases acceptance of wrong recommendations works against this provision even if it satisfies Article 13. The provisions are framed as capabilities that the provider must enable, not as results that must be demonstrated; the methods of this chapter are how a provider or deployer can show that oversight personnel are in fact enabled to interpret outputs and to resist automation bias. Chapter 79 examines Article 86 and the related rights under data protection law.

A method for human-centered explanation design

The elements of the chapter combine into a method for any system whose outputs are explained to people.

1. Identify the people and decisions. List the roles that receive explanations (Chapter 70), the decisions each makes and the actions the explanation should support.
2. Elicit explainability needs. Use the question bank categories with representative users to find which questions matter, why they are asked and what action follows from the answer.
3. Select faithful methods. For each need, choose an explanation method or documentation item that answers it, and confirm its fidelity and robustness for the relevant groups (Chapter 77).
4. Design for the workflow. Decide where and when explanations appear, how uncertainty is shown, and whether a forcing design or a provisional own judgment is required for high-stakes cases.
5. Test with people. Run an evaluation with realistic users and cases that include AI errors, a no-explanation control and pre-planned group comparisons; measure understanding, overreliance, underreliance and team accuracy.
6. Check equality of effects. Compare reliance and accuracy across groups of decision subjects and across groups of users (for example by experience or need for cognition).
7. Monitor in deployment. Track override rates and, where outcomes become known, overreliance and underreliance by group, and revisit the design when they drift (Chapter 61).

Competing interpretations and limitations

Laboratory and field. Many findings come from crowd workers on simplified tasks (review sentiment, mazes, apartment prices). They establish that explanations can fail in specific ways but do not say how large the effects are for trained professionals in real workflows. Field evidence is scarce, partly because errors of the AI are rarely known at decision time.

Whose needs count. Human-centered design starts from users' questions, but the users who are easiest to reach (operators and executors) are not the people most affected (decision subjects). An explanation designed around a caseworker's needs may not give an applicant what is needed to contest a decision, and the reflective approach of Ehsan and Riedl (2020) asks designers to consider this explicitly.

Satisfaction versus benefit. Users often prefer explanations that do not improve their decisions and dislike designs that do. Product incentives favor satisfaction and adoption; the evidence favors measuring reliance. The tension is real and should be resolved explicitly, not by default.

Explanations or other levers. Several studies suggest that confidence information, forcing designs or changes to incentives can matter more than the explanation itself. It is an open question in each application whether resources are better spent on explanations or on these other levers.

Persuasion. Because explanations shape trust, the line between explaining and persuading is thin. An explanation optimized for acceptance is a persuasive device, and the manipulation results show that persuasion can be decoupled from truth.

Common questions

If users say an explanation is helpful, is that evidence that it helps?
No. Satisfaction and perceived usefulness are worth recording but are not evidence of better decisions. In the study of Buçinca et al. (2021), the designs that reduced overreliance most were rated least favorably. Measure overreliance, underreliance and team accuracy against known correct outcomes.

Should a transparent, interpretable model always be shown to users in full?
Not necessarily. Poursabzi-Sangdeh et al. (2021) found that showing a clear model made participants less able to detect its sizable mistakes, seemingly because of information overload. How much of the model to show, and to whom, is a design decision to be tested.

Why can explanations make human-AI teams worse?
Because they can raise acceptance of the AI's advice without improving the ability to distinguish correct from incorrect advice (Bansal et al., 2021). When the AI is wrong, more acceptance means more errors.

What is the simplest design change that reduces overreliance?
Asking the decision maker for a provisional judgment before showing the recommendation is one form of cognitive forcing that reduced overreliance in controlled studies (Buçinca et al., 2021). It costs time and user satisfaction, and its benefits may be unequal across users, so it should be tested in context.

Can explanations make human decisions fairer?
Not reliably. Feature-based explanations shifted reliance in ways that could either counter or reinforce stereotype-aligned errors, depending on which features they highlighted (Schoeffer et al., 2024). Fairness of the human-AI system has to be measured by group, not inferred from the presence of explanations.

Does the AI Act require explanations to be tested with users?
It requires providers to enable human overseers to understand the system's limitations, remain aware of automation bias and correctly interpret outputs (Article 14(4)). It does not prescribe user testing, but testing with the people who exercise oversight is the most direct way to show that these capabilities exist.

Conclusions

Explanations are given to people, and their value is measured in what those people understand and decide. The evidence reviewed in this chapter is sobering: explanations frequently raise acceptance of AI advice regardless of whether the advice is right, transparency can overwhelm rather than inform, experts misread their own tools, and explanations can be designed to mislead. None of this makes explanations useless. It makes untested explanations unreliable, and it moves the burden of proof from the explanation algorithm to the human-AI system in which it is used.

The methodological conclusions are concrete. Explanation design starts from the questions and goals of identified users, many of which are answered by documentation of data and performance rather than by explanation algorithms. Explanation designs are evaluated by overreliance, underreliance and team accuracy against known outcomes, with a control condition and with AI errors in the test cases, not by satisfaction or agreement. And because the human step of an AI-assisted decision can create or widen disparities, reliance is measured by group of decision subjects and the benefits of interventions are checked across groups of users. Under these conditions explanations can contribute to what the AI Act calls effective human oversight; without them, they risk becoming an instrument of the automation bias that oversight is meant to prevent.

Open questions

• Which explanation designs sustain appropriate reliance in real professional workflows over long periods of use?
• How can explanations be designed for decision subjects, whose needs (contesting, recourse) differ from those of operators, and evaluated with them?
• How can the unequal benefits of interventions such as cognitive forcing be avoided, so that explanation designs do not widen differences between users?
• How large must a pilot be to estimate overreliance by group with useful precision, given that AI errors are the minority of cases?
• Where is the boundary between legitimate explanation and persuasion, and how should it be policed in high-stakes uses?
• How do the findings for feature-based explanations transfer to natural-language explanations produced by generative systems (Chapter 80)?

References

• Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1 - 16). ACM. https://doi.org/10.1145/3411764.3445717
• Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018). 'It's reducing a human being to a percentage': Perceptions of justice in algorithmic decisions. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1 - 14). ACM. https://doi.org/10.1145/3173574.3173951
• Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1 - 21. https://doi.org/10.1145/3449287
• Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019). Explaining models: An empirical study of how explanations impact fairness judgment. In Proceedings of the 24th International Conference on Intelligent User Interfaces (pp. 275 - 285). ACM. https://doi.org/10.1145/3301275.3302310
• Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning (arXiv:1702.08608). arXiv. https://arxiv.org/abs/1702.08608
• Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021). Expanding explainability: Towards social transparency in AI systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1 - 19). ACM. https://doi.org/10.1145/3411764.3445188
• Ehsan, U., & Riedl, M. O. (2020). Human-centered explainable AI: Towards a reflective sociotechnical approach. In HCI International 2020: Late breaking papers: Multimodality and intelligence (Lecture Notes in Computer Science, pp. 449 - 466). Springer. https://doi.org/10.1007/978-3-030-60117-1_33
• Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018). Metrics for explainable AI: Challenges and prospects (arXiv:1812.04608). arXiv. https://arxiv.org/abs/1812.04608
• Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020). Interpreting interpretability: Understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1 - 14). ACM. https://doi.org/10.1145/3313831.3376219
• Lakkaraju, H., & Bastani, O. (2020). "How do I fool you?": Manipulating user trust via misleading black box explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 79 - 85). ACM. https://doi.org/10.1145/3375627.3375833
• Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50 - 80. https://doi.org/10.1518/hfes.46.1.50_30392
• Liao, Q. V., Gruen, D., & Miller, S. (2020). Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1 - 15). ACM. https://doi.org/10.1145/3313831.3376590
• Liao, Q. V., & Varshney, K. R. (2021). Human-centered explainable AI (XAI): From algorithms to user experiences (arXiv:2110.10790). arXiv. https://arxiv.org/abs/2110.10790
• Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1 - 38. https://doi.org/10.1016/j.artint.2018.07.007
• Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023). From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable AI. ACM Computing Surveys, 55(13s), 1 - 42. https://doi.org/10.1145/3583558
• Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J., & Wallach, H. (2021). Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1 - 52). ACM. https://doi.org/10.1145/3411764.3445315
• Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L series, 12 July 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
• Schoeffer, J., De-Arteaga, M., & Kühl, N. (2024). Explanations, fairness, and appropriate reliance in human-AI decision-making. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1 - 18). ACM. https://doi.org/10.1145/3613904.3642621
• Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023). Explanations can reduce overreliance on AI systems during decision-making. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), 1 - 38. https://doi.org/10.1145/3579605
• Wang, X., & Yin, M. (2021). Are explanations helpful? A comparative study of the effects of explanations in AI-assisted decision-making. In 26th International Conference on Intelligent User Interfaces (pp. 318 - 328). ACM. https://doi.org/10.1145/3397481.3450650
• Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 295 - 305). ACM. https://doi.org/10.1145/3351095.3372852