Large Language Models

Overview

A large language model (LLM) is a neural network trained to predict text, at a scale of billions of parameters and on corpora drawn largely from the web, and then adapted to follow instructions and hold conversations. LLMs are used to summarize documents, draft letters, answer questions, screen applications and support decisions in education, healthcare, employment and law. They differ from the classifiers treated in Part II in three ways that matter for fairness. Their output is open-ended text rather than a label, so the question "was the decision correct?" often has no single answer. Their behavior depends on the prompt and on the examples placed in the context window, so the same model can behave differently in different applications. And they are rarely trained by the organization that deploys them: a foundation model is adapted downstream, so defects in the base model are inherited by every adapted system (Bommasani et al., 2021).

These differences explain why standard fairness approaches falter. A demographic parity constraint that can be imposed on a loan approval classifier (Chapter 14) has no direct meaning for a model that writes an assessment of a personal statement. Fairness work on LLMs therefore has to decide first what is being evaluated (a score extracted from text, the text itself, or a downstream decision informed by the text), and then choose measurement and mitigation methods that fit that object. This chapter builds on the treatment of representation learning in deep networks (Chapter 65) and of feedback loops in ranking systems (Chapter 66). Chapter 68 extends the analysis to images, audio and video, and Chapter 69 to generative and agentic systems that act on their outputs. Research frontiers are collected in Chapter 112.

The chapter uses a running example from higher education: an LLM-based assistant that reads application materials and drafts assessments for admissions officers. In the European Union, AI systems intended to determine access or admission to educational institutions are classified as high-risk (Regulation (EU) 2024/1689, Annex III, point 3(a)), so the example is also a regulated use.

Learning objectives:

• Explain why fairness criteria built for classifiers do not transfer directly to generated text, and which quantities can be measured instead.
• Identify the mechanisms through which bias enters an LLM system: pre-training data, adaptation, prompts and context, emergent behaviors and deployment filters.
• Compute counterfactual consistency, the counterfactual token fairness gap, the BBQ bias scores and a demographic representation score from evaluation records.
• Compare prompt-based, fine-tuning and guardrail mitigations and state the published evidence and limits for each.
• Design a layered implementation and evaluation framework for a high-stakes LLM application, including intersectional testing and monitoring.

Key questions

• How do large language models create fairness challenges beyond those of traditional machine learning models, particularly through scale, emergent behaviors and generative capabilities?
• Which fairness implementation strategies address these LLM-specific challenges while preserving the models' beneficial capabilities?
• What can be measured about the fairness of generated text, and with what validity?
• Which mitigations have published evidence behind them, and which rest only on plausibility?

Definitions and terminology

Large language model (LLM). A neural language model with a very large number of parameters, pre-trained on a large text corpus to predict the next token, and usually adapted afterwards by instruction tuning and preference optimization.

Foundation model. A model trained on broad data at scale that can be adapted to a wide range of downstream tasks (Bommasani et al., 2021). In EU law the corresponding legal category is the general-purpose AI model: a model that "displays significant generality and is capable of competently performing a wide range of distinct tasks" and can be integrated into downstream systems (Regulation (EU) 2024/1689, Article 3(63)).

Pre-training and fine-tuning. Pre-training fits the model to a large unlabeled corpus. Fine-tuning continues training on a smaller, task-specific or value-specific dataset.

Reinforcement learning from human feedback (RLHF). An adaptation method in which human raters rank model outputs, a reward model is trained on the rankings, and the language model is optimized against the reward model (Ouyang et al., 2022).

Prompt, system prompt and context window. The prompt is the text given to the model. A system prompt is an instruction set fixed by the deployer and placed before user input. The context window is the total text the model conditions on, including any examples supplied in the prompt (in-context learning).

Representational harm and allocational harm. A representational harm occurs when a system portrays a group in a demeaning, stereotyped or invisible way; an allocational harm occurs when a system distributes a resource or opportunity unequally. LLMs can cause both, often in the same output.

Counterfactual evaluation. Testing whether a model's output changes when only a group-identifying element of the input (a name, a pronoun, an identity term, a dialect feature) is changed.

Red teaming. Systematic adversarial testing in which people, or other models, try to elicit harmful or biased outputs (Perez et al., 2022; Ganguli et al., 2022).

Guardrail. A control placed around the model, such as an input classifier, an output classifier or a rule that routes a case to human review.

Sycophancy. The tendency of a model to give responses that match the user's stated views rather than accurate ones (Sharma et al., 2023).

Hallucination. Generated text that is nonsensical or unfaithful to the provided source content, often while appearing fluent and natural (Ji et al., 2023).

Why classification fairness does not transfer directly

The criteria of Part II are statements about the joint distribution of a protected attribute A, a true outcome Y and a prediction Ŷ. They presuppose that Ŷ takes a small number of values and that Y is observed. A generated assessment of an applicant satisfies neither presupposition. The output is a paragraph; whether it is "correct" depends on the facts in the file, on standards of evaluation and on the reader. Two descriptions that are both accurate can still differ in warmth, confidence or the qualities they emphasize, and those differences can shape the decision that follows.

Three measurement strategies follow from this. The first extracts a quantity from the text, such as a numeric rating, a recommendation or a sentiment score, and applies the criteria of Part II to that quantity. The second compares outputs for inputs that differ only in a group signal, which is the counterfactual logic of Chapter 18 applied to text. The third measures properties of the text itself, such as which groups are mentioned, which attributes they are associated with and how often the text is toxic. Each strategy captures part of the problem. None alone establishes that a system is fair, a point that broad evaluation programs make by measuring several properties side by side (Liang et al., 2023).

Pre-training data bias and amplification

Traditional bias mitigation often starts with a balanced training dataset (Chapter 47). That route is largely closed for LLMs, whose pre-training corpora are scraped from the web at a scale no team can curate by hand. The historical biases, stereotypes and harmful content in such corpora do not merely transfer to the model; pattern learning at scale can amplify them (Chapter 27 treats amplification in general). Bender et al. (2021) questioned the race toward ever larger models and recommended investing in curating and carefully documenting datasets rather than ingesting everything on the web. The underlying point is that scale does not guarantee diversity: a larger scrape of the web is not a more representative sample of any population, and what it contains is unknown unless it is documented.

Five mechanisms can be distinguished; the survey literature describes the resulting harms in more detail (Weidinger et al., 2021; Gallegos et al., 2024):

1. Representation disparities. Minority groups, cultures, dialects and languages appear less often, so the model learns less about them and performs worse for them.
2. Stereotype encoding. Co-occurrence patterns link demographic groups with stereotyped attributes, occupations or traits.
3. Distributional bias. Majority perspectives and cultural frameworks dominate, so they become the model's default framing.
4. Historical discrimination. Text written under discriminatory norms carries discriminatory language and framing into the model.
5. Harmful content. Toxic, abusive and discriminatory text is present in the corpus and can be reproduced.

Each mechanism has published evidence. Gehman et al. (2020) built RealToxicityPrompts, 100,000 naturally occurring prompts from web text, and found that pretrained language models can degenerate into toxic text even from seemingly innocuous prompts; their analysis of two web corpora used for pre-training found a significant amount of offensive, factually unreliable and otherwise toxic content, and none of the controllable generation methods they tested was failsafe. StereoSet (Nadeem et al., 2021) and CrowS-Pairs (Nangia et al., 2020) showed that widely used pretrained models prefer stereotypical over anti-stereotypical associations across gender, race, religion and other categories. Hofmann et al. (2024) found covert dialect prejudice: when shown text in African American English rather than Standard American English, language models were more likely to suggest that the speakers be assigned less prestigious jobs, be convicted of crimes and be sentenced to death, even though the models' overt stereotypes about African Americans were more positive.

Filtering the corpus does not remove the problem and can create a new one. Dodge et al. (2021) documented the Colossal Clean Crawled Corpus (C4) and showed that its blocklist filtering disproportionately removed text from and about minority individuals. A filter meant to remove harmful content thus reduced the representation of the groups it was meant to protect.

In the admissions example, a model may have absorbed associations between ethnic groups and academic stereotypes, or describe male applicants in more active, achievement-oriented language than female applicants. These patterns shape vocabulary, contextual representations, factual associations and generative tendencies, and they surface later when the model writes or evaluates text. Mitigation therefore has to address both the pre-training phase (through the choice and documentation of a base model) and the adaptation phase.

Where bias enters an LLM system

Bias can enter, be amplified or be reduced at every stage of the life cycle of an LLM application. The figure below summarizes the stages and the main intervention points used in the rest of this chapter.

Source: validant.ai.

The stages differ in who controls them. A deployer typically controls the system prompt, the context, the guardrails, the human workflow and monitoring; it may control domain fine-tuning; it rarely controls pre-training. The EU AI Act reflects this division: providers of general-purpose AI models must document the model and give downstream providers information about its capabilities and limitations (Article 53(1)(a) and (b); models released under a free and open-source licence with public parameters are exempt unless they have systemic risk, Article 53(2)), while the provider of a high-risk system built on such a model carries the obligations for the high-risk system, including the examination of possible biases in its training, validation and testing data (Article 10(2)(f) and (g)).

Prompt-based fairness strategies

Traditional fairness interventions modify a model through its data, its training objective or its outputs (Part V). LLMs open a further route: changing the input. Prompt engineering can steer outputs without retraining, which makes it fast and cheap to try and available to deployers who cannot change the model. Five strategies are commonly distinguished:

1. Fairness prompting. Explicit instructions in the system prompt that demographic characteristics must not influence the output.
2. Self-critique. Prompts asking the model to review its own output for bias and revise it.
3. Scaffolded generation. Multi-step prompts that guide the model through a fixed sequence of reasoning steps.
4. Counterfactual prompting. Generating alternative demographic framings of the same input to detect differences in the output, used as a test rather than a fix.
5. Chain-of-thought fairness. Asking the model to state its reasoning before answering, in the hope that explicit steps reduce reliance on implicit associations.

An admissions system might use a template such as: "Evaluate this personal statement focusing only on academic achievements and potential, regardless of the applicant's demographic background, and give a step-by-step analysis of strengths and weaknesses." Prompt design matters at three stages: in system design, where templates are written; in deployment, where they shape how users interact with the model; and in monitoring, where they define how outputs are checked.

The published evidence is more mixed than the enthusiasm for prompting suggests. Ganguli et al. (2023) tested whether models trained with RLHF can "morally self-correct" when instructed. On the BBQ benchmark, at 175 billion parameters, adding an instruction to avoid stereotypes and a request to reason first reduced the bias score by 84 percent relative to the plain question; the capability appeared at about 22 billion parameters and improved with scale and RLHF training. In their admissions-style experiment, built on a law school dataset, the same model discriminated against Black relative to white students by 3 percent without intervention and in favor of Black students by 7 percent with instruction and reasoning, showing that a prompt can overshoot the target.

Tamkin et al. (2023) generated decision prompts for 70 scenarios, varied age, gender and race, and found both positive and negative discrimination in one commercial model; appended statements, for example that demographic information must be ignored or that it is illegal to take protected characteristics into account, reduced their discrimination score substantially, in some interventions to about 0.15 while keeping about 92 percent correlation with the original decisions.

Chain-of-thought prompting has a contested record. Wei et al. (2022a) showed that chain-of-thought prompting improves performance on arithmetic, commonsense and symbolic reasoning tasks in sufficiently large models; that paper does not report fairness results. Shaikh et al. (2023) then found that zero-shot chain-of-thought reasoning in socially sensitive domains significantly increased the likelihood of harmful or undesirable output on harmful questions and stereotype benchmarks, with the effect growing with model size and shrinking with better instruction following. Reasoning prompts therefore need to be tested for their fairness effect in each application, not assumed to help.

Fine-tuning for fairness

LLMs usually arrive pre-trained, so fine-tuning is the main point at which a deployer or model developer can change the model itself. Five strategies are in use:

1. Balanced fine-tuning datasets. Examples chosen to cover demographic groups and to apply consistent standards.
2. Fairness-specific preference optimization. RLHF or related preference methods in which raters are instructed to penalize biased outputs.
3. Counterfactual data augmentation (CDA). Adding copies of training examples with group terms swapped, so that the model sees the same content with different demographic framings.
4. Bias-targeted adapters. Small trainable components inserted into a frozen model and trained for debiasing.
5. Multi-objective fine-tuning. Optimizing task performance and an explicit fairness measure together (Chapter 54).

Each has a published basis. Lu et al. (2020) introduced counterfactual data augmentation, "a generic methodology for corpus augmentation via causal interventions" that breaks associations between gendered and gender-neutral words, and found that it decreased gender bias in coreference resolution and language modeling while preserving accuracy. Webster et al. (2020) showed that models with similar accuracy can encode gendered correlations at very different rates and that general-purpose techniques can reduce them, with trade-offs that differ by strategy. Lauscher et al. (2021) proposed debiasing adapters (ADELE) trained on a counterfactually augmented corpus while the base model stays frozen, and reported that the adapters reduced gender bias in BERT and retained fairness after downstream training.

Solaiman and Dennison (2021) described PALMS, an iterative process of fine-tuning on a small, hand-curated values-targeted dataset; it performed better than baseline and control models on adherence to target values, toxicity and the words most associated with social categories, across GPT-3 sizes and without compromising capability integrity in their evaluations, and its effectiveness increased with model size. Ouyang et al. (2022) reported that RLHF-trained InstructGPT models reduced toxic output generation relative to GPT-3, although the same paper notes that the models still make simple mistakes.

In the admissions example, a fine-tuning dataset would contain diverse applicant profiles evaluated by consistent standards, and counterfactual augmentation would ensure that applications differing only in demographic signals receive the same evaluation. Fine-tuning affects fairness during dataset creation (which examples are selected), training (which objective is optimized) and evaluation (how improvement is measured).

Two cautions apply. First, preference data carry the views of the raters who produce them, so a fairness-oriented RLHF process is only as representative as its rater pool and instructions. Second, preference optimization can create new problems: Perez et al. (2023) found that more RLHF training made models express stronger political views on some topics (gun rights and immigration) and that RLHF did not train away sycophancy, which the preference models used for RLHF rewarded; Sharma et al. (2023) found that human preference judgments favor responses that match the user's views.

Emergent behaviors and safety guardrails

Traditional fairness work assumes that a model behaves consistently and that testing on a representative sample predicts behavior in deployment. LLMs weaken this assumption. Wei et al. (2022b) described "emergent abilities", capabilities that are absent in smaller models and present in larger ones and so cannot be predicted by extrapolating from small models. Schaeffer et al. (2023) argued that many apparent emergent abilities are produced by the choice of nonlinear or discontinuous evaluation metrics, and that continuous metrics show smooth change. For fairness, the practical point holds under either reading: behaviors relevant to bias can differ sharply between model versions, so every model change requires re-evaluation.

Five behaviors have direct fairness implications:

1. Instruction following. The capability to execute complex directives, including harmful or discriminatory ones supplied by a user.
2. In-context learning. Adaptation to examples in the prompt, including biased examples. Few-shot behavior is unstable: the choice and order of examples can move accuracy from near chance to near state of the art, partly because models favor answers that are common in pre-training data or placed near the end of the prompt (Zhao et al., 2021).
3. Sycophancy. Agreement with the user's stated views. Perez et al. (2023) found that larger models repeat back a dialogue user's preferred answer, and Sharma et al. (2023) found sycophancy across five state-of-the-art assistants and four free-form tasks.
4. Jailbreaking vulnerability. Susceptibility to adversarial prompts that bypass safety measures. Zou et al. (2023) found automatically generated adversarial suffixes that caused aligned models to produce objectionable content and that transferred to other models, including black-box ones.
5. Hallucination. Fluent text that is unfaithful to the source or to fact (Ji et al., 2023). Its fairness relevance lies in how errors are distributed. A working hypothesis, which an assessor should test rather than assume, is that a model hallucinates more about institutions, regions or groups that are thinly covered in its training data; false statements about applicants or their schools can harm the individuals concerned.

In the admissions example, these behaviors combine. The model may agree with an officer's biased remark about a group's academic potential (sycophancy), state different "facts" about universities depending on the applicant's background (hallucination) and copy biases from example assessments placed in the prompt (in-context learning).

Guardrails respond to these behaviors. During model selection they guide the choice of base model; during deployment they take the form of input and output filters and routing rules; during monitoring they support detection of new harmful behaviors. Classifier-based safeguards that label prompts and responses against a risk taxonomy are now common (Inan et al., 2023). Guardrails are themselves models and can be biased. Sap et al. (2019) found that hate speech detection models trained on widely used datasets labeled tweets in African American English and tweets by self-identified African Americans as offensive up to two times more often than others. An output filter can therefore suppress the speech of the groups it is meant to protect, in the same way as the corpus filter documented by Dodge et al. (2021). Guardrails need their own fairness evaluation, disaggregated by dialect and group.

Measuring fairness in LLM applications

Classification metrics such as demographic parity or equal opportunity are insufficient on their own for LLMs, because generated outputs cannot be checked against a single correct label. LLM fairness requires evaluation methods that address the generative nature of the output. Five families of methods are used: red teaming, counterfactual evaluation, benchmark suites, human evaluation protocols and multi-dimensional measurement. This section defines the formal measures used within them.

Counterfactual consistency and the counterfactual token fairness gap

Let x be an input text (for example an application file), let Φ(x) be a set of counterfactual versions of x in which group signals are substituted, and let f(x) be a numeric quantity extracted from the model's output (a rating, a probability of a "yes", a sentiment score). Garg et al. (2019) defined counterfactual token fairness for text classifiers: a model satisfies it with tolerance ε when the output changes by at most ε under every substitution of identity tokens.

They measured violations with the counterfactual token fairness gap, the average absolute change over the counterfactuals of an example, averaged again over all examples that have valid counterfactuals.

For a discrete output such as a rating from 1 to 5, a simpler measure is counterfactual consistency, the share of counterfactual pairs whose output is identical.

The signed mean shift, the average of f(x') − f(x), shows whether changes go in a consistent direction. A gap with no consistent direction indicates instability; a gap with a consistent direction indicates bias. HELM uses the same perturbation logic, substituting race and gender terms and dialect features, and reports accuracy on the perturbed instances (Liang et al., 2023). Tamkin et al. (2023) fit a mixed-effects model to the log-odds of a "yes" decision and report the coefficients of demographic variables, relative to a baseline profile, as discrimination scores; a score of +1.0 corresponds to an average probability of a "yes" of 0.73 when the baseline probability is 0.5.

Counterfactual tests have limits. A substitution can change more than the group signal: a name can imply nationality, religion or social class at once. And perturbations are generated from one group's data, so they miss unfairness that arises when the groups' real data differ in more complex ways, which is why HELM also reports performance disparities on data with genuine group metadata (Liang et al., 2023).

Bias benchmarks

Benchmark suites provide standardized tests for specific fairness dimensions. Their design differs in ways that matter for interpretation.

Benchmark	What it tests	Format	Reference
StereoSet	Stereotypical associations in four domains	Sentence and context choices	Nadeem et al. (2021)
CrowS-Pairs	Preference for stereotyping sentences, nine bias types	1,508 minimal sentence pairs	Nangia et al. (2020)
BBQ	Reliance on stereotypes in question answering, nine dimensions	Ambiguous and disambiguated contexts	Parrish et al. (2022)
BOLD	Bias in open-ended generation, five domains	23,679 prompts	Dhamala et al. (2021)
RealToxicityPrompts	Toxic degeneration from natural prompts	100,000 prompts	Gehman et al. (2020)
WinoMT	Gender bias in machine translation	Coreference test set, eight languages	Stanovsky et al. (2019)

Source: the cited papers.

BBQ illustrates how a benchmark turns behavior into a number. Each question appears with an ambiguous context, in which the correct answer is "unknown", and with a disambiguated context, in which the text supplies the answer. A biased answer is the stereotyped target in a negative question or the non-target in a non-negative question. With n_biased the number of biased answers and n_non-UNKNOWN the number of answers that are not "unknown", the bias score in disambiguated contexts is:

In ambiguous contexts the same ratio is computed from the answers to the ambiguous questions and then scaled by the error rate, because a biased answer is more harmful when it is given more often (the paper writes the factor s_DIS for this ratio):

A score of 0 means no measured bias, +1 (100 percent) that all non-unknown answers follow the bias, and −1 that all go against it (Parrish et al., 2022). The BBQ authors found that models often rely on stereotypes when the context is under-informative, and that even in informative contexts accuracy was on average up to 3.4 percentage points higher when the correct answer aligned with a social bias, widening to over 5 points on questions about gender for most models tested.

Benchmarks have known weaknesses. Blodgett et al. (2021) inventoried pitfalls in StereoSet, CrowS-Pairs and two coreference benchmarks, finding that the benchmarks frequently lack clear articulations of what is being measured and contain ambiguities and unstated assumptions in how they conceptualize and operationalize stereotyping. Most benchmarks are in English and centered on United States social categories, as the BBQ authors state for their own dataset. A good benchmark score is evidence about the benchmark, not a certificate for an application.

Representation in generated text

When outputs are open-ended, one measurable property is how often groups are mentioned. HELM computes demographic representation as follows (Liang et al., 2023). For k groups, each with a word list A_i, count the occurrences C(i) of the group's words over all generations, normalize the counts into an observed distribution P_obs and compute its total variation distance from a reference distribution P_ref, which HELM sets to uniform:

With a uniform reference the maximum is (k − 1)/k, that is 0.5 for two gender groups and 2/3 for three race groups. Stereotypical association applies the same calculation only to generations that contain a target term such as an occupation, and averages over a list of terms. The uniform reference is a modeling choice: for some applications the appropriate reference is the composition of the relevant population, and the choice must be justified (Chapter 22).

Red teaming and human evaluation

Red teaming searches for failures that fixed benchmarks miss. Perez et al. (2022) used one language model to generate test cases for another and found tens of thousands of offensive replies in a 280-billion-parameter chatbot, including groups of people that the chatbot discussed in offensive ways. Ganguli et al. (2022) released 38,961 human red-team attacks and found that RLHF models became harder to red team as they scaled, while other model types showed a flat trend. Red teaming gives evidence that a failure exists; it does not estimate how often the failure occurs in deployment. CheckList-style behavioral testing (Ribeiro et al., 2020) complements it with systematic test suites for invariance (outputs should not change under group substitutions) and directional expectations.

Human evaluation protocols assess what automated measures cannot, such as whether two assessments apply the same standard. They require trained raters, written rubrics, raters from the communities concerned, measures of inter-rater agreement, and blinding of raters to the group signal where possible.

Multi-dimensional measurement

No single number summarizes LLM fairness. HELM measures accuracy, calibration, robustness, fairness, bias, toxicity and efficiency together and found that the most accurate model was not always the most robust or the most fair; where demographic metadata existed, it found consistent performance disparities for all models, for example language modeling performance on African American English worse than on White English for every model, with the best model at 1.506 bits per byte for White English and 2.114 for African American English (lower is better) (Liang et al., 2023). Weidinger et al. (2022) organized the risks of language models into six areas, the first of which, discrimination, hate speech and exclusion, includes social stereotypes and unfair discrimination, hate speech and offensive language, exclusionary norms and lower performance for some languages and social groups.

Worked example: counterfactual testing of an admissions assistant

An admissions team audits an LLM assistant that reads an application file and returns a "potential" rating from 1 to 5. The team draws 400 real application files, removes identifying information, and creates three counterfactual versions of each: one with a female-coded name, one with cues of an international applicant (an overseas school and a non-native English profile), and one with both cues. All other content is identical. Each original and each counterfactual is rated once at temperature zero. The table records, for each variant, how many of the 400 pairs changed and by how much.

Variant	−2	−1	0	+1	+2
Female-coded name	4	30	348	16	2
International cue	8	44	330	16	2
Both cues	22	70	290	16	2

Source: constructed example.

The summary measures follow from the definitions above. For the female-coded variant, counterfactual consistency is 348/400 = 0.870. The signed mean shift is (−2 × 4 − 1 × 30 + 1 × 16 + 2 × 2)/400 = −18/400 = −0.045 rating points, and the counterfactual gap is (2 × 4 + 30 + 16 + 2 × 2)/400 = 58/400 = 0.145. Ratings fell in 34 pairs and rose in 18. Under the null hypothesis that a change is equally likely to go either way, a two-sided sign test on 52 changed pairs gives p ≈ 0.036.

Variant	Consistency	Mean shift	CF gap	Down / up	Sign test p
Female-coded name	0.870	−0.045	0.145	34 / 18	0.036
International cue	0.825	−0.100	0.200	52 / 18	0.00006
Both cues	0.725	−0.235	0.335	92 / 18	< 0.0001

Source: constructed example; values computed from the change counts above.

Source: validant.ai.

Three lessons follow. First, a high consistency rate can hide a directional bias: 87 percent of ratings are unchanged for the female-coded variant, yet the changes that occur run nearly two to one against the counterfactual group, and the sign test indicates that this is unlikely to be chance. Second, the effect at the intersection is larger than the sum of the single effects. If the two cues acted additively, the expected shift for both cues would be −0.045 − 0.100 = −0.145; the observed shift is −0.235, so the interaction is −0.090 points. Single-attribute testing would have missed more than a third of the effect for international female applicants (0.090 of 0.235 points). Third, the measures describe the model's response to the substituted cues, not the fairness of admissions outcomes: the cues used may also carry information that the institution is entitled to consider (for example, language proficiency where it is a published requirement), and that question must be settled before the result is interpreted as discrimination.

Two further calculations show how benchmark and representation scores are read. Suppose the same model is run on 1,000 ambiguous BBQ-style questions about academic ability and answers "unknown" correctly in 620 of them. Of the 380 other answers, 266 follow the stereotype. Then 2 × 266/380 − 1 = 0.40, and s_AMB = (1 − 0.62) × 0.40 = 0.152. On 1,000 disambiguated questions with 960 non-unknown answers, 528 of which are biased, s_DIS = 2 × 528/960 − 1 = 0.10. Finally, suppose that in 2,000 generated feedback letters male-gendered words occur 1,260 times and female-gendered words 740 times. Then P_obs = (0.63, 0.37), and the total variation distance from the uniform reference is 0.13, against a maximum of 0.5 for two groups.

The layered implementation framework

A five-part framework organizes the implementation of fair LLM applications. It is presented here as a recommended practice, with the evidence for each layer noted in the sections above.

1. Pre-training considerations. Select base models whose pre-training data are documented; evaluate base model bias across demographic dimensions; assess performance disparities across languages, dialects and cultural contexts; and identify specific bias patterns for targeted mitigation.
2. Prompt engineering for fairness. Write explicit fairness directives into system prompts; add self-assessment prompts that check outputs for bias; create structured reasoning templates; and build counterfactual prompt tests to detect bias.
3. Fine-tuning strategies. Curate balanced, representative fine-tuning datasets; apply counterfactual data augmentation across demographic dimensions; design preference optimization objectives with explicit fairness components; and apply targeted training for identified bias patterns.
4. Safety guardrails. Filter inputs for harmful prompt patterns; classify outputs to detect biased responses; monitor for emergent harmful behaviors; and set intervention protocols for identified issues.
5. Evaluation and monitoring. Run fairness benchmark suites; red team continuously; log and monitor production use; and establish channels for reported issues.

Source: validant.ai.

The framework extends existing LLM workflows rather than replacing them: model selection gains fairness criteria, prompt design gains fairness directives, fine-tuning gains fairness objectives and safety systems gain fairness-specific checks. It aims at measurable improvement within current technical limits rather than at a guarantee of fairness, which no current method provides.

The case for layering is that each layer catches failures the others miss. Prompt instructions can be overridden by users; fine-tuning changes behavior on the training distribution but not necessarily elsewhere; guardrails catch only what their classifiers recognize; human reviewers suffer from automation bias. The EU AI Act names this last risk explicitly: human oversight measures for high-risk systems must enable overseers to remain aware of "the possible tendency of automatically relying or over-relying on the output" (Article 14(4)(b)).

Implementation challenges

Four pitfalls recur.

1. Fairness-performance trade-offs. Excessive safety constraints produce overly conservative models that refuse legitimate requests and lose useful capabilities. The response is calibrated interventions that target specific harmful behaviors rather than broad restrictions, fairness measures at several points rather than one heavy-handed intervention, and evaluation of both fairness and capability effects. Tamkin et al. (2023) illustrate the measurement: they report both the reduction in discrimination and the correlation with the original decisions.
2. Emergent behavior adaptation. Interventions lose effect as users find new ways to elicit biased outputs. The response is continuous red teaming with diverse adversarial approaches, safety layers updated from observed patterns and detection of new exploitation methods.
3. Domain adaptation. General interventions miss the needs of specialized domains. The response is domain-specific fine-tuning with domain experts, context-aware evaluation and prompt templates that encode the domain's fairness standards.
4. Evaluation complexity. Simple metrics fail to capture LLM-specific fairness dimensions. The response is multi-dimensional evaluation, automated measures combined with structured human evaluation, and longitudinal testing across varied contexts. Weidinger et al. (2021) make the same point about the breadth of the risk landscape.

Communication should emphasize progressive improvement rather than claims of a solved problem. Executives need to know how fairness work improves product safety and reduces legal and reputational risk; product teams need clear implementation guidance from the guardrail design; users need evidence that the assistance they receive is reliable for people like them. The resources required are fairness-oriented fine-tuning data, red-team capacity, structured evaluation frameworks and production monitoring.

Evaluation approach and targets

Five measures summarize an implementation: representation fairness (how equitably groups are represented in outputs), stereotyping (the amount of harmful stereotyping in outputs), counterfactual consistency (how outputs vary when only demographic attributes change), safety effectiveness (the share of harmful or biased outputs prevented) and capability preservation (the share of beneficial capability retained). Measuring them together follows the multi-metric approach of HELM (Liang et al., 2023). Each measure links to an outcome: representation and stereotype measures address harm to marginalized groups, safety effectiveness addresses explicit bias that would otherwise reach users, and capability preservation shows that fairness work has not removed the system's usefulness.

Starting targets of the following kind can anchor an implementation: counterfactual consistency above 90 percent for core use cases; a stereotype reduction of at least 80 percent compared with the unmitigated baseline; safety effectiveness above 95 percent for high-risk scenarios; capability preservation of at least 90 percent; and disparities below 10 percent across major demographic groups. These figures are not established standards and no published study validates them. They are offered as internal targets that an organization may adopt and must justify for its context, together with a statement of the sample size needed to detect a violation (Chapters 41 and 43). The worked example shows why a consistency target alone is insufficient: 87 percent consistency coexisted with a statistically detectable directional bias.

For organizations new to LLM fairness, a minimum starting point is: fairness-oriented system prompts; counterfactual testing across demographic dimensions, including intersections; basic monitoring of outputs for bias; and clear escalation paths for identified issues. Prompt-based approaches are a sensible first step because they are cheap to test, provided their effect is measured rather than assumed.

Intersectionality

Fairness work on LLMs often examines one dimension at a time, gender stereotypes or racial representation. This misses patterns at intersections, where combinations of attributes produce effects that neither attribute alone predicts. Crenshaw (1989) showed how antidiscrimination doctrine that treats race and sex separately fails Black women, whose experience is not captured by either dimension alone; the same logic applies to model behavior (Chapter 29). The worked example above is a case: the effect for applicants with both cues exceeded the sum of the single effects. HELM's authors note that intersectional evaluation is straightforward within their perturbation approach but left it for future work (Liang et al., 2023).

To embed intersectional principles: design prompt templates that address intersectional fairness explicitly; build fine-tuning datasets with intersectional coverage; evaluate outputs across demographic intersections; make guardrails sensitive to intersectional harms; and train red teams to look for failures that single-dimension tests miss. The number of intersections grows quickly, so coverage must be balanced against feasibility: an assessment prioritizes intersections by the stakes for the people concerned and by prior evidence of harm, and reports which intersections were not tested.

Domain modeling perspective

Viewed as a domain model, LLM fairness extends classification fairness with elements that simpler models do not have: emergent behaviors, generative output, context sensitivity and instruction following. These elements interact. Pre-training sets baseline behavior and fine-tuning adjusts it; prompts set the context of generation while model scale determines which behaviors appear; guardrails set boundaries and evaluation checks whether they hold. Fairness properties arise from the whole configuration, not from the model alone.

The stakeholders differ accordingly: system developers who choose and adapt models, prompt engineers who shape interactions, domain experts who validate outputs, governance bodies who set standards, and the users and affected people whose experience determines whether the system is fair in practice. Each brings a different view of what fair behavior means. Bender et al. (2021) asked how big is too big and which risks grow with the size of models and their training data; in governance terms, the scale and generality of foundation models are the reason for distinct legal obligations for their providers (Regulation (EU) 2024/1689, Articles 53 and 55).

Case study: an LLM admissions assistant

This case is illustrative. It combines patterns reported in the literature cited in this chapter into one scenario.

Context

A selective university introduces an LLM-based assistant to support application review for undergraduate and graduate programs. The assistant analyzes personal statements, recommendation letters, academic records and extracurricular activities, extracts relevant information and drafts balanced assessments for admissions officers. Stakeholders are the admissions officers who use it, the applicants who are evaluated, the university administration, faculty committees and the diversity office. In the EU, such a system is high-risk under Annex III, point 3(a), of the AI Act.

Problem analysis

An early audit finds several patterns. Assessments are more positive for applicants from prestigious schools regardless of achievements. The assistant highlights technical skills and leadership for male applicants and communication and collaboration for female applicants. It writes more enthusiastic language for native English speakers than for international applicants with similar qualifications, and emphasizes different extracurricular activities depending on applicant demographics (sports for some ethnic groups, academic competitions for others). When asked to evaluate "potential", it favors applicants from privileged backgrounds with similar credentials.

The ethics team traces five causes, each linked to a mechanism described earlier:

1. Pre-training bias. The base model has absorbed associations between demographic groups, prestigious institutions and academic potential, which surface as more tentative language for some groups and more confident language for others.
2. Prompt sensitivity. Terms such as "leadership potential" and "academic excellence" in the prompts trigger learned demographic associations; even neutral wording does not remove them.
3. Sycophancy. When an officer implies that an applicant may not fit, the assistant generates supporting evidence regardless of the file.
4. Inconsistent standards. Identical community service is described as "leadership" for affluent applicants and as "participation" for low-income applicants.
5. Intersectional blindness. International female applicants in STEM receive markedly lower assessments than either attribute alone predicts, the pattern shown in the worked example.

The stakes are high because admissions decisions shape access to education and life opportunities, and because non-discrimination law applies to them.

Solution

The team implements five layers.

1. Fairness-oriented prompting. A system prompt directs evaluation on achievements and demonstrated potential only, without regard to demographic factors, institutional prestige or socioeconomic indicators. A structured template asks first for academic achievements with specific evidence, then for depth and impact of extracurricular involvement, then for personal qualities shown by concrete examples. A self-critique step asks whether consistent standards were applied and whether the assessment rests on demonstrated achievements. A counterfactual check asks whether the assessment would change for an applicant from a different background. Given the evidence of Shaikh et al. (2023), each element is retained only if counterfactual tests show that it helps.
2. Targeted fine-tuning. A balanced dataset of admissions materials, counterfactual augmentation of applications that differ only in demographic factors, preference optimization with explicit fairness criteria, and an adapter for admissions evaluation that leaves the general model unchanged.
3. Guardrails. Input classification to detect biased framing in officer queries, output checks for unequal evaluations, an explanation of the factors behind each assessment, and human review triggers for borderline cases. The guardrail classifiers are themselves tested for dialect and group disparities.
4. Evaluation. An admissions-specific benchmark with diverse profiles, regular red teaming with biased queries, counterfactual evaluation sets and production monitoring.
5. Human-AI collaboration. The workflow is redesigned as explicit collaboration rather than automation: fact extraction is separated from subjective assessment, confidence indicators accompany each assessment component, disagreements between the assistant and officers are surfaced, and documentation records which judgments were made by the AI and which by people.

The team augments an existing foundation model rather than building a new one, which keeps the approach feasible.

Outcomes and lessons

After the changes, the team repeats the counterfactual audit on the same 400 files and reports consistency, mean shift and sign tests by variant and intersection, alongside officers' assessments of quality and the processing time of the review. In the illustrative scenario the measured disparities fall substantially, the influence of school prestige on assessments declines, overall assessment quality is maintained, extraction accuracy improves through structured prompting, processing time rises only marginally, officers report more trust in the assistant, and review consistency across officer teams improves. Documentation of the interventions supports the university's legal review, the diversity office can verify outcomes for underrepresented applicants, and the surfaced disagreements help officers recognize their own bias triggers.

Four lessons generalize.

1. Prompts shape fairness. Carefully designed and tested prompts reduced bias without architectural changes, and structured reasoning produced more consistent evaluations. The published evidence supports this when the effect is measured (Ganguli et al., 2023; Tamkin et al., 2023) and warns against assuming it (Shaikh et al., 2023).
2. Layers outperform single interventions. No single strategy solved every problem; prompting, fine-tuning, guardrails and human oversight each caught different failures.
3. Collaboration supports fairness. Framing the system as assistance rather than autonomous evaluation, and making its reasoning visible, helped people find the remaining bias.
4. Fairness requires ongoing vigilance. New bias patterns appeared in deployment as users found new ways to frame queries, so monitoring and adaptation must continue.

Competing interpretations and limitations

Is LLM fairness a property of the model or of the application? Benchmarks evaluate models; harms occur in applications. A model with a good benchmark score can still discriminate in an application whose prompts, context and users differ from the benchmark, and a model with known biases can be deployed acceptably behind strong controls. This chapter treats fairness as a property of the configured application and benchmarks as evidence about one component.

Fairness through unawareness in prompts. Instructions to ignore demographics are a form of fairness through unawareness, which is weak for classifiers because proxies remain (Chapter 24). In LLMs proxies are pervasive: names, schools, dialect and writing style. Instructions can reduce measured differences, but they do not remove proxies and can overshoot (Ganguli et al., 2023).

Positive and negative discrimination. Tamkin et al. (2023) and Ganguli et al. (2023) both observed models favoring historically disadvantaged groups after alignment or instruction. Whether such shifts are acceptable depends on law and on the normative choices of Chapter 22; a symmetric metric treats them as violations, while some legal frameworks permit certain positive action. An assessment states which view it takes.

Validity of measures. Counterfactual tests depend on substitution lists; benchmarks are often English-only and centered on United States categories; representation scores depend on the reference distribution; human evaluation depends on raters. Measurement validity (Chapter 45) is the main constraint on what an LLM fairness assessment can claim.

Emergence. Whether emergent abilities are real discontinuities or artifacts of metrics is disputed (Wei et al., 2022b; Schaeffer et al., 2023). The practical requirement, re-evaluation at every model change, does not depend on the answer.

Common questions

How can guardrails be implemented without making the system so restricted that it loses its useful capabilities?
Target specific harms rather than imposing blanket restrictions. Start with a domain-specific taxonomy of harmful and beneficial behaviors; in admissions, distinguish demographic assumptions from legitimate expertise about academic achievement. Prefer interventions that change behavior, such as instruction tuning, to interventions that only block outputs. Use context-sensitive filters rather than universal keyword lists, which also avoids the disparate filtering documented by Dodge et al. (2021) and Sap et al. (2019). Scale scrutiny with risk, and for borderline cases use softer responses such as adding context or routing to a person instead of refusing. Measure both sides of the trade-off: Tamkin et al. (2023) report discrimination together with agreement with the original decisions, and HELM reports fairness alongside accuracy (Liang et al., 2023). The principle is precise intervention, not blunt restriction.

How should bias in emergent behaviors that appear only in production be handled?
Build a detection and response system. Log interactions with enough context to reconstruct what happened, within data protection limits. Monitor actively, with automated measures and human review of samples, focusing on edge cases and unexpected interactions. Provide users with simple channels to report unfair outputs. Assign clear ownership and escalation paths for new bias patterns. Most importantly, close the loop: identified issues should feed prompt revisions, fine-tuning updates and guardrail changes, each followed by re-running the counterfactual and benchmark tests. In LLM applications, fairness is maintained continuously rather than achieved once. For providers of general-purpose AI models with systemic risk, the EU AI Act requires adversarial testing, assessment and mitigation of systemic risks and reporting of serious incidents (Article 55(1)).

Does a good score on a bias benchmark mean the model is safe to use?
No. Benchmarks test specific phenomena in specific formats, mostly in English, and some have validity problems (Blodgett et al., 2021). The deployer must test the application with its own prompts, data and user population.

Should demographic data be collected to test an LLM application?
Counterfactual tests can be run without collecting demographic data about real people, because group signals are substituted synthetically. Measuring performance disparities on real users requires group information. In the EU, providers of high-risk systems may exceptionally process special categories of personal data where strictly necessary for bias detection and correction, under the conditions of Article 4a of the AI Act, inserted by Regulation (EU) 2026/1744 in place of the former Article 10(5).

Conclusions

LLMs require a change in the unit of fairness analysis. In place of a single prediction compared across groups, an assessor examines a configured application: a base model with inherited biases, an adaptation process, prompts and context, guardrails, a human workflow and a monitoring loop. Fairness criteria from Part II still apply wherever a quantity can be extracted from the output, and counterfactual reasoning applies naturally to text, but these must be complemented by benchmarks, representation measures, red teaming and structured human evaluation, each with stated limits.

The evidence supports several firm conclusions. Pretrained models reproduce social biases, including covert ones. Filters and classifiers used for safety can themselves discriminate. Prompt instructions and fine-tuning can reduce measured bias, sometimes substantially, but their effects depend on scale, wording and test, and can overshoot. Methodologically, validant.ai's position is that an LLM fairness claim should name the object evaluated, the tests used, the intersections covered, the effect of each mitigation layer and the conditions under which the evidence would need to be renewed, in particular any change of base model.

Open questions

• How can fairness evaluations of generated text be validated against real-world harm rather than against benchmark constructs?
• Which reference distributions are appropriate for representation measures in different applications, and who should choose them?
• How stable are prompt-based and fine-tuning mitigations across model updates and real user input distributions?
• How can counterfactual substitutions be built for languages, cultures and protected characteristics outside the English-language, United States-centered benchmarks?
• How should responsibility for bias be divided between providers of foundation models and deployers who adapt them, and what evidence should pass between them?
• How can intersectional coverage be achieved when the number of intersections exceeds what can be tested?

References

• Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610 - 623). ACM. https://doi.org/10.1145/3442188.3445922
• Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021). Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 1004 - 1015). ACL. https://doi.org/10.18653/v1/2021.acl-long.81
• Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. https://arxiv.org/abs/2108.07258
• Crenshaw, K. (1989). Demarginalizing the intersection of race and sex: A Black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. University of Chicago Legal Forum, 1989(1), 139 - 167. https://chicagounbound.uchicago.edu/uclf/vol1989/iss1/8
• Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021). BOLD: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 862 - 872). ACM. https://doi.org/10.1145/3442188.3445924
• Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021). Documenting large webtext corpora: A case study on the Colossal Clean Crawled Corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 1286 - 1305). ACL. https://doi.org/10.18653/v1/2021.emnlp-main.98
• European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L, 2024/1689. http://data.europa.eu/eli/reg/2024/1689/oj
• European Parliament and Council of the European Union. (2026). Regulation (EU) 2026/1744 of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI). Official Journal of the European Union, L, 2026/1744. https://eur-lex.europa.eu/eli/reg/2026/1744/oj
• Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024). Bias and fairness in large language models: A survey. Computational Linguistics, 50(3), 1097 - 1179. https://doi.org/10.1162/coli_a_00524
• Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022). Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858. https://arxiv.org/abs/2209.07858
• Ganguli, D., Askell, A., Schiefer, N., Liao, T. I., Lukošiūtė, K., Chen, A., Goldie, A., Mirhoseini, A., Olsson, C., Hernandez, D., et al. (2023). The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459. https://arxiv.org/abs/2302.07459
• Garg, S., Perot, V., Limtiaco, N., Taly, A., Chi, E. H., & Beutel, A. (2019). Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (pp. 219 - 226). ACM. https://doi.org/10.1145/3306618.3317950
• Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020). RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 3356 - 3369). ACL. https://doi.org/10.18653/v1/2020.findings-emnlp.301
• Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024). AI generates covertly racist decisions about people based on their dialect. Nature, 633(8028), 147 - 154. https://doi.org/10.1038/s41586-024-07856-5
• Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., & Khabsa, M. (2023). Llama Guard: LLM-based input-output safeguard for human-AI conversations. arXiv preprint arXiv:2312.06674. https://arxiv.org/abs/2312.06674
• Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1 - 38. https://doi.org/10.1145/3571730
• Lauscher, A., Lueken, T., & Glavaš, G. (2021). Sustainable modular debiasing of language models. In Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 4782 - 4797). ACL. https://doi.org/10.18653/v1/2021.findings-emnlp.411
• Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research. https://arxiv.org/abs/2211.09110
• Lu, K., Mardziel, P., Wu, F., Amancharla, P., & Datta, A. (2020). Gender bias in neural natural language processing. In Logic, Language, and Security (Lecture Notes in Computer Science, Vol. 12300, pp. 189 - 202). Springer. https://doi.org/10.1007/978-3-030-62077-6_14
• Nadeem, M., Bethke, A., & Reddy, S. (2021). StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 5356 - 5371). ACL. https://doi.org/10.18653/v1/2021.acl-long.416
• Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020). CrowS-Pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (pp. 1953 - 1967). ACL. https://doi.org/10.18653/v1/2020.emnlp-main.154
• Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. https://arxiv.org/abs/2203.02155
• Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022). BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022 (pp. 2086 - 2105). ACL. https://doi.org/10.18653/v1/2022.findings-acl.165
• Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022). Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 3419 - 3448). ACL. https://doi.org/10.18653/v1/2022.emnlp-main.225
• Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2023). Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 13387 - 13434). ACL. https://doi.org/10.18653/v1/2023.findings-acl.847
• Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 4902 - 4912). ACL. https://doi.org/10.18653/v1/2020.acl-main.442
• Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019). The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1668 - 1678). ACL. https://doi.org/10.18653/v1/P19-1163
• Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.15004. https://arxiv.org/abs/2304.15004
• Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023). On second thought, let's not think step by step! Bias and toxicity in zero-shot reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4454 - 4470). ACL. https://doi.org/10.18653/v1/2023.acl-long.244
• Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., et al. (2023). Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548. https://arxiv.org/abs/2310.13548
• Solaiman, I., & Dennison, C. (2021). Process for adapting language models to society (PALMS) with values-targeted datasets. arXiv preprint arXiv:2106.10328. https://arxiv.org/abs/2106.10328
• Stanovsky, G., Smith, N. A., & Zettlemoyer, L. (2019). Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1679 - 1684). ACL. https://doi.org/10.18653/v1/P19-1164
• Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J., & Ganguli, D. (2023). Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689. https://arxiv.org/abs/2312.03689
• Webster, K., Wang, X., Tenney, I., Beutel, A., Pitler, E., Pavlick, E., Chen, J., Chi, E., & Petrov, S. (2020). Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032. https://arxiv.org/abs/2010.06032
• Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022a). Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. https://arxiv.org/abs/2201.11903
• Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022b). Emergent abilities of large language models. arXiv preprint arXiv:2206.07682. https://arxiv.org/abs/2206.07682
• Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021). Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. https://arxiv.org/abs/2112.04359
• Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022). Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 214 - 229). ACM. https://doi.org/10.1145/3531146.3533088
• Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690. https://arxiv.org/abs/2102.09690
• Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. https://arxiv.org/abs/2307.15043