Deep Learning Systems

Overview

Earlier chapters treat fairness as a property of a model's outputs: the selection rates, error rates and calibration of a score or a decision across groups (Chapters 14 to 16 and 39). These criteria remain valid for deep neural networks. What changes is where unfairness originates and where it can be corrected. A deep network does not only draw a decision boundary in a fixed feature space. It learns a sequence of internal representations, and the content of those representations is decided by optimization, not by a designer. Information about a protected attribute can enter these representations through proxies, be strengthened through the layers, and remain available to the output even when the attribute itself was never an input.

Specialized architectures also bring their own bias patterns. Large language models can produce discriminatory text while passing classification-based fairness tests (Chapter 67). Recommendation systems amplify preferences through feedback loops that a single snapshot does not reveal: a movie recommender can sort users into demographic bubbles while click rates stay equal across groups (Chapter 66).

Vision models carry biases that vary with layer depth and with the feature extraction method, and a face analysis system can perform well for broad demographic groups yet fail badly at a particular intersection (for example, working for women and for people with darker skin on average while failing for older women with darker skin). Intersectional failures of this kind have been documented for commercial gender classifiers (Buolamwini and Gebru, 2018; Chapter 68). Multimodal systems that process text, images and audio together must, in addition, keep fairness consistent across modalities (Chapter 68). Across all of these architectures, data preprocessing needs architecture-aware bias detection, model architectures embed different inductive biases, evaluation needs metrics that capture architecture-specific failure modes, and monitoring needs dashboards and alerts that track the fairness indicators relevant to the architecture (Chapters 61 to 63).

This chapter covers the deep learning foundations shared by these architectures. It explains five mechanisms (representation learning, layer-specific bias, gradient-based training, transfer learning and opacity), presents the mitigation families that act on each, and sets out an evaluation approach that looks inside the network as well as at its outputs. It ends with an illustrative university admissions case and common questions. General in-processing methods are treated in Chapters 51 to 54; this chapter shows how they are adapted to deep architectures.

Learning objectives:

• Explain how representation learning, layer depth, training dynamics and transfer learning create fairness problems that simple models do not show.
• Describe fair representation learning and adversarial debiasing, including the gradient update of Zhang, Lemoine and Mitchell, and state their known limitations.
• Design a training objective that combines task loss with fairness terms and handles conflicting gradients.
• Assess pretrained components for inherited bias and choose between full fine-tuning, targeted fine-tuning and adapters.
• Evaluate a deep model with representation probing, attribution comparison, counterfactual consistency and intersectional metrics, and interpret each correctly.

Key questions

• How do deep learning architectures create fairness challenges that simple classification models do not exhibit?
• Which fairness strategies address these architecture-specific challenges without giving up more predictive performance than necessary?
• How can an assessor tell whether a network has merely hidden protected information from one test or has actually stopped using it?
• What must be checked before a pretrained model is adopted as a component of a high-stakes system?

Definitions and terminology

Representation (embedding). The vector h = φ(x) that a network computes from an input x at some internal layer. The encoder φ is learned during training. The final layers map h to a score R or prediction Ŷ.

Representation learning. Training in which the network learns its own features rather than receiving them from a designer. The learned features abstract from surface inputs, can create correlations that are not visible in any single input, can encode protected attributes implicitly, and interact nonlinearly.

Leakage. The extent to which the protected attribute A can be recovered from a representation h or from the model's outputs. It is measured by training an auxiliary classifier, a probe, to predict A from h.

Fair representation learning. Learning an encoder whose representations retain information useful for the task while removing, or limiting, information about A (Zemel et al., 2013).

Adversarial debiasing. Training a predictor together with an adversary that tries to predict A from the predictor's representation or output; the predictor is rewarded for defeating the adversary (Edwards and Storkey, 2016; Zhang et al., 2018).

Bias amplification. A trained model exhibits a stronger association between a label and a protected attribute than the training data contain (Zhao et al., 2017).

Transfer learning. Reusing a model pretrained on one task or corpus as the starting point for another task, by fine-tuning all its weights, part of them, or small added modules (adapters).

Inherited bias. Associations learned during pretraining that persist in a downstream model after fine-tuning.

Worst-group performance. The lowest value of a performance metric across a predefined set of groups, the quantity that group distributionally robust optimization targets (Sagawa et al., 2020).

Counterfactual consistency. The share of cases whose prediction does not change when only the protected attribute, or a direct reference to it in the input, is changed (Garg et al., 2019; Chapter 32 for the causal concept).

Why deep learning changes the fairness problem

Standard fairness methods were mostly developed for models with a fixed, human-defined feature space, such as logistic regression on tabular credit data. In that setting an assessor can list the features, check each for proxy relationships with A (Chapter 24), and constrain or post-process a single score. Several of these assumptions fail for deep networks. A constraint designed for a linear model, such as decorrelating a weight vector from a protected attribute, has no direct counterpart when the decision depends on millions of parameters through many nonlinear layers.

The first difference is that the features are learned. A university admissions model that reads essays, activity lists and school information can form internal features that track socioeconomic status through writing style, the type of extracurricular activities and the profile of the school. No single input variable is a recognizable proxy, so a feature-by-feature review finds nothing, yet the learned representation can separate applicants by background and drive differences in predictions.

The second difference is that bias can change in strength as information passes through the network. Zhao et al. (2017) showed that in a visual recognition dataset the activity cooking was over 33 percent more likely to involve women than men in the training set, and that a trained model amplified the disparity to 68 percent at test time. Wang et al. (2019) found that even when a dataset was balanced so that each label co-occurred equally with each gender, the learned models still amplified the association between labels and gender, roughly as much as without balancing. Balancing the data, a standard remedy for simpler models, did not by itself prevent the network from encoding the protected attribute.

The third difference is that much of a modern network is not trained on the task data at all. Pretrained language and vision encoders bring associations from their original corpora. Word embeddings trained on news text encode gender stereotypes as a geometric direction (Bolukbasi et al., 2016); standard embeddings replicate a range of human biases measured by the Implicit Association Test (Caliskan, Bryson and Narayanan, 2017); and image representations learned without labels from ImageNet reproduce racial, gender and intersectional biases documented in social psychology (Steed and Caliskan, 2021).

Representation learning and fairness

Traditional fairness approaches often assume that the relation between features and the protected attribute is fixed and inspectable. Representation learning breaks that assumption: the network transforms its inputs into new representations optimized for the task, and this transformation can amplify an existing disparity, mask it from simple tests, or create a new one. Four properties of learned representations matter for fairness:

1. They abstract from surface features, so a protected attribute removed from the inputs can be reconstructed from higher-level patterns.
2. They create emergent correlations, connections between inputs that no single input shows.
3. They can encode protected attributes implicitly, even when those attributes were excluded from the data.
4. They interact nonlinearly, so a constraint that is linear in the features or in one layer may not bind the behavior of the whole network.

The same property that causes the problem also offers a remedy. Zemel et al. (2013) formulated fairness as the search for a representation with two competing goals: to encode the data as well as possible while obfuscating information about membership in the protected group. Their learned intermediate representation could also be reused for other classification tasks. This idea, that a representation can keep task-relevant information while removing group information, underlies most of the deep learning methods in this chapter. The central difficulty is to steer representations toward fairness without discarding the information the task legitimately needs, especially where that information is itself correlated with A.

Representation learning affects fairness at every depth of the network. Early layers extract features from the inputs, middle layers transform them into increasingly abstract representations, and final layers map those representations to outputs. Bias can emerge, or be amplified, at any of these stages, which is why the next subsection distinguishes them.

Layer-specific bias patterns

Output metrics treat the network as a black box: they show whether outcomes differ, not where in the network the difference is produced. Distinguishing the stages of processing helps to locate the source and to choose an intervention:

1. Input layer bias: the initial encoding reflects biases in the data distribution, such as underrepresentation of a group or historically skewed labels (Chapter 25).
2. Hidden layer bias: nonlinear transformations can create or strengthen group-dependent features.
3. Representation layer bias: the latent space encodes protected information that downstream layers can use.
4. Decision layer bias: the final mapping from representations to outputs, including thresholds, produces disparate outcomes.

Each stage has a matching family of interventions. Input-level debiasing (reweighting, resampling, augmentation; Chapters 47 to 50) addresses correlations in the data. Regularization of hidden layers limits the flow of group information. Adversarial methods act on the representation layer. Threshold adjustment and other post-processing act at the decision layer (Chapters 55 to 57). Madras et al. (2018) showed that the adversarial objective attached to the representation must match the fairness criterion sought: they connected demographic parity, equalized odds and equal opportunity to different adversarial objectives and showed that this choice is crucial to fair prediction.

The table below maps the stages to the evidence an assessor can collect and the interventions that act there.

Stage	Typical mechanism	Evidence to collect	Interventions
Input	Skewed sampling or labels	Group counts, label rates by group	Reweighting, resampling, augmentation
Hidden layers	Nonlinear proxy features	Probes at several depths	Regularization, information bottlenecks
Representation	Encoded protected attribute	Probe leakage, clustering by group	Adversarial or projection methods
Decision	Disparate thresholds or scores	Selection and error rates by group	Constraints, threshold adjustment
Pretrained parts	Inherited associations	Association tests, benchmark probes	Targeted fine-tuning, adapters

Source: compiled for this compendium from the mechanisms described in this chapter.

Source: validant.ai.

Gradient-based fairness optimization

Preprocessing and post-processing treat the trained model as fixed. Deep learning allows fairness to be written directly into the training objective and optimized by the same gradient methods that optimize accuracy. Four techniques are common:

1. Multi-objective optimization: the loss combines the task loss with one or more fairness penalties.
2. Dynamic weighting: the weights of the loss components change during training, for example by increasing the fairness weight once the task has been partly learned.
3. Fairness-aware regularization: penalties on representations or parameters that carry group information, possibly restricted to specific layers.
4. Gradient projection: updates are modified so that a step that improves one objective does not undo progress on another.

Gradient-based fairness touches every stage of development: at architecture design it determines the structure of the loss function, during training it guides each parameter update, and at fine-tuning it allows fairness to be improved without retraining the model from scratch.

A generic penalized objective, with task loss L_task, a differentiable fairness penalty L_fair and a weight λ ≥ 0, is:

Rate-based fairness criteria, such as differences in selection or true positive rates, are not differentiable in the model parameters. Cotter et al. (2019) addressed this by expressing such goals as rate constraints and solving the constrained problem as a two-player game between the model and Lagrange multipliers, using differentiable proxy constraints for the model player and the original constraints for the multiplier player (the proxy-Lagrangian). The solution is a stochastic classifier, which can be reduced to a mixture of at most m + 1 models for m constraints (Cotter, Jiang and Sridharan, 2019). Chapters 51 and 52 treat constrained training in general.

Adversarial debiasing is the most widely used gradient-based technique for deep models; Chapter 53 gives its full treatment (objective, information-theoretic guarantees, training dynamics and leakage audits), and this section summarizes what matters for deep architectures. Zhang, Lemoine and Mitchell (2018) train a predictor with weights W and loss L_P, and an adversary with weights U and loss L_A that predicts the protected attribute from the predictor's output. For demographic parity, the adversary sees only the predicted output; for equalized odds, it also receives the true label Y; for equal opportunity, its training set is restricted to examples with a given value of Y. The adversary is updated to minimize L_A. The predictor is updated with the modified gradient:

The middle term removes the component of the predictor's gradient that would help the adversary; the last term, weighted by a hyperparameter α, actively pushes the predictor to increase the adversary's loss. Without the projection the predictor can end up helping the adversary; without the last term it never tries to hurt it (Zhang et al., 2018). The authors prove that if predictor and adversary converge, the adversary gains no advantage from the predictor's output, under the assumption that the adversary's loss is convex in its own weights and concave in the predictor's weights; they note themselves that these assumptions are not satisfied in most uses involving neural networks, so for deep models the guarantee is a motivation, not an assurance. The same paper reports that training is sensitive: wrong hyperparameters lead to quick divergence, and in the income prediction experiment the authors increased α over time (α = √t, with step size proportional to 1/t) to reach equality of odds rather than a result closer to demographic parity.

A more general adversarial formulation acts on the internal representation h = φ(x) rather than on the output. The encoder and predictor minimize the task loss while the adversary g maximizes its ability to recover A, a min-max problem used to learn fair representations by Edwards and Storkey (2016):

In words: the encoder φ and predictor f are chosen to keep the task loss ℓ small and the adversary's loss ℓ_A large, while the adversary g is chosen to keep its own loss small; λ sets the trade-off. In practice the two players are trained by alternating updates or with a gradient reversal layer, a device introduced for domain adaptation that flips the sign of the adversary's gradient as it flows into the encoder (Ganin et al., 2016). Beutel et al. (2017) found that only a small amount of data with known protected attributes is needed to train such an adversary, and that the distribution of that data drives which notion of fairness the adversary enforces.

Worked example: one projected update

Suppose that at one training step the predictor's gradient is ∇_W L_P = (2, 3) and the adversary's gradient with respect to the predictor's weights is ∇_W L_A = (1, 0), with α = 1. A plain gradient step moves the weights in the direction −(2, 3). To first order this changes the adversary's loss by (1, 0) · (−2, −3) = −2 per unit step, so the plain step helps the adversary.

The projection of the predictor's gradient onto the adversary's gradient is (2, 0). The modified gradient is (2, 3) − (2, 0) − 1 × (1, 0) = (−1, 3), and the step moves in the direction (1, −3). To first order the adversary's loss now changes by (1, 0) · (1, −3) = +1, so the step hurts the adversary, while the predictor's loss changes by (2, 3) · (1, −3) = −7, so the step still improves the task. The plain step would have reduced the predictor's loss faster (by 13 rather than 7 per unit step): the difference is the price paid, at this step, for not leaking information to the adversary. When the task gradient and the adversary's gradient are more closely aligned, the modified step can even increase the predictor's loss; Chapter 53 works through such a case.

Worked example: published results of adversarial debiasing

Zhang et al. (2018) applied their method to income prediction on the UCI Adult data, with sex as the protected attribute and equality of odds as the target. The predictor in this experiment was a logistic regression, not a deep network: the method is model-agnostic, and the experiment shows what the adversarial game achieves, not how it behaves in a deep architecture. They report confusion matrices on the test set with and without debiasing; the rates below are recomputed from those matrices (Chapter 53 derives further quantities, such as the mutual information between sex and the prediction, from the same tables).

Group and model	FPR	FNR	TPR	Selection rate
Women, without debiasing	0.025	0.449	0.551	0.082
Women, with debiasing	0.065	0.446	0.554	0.118
Men, without debiasing	0.092	0.367	0.633	0.254
Men, with debiasing	0.070	0.435	0.565	0.219

Source: recomputed from the confusion matrices in Zhang, Lemoine and Mitchell (2018), Table 3 (n = 5,421 women and 10,860 men in the test set).

Before debiasing, the false positive rate of men was 0.092 against 0.025 for women, and the false negative rate of women was 0.449 against 0.367 for men. After debiasing the rates are close: FPR 0.065 against 0.070 and FNR 0.446 against 0.435. The authors report that neither remaining difference is statistically significant (two-proportion z-test, p = 0.25 for Y = 0 and p = 0.62 for Y = 1), and that overall accuracy fell from 86.0 to 84.5 percent, which the recomputation confirms (0.860 and 0.845).

Two further lessons follow from the recomputed table. First, equalized odds was reached by moving both groups: the false positive rate of women rose and the true positive rate of men fell. Equalizing error rates does not mean improving the disadvantaged group, and the change in each group's rates should be reported, not only the gap (Chapter 20). Second, the selection rates remain unequal after debiasing (0.118 against 0.219), as expected, because the base rates differ (0.109 of women and 0.300 of men in the test set have the positive label) and the method enforces separation, not independence (Chapters 14, 15 and 19).

Source: validant.ai.

Transfer learning and inherited bias

Most deep systems are built on pretrained encoders, such as language models for text, image encoders for photographs or speech models for audio, which are then fine-tuned on task data. The pretrained component brings capabilities that task data alone could not provide, and it also brings associations learned from its original corpus. Four mechanisms create fairness problems:

1. Embedding bias: inherited word or visual embeddings carry demographic associations (Bolukbasi et al., 2016; Caliskan et al., 2017; Steed and Caliskan, 2021).
2. Domain shift: the pretrained model is applied to a population it was not trained for, so its representations may be less informative for some groups (Chapter 34).
3. Hidden associations: correlations from pretraining persist after fine-tuning and influence predictions in ways the fine-tuning data do not show.
4. Representation mismatch: features optimized for the original objective do not align with the fairness needs of the new task.

For example, an admissions system built on a language model pretrained on internet text may inherit associations that disadvantage non-standard English or expressions typical of international applicants, and so disadvantage first-generation and international candidates even when the fine-tuning data are balanced. Zhao et al. (2019) found that the contextualized embeddings of ELMo systematically encode gender information, encode it unequally for male and female entities, and that a coreference system built on them inherits the bias, which they could remove on the WinoBias benchmark with two mitigation methods. The general point is that inherited bias must be measured in the downstream system, not assumed away because the task data look balanced.

Transfer affects three moments of development. At model selection, the bias profile of candidate pretrained models should be part of the choice. At fine-tuning, the team decides how strongly to adapt representations and whether to target known problematic associations. At evaluation, the system must be tested for inherited biases that the current training data would not have produced.

Three families of countermeasures exist. Embedding transformation modifies representations directly: Bolukbasi et al. (2016) identify a gender direction and remove it from gender-neutral words, and iterative nullspace projection repeatedly trains linear classifiers for the protected attribute and projects the representations onto their nullspace, so that the attribute can no longer be linearly separated (Ravfogel et al., 2020). Targeted fine-tuning trains on data designed to break specific associations, such as counterfactually augmented text. Adapters insert small trainable modules between frozen pretrained layers (Houlsby et al., 2019). Lauscher, Lüken and Glavaš (2021) trained dedicated debiasing adapters on a counterfactually augmented corpus while keeping BERT frozen, found them effective on intrinsic and extrinsic bias measures, and found that the debiasing was retained when task adapters were added and trained downstream.

Explainability and fairness validation

Deep models resist direct inspection, which complicates both assessment and the targeting of interventions. Explainability methods (Part VIII) provide four kinds of visibility:

1. Feature attribution: which inputs drive predictions, compared across groups, for example with integrated gradients (Sundararajan, Taly and Yan, 2017) or SHAP values (Lundberg and Lee, 2017).
2. Activation analysis: which units or layers respond differently to inputs from different groups.
3. Counterfactual exploration: how predictions change under controlled modifications of the input, such as replacing a name or an identity term.
4. Representation visualization: whether the latent space clusters by group.

Attribution comparisons can reveal disparities that aggregate metrics hide. An admissions model can show similar acceptance rates across groups while relying mainly on essay features for underrepresented applicants and mainly on academic records for majority applicants; the different decision basis is visible only when attributions are compared by group. Explainability serves fairness throughout the lifecycle: during development it informs the choice of architectures whose behavior can be inspected, during validation it adds evidence beyond aggregate metrics, and during deployment it supports more meaningful explanations to the people affected and to overseers. Doshi-Velez and Kim (2017) note that explanations are often used to assess other criteria such as non-discrimination, and argue that interpretability itself needs rigorous definition and evaluation.

Explanations are evidence with known weaknesses. Slack et al. (2020) showed that perturbation-based explanation methods such as LIME and SHAP can be fooled: a scaffolding technique lets a classifier keep making biased predictions on real inputs while its post hoc explanations look innocuous, demonstrated with deliberately racist classifiers on real datasets including COMPAS. An attribution comparison that shows no reliance on a protected attribute is therefore not proof of non-discrimination, especially when the party being assessed controls the model (Chapters 77 and 82).

Domain modeling perspective

From a domain modeling perspective, fairness in deep learning extends beyond the decision boundary to representation spaces, optimization dynamics and the flow of information through layers. Representations control which information reaches the output; attention mechanisms determine which inputs receive focus; activation functions shape the nonlinear behavior. Together they create interactions that simpler models do not exhibit.

Several stakeholders hold different parts of this domain. Data scientists design architectures and objectives; engineers implement and serve the models; domain experts validate whether outputs and explanations make sense; and the people affected by the system experience its outcomes. Each sees a different aspect of how fairness should manifest, and an assessment of a deep system needs all of them (Chapter 106).

Intersectionality offers a parallel. Crenshaw (1989) argued that the experience of Black women cannot be understood by adding race discrimination and sex discrimination considered separately. Deep networks show an analogous structural property: their behavior emerges from interactions between components and cannot be predicted from each component in isolation. The analogy is a reason to evaluate the system as a whole, not a claim about discrimination law.

Intersectionality in deep learning

Deep learning practice often treats protected attributes one at a time, optimizing separately for gender and for ethnicity. This misses disparities that exist only at intersections. Buolamwini and Gebru (2018) evaluated three commercial gender classification systems on a benchmark balanced by gender and skin type and found darker-skinned women to be the most misclassified group, with error rates up to 34.7 percent, against a maximum of 0.8 percent for lighter-skinned men, a result that single-attribute analysis would not have exposed. Their audit also found the two existing benchmarks to be overwhelmingly composed of lighter-skinned subjects (79.6 and 86.2 percent).

Embedding intersectional principles in a deep system involves five practices:

• design architectures and representations that can represent intersectional experience without conflating distinct groups;
• use loss functions that penalize disparities at intersections explicitly, not only along single attributes;
• build validation that detects intersectional fairness problems;
• regularize in a way that preserves information about intersectional patterns where it is legitimately needed;
• check that representation learning does not abstract away factors that matter for particular intersections.

These practices raise practical problems. The optimization objective becomes more complex. Validation needs enough data at each intersection to support statistically valid statements, which is rarely available for small groups (Chapters 29, 41 and 43). Architectures should support intersectional fairness without building a separate pipeline for every demographic combination. Two families of methods address this directly. Group distributionally robust optimization minimizes the worst-case training loss over a predefined set of groups:

Sagawa et al. (2020) found that applied naively to overparameterized networks this fails, because such networks fit the training data perfectly and the worst-group training loss also vanishes; combined with stronger regularization or early stopping, group DRO improved worst-group accuracy by 10 to 40 percentage points on one language and two image tasks while keeping average accuracy high. Multiaccuracy auditing and post-processing take a complementary route: with black-box access to a predictor and a small labeled audit set, Kim, Ghorbani and Zou (2019) improve accuracy on identifiable subgroups, including subgroups such as Black women, even when the sensitive features are not given to the algorithm. Creager et al. (2019) learn representations that can be adjusted at test time to achieve subgroup demographic parity for several sensitive attributes and their conjunctions.

Worked example: a gap that only the intersection shows

An admissions model is evaluated for equal opportunity: the true positive rate among applicants who would succeed. Qualified applicants are split by gender and by first-generation status.

Group	Qualified applicants	TPR	95% interval half-width
Men, continuing-generation	300	0.80	0.045
Men, first-generation	100	0.76	0.084
Women, continuing-generation	150	0.84	0.059
Women, first-generation	50	0.64	0.133

Source: constructed example; normal-approximation intervals.

The marginal TPR is 0.79 for men ((240 + 76)/400) and 0.79 for women ((126 + 32)/200), so the gender gap is zero. By first-generation status the TPRs are 0.72 and 0.81, a gap of 0.09. At the intersection, first-generation women have a TPR of 0.64 against 0.80 for continuing-generation men, a gap of 0.16. A single-attribute audit by gender would report no disparity at all. The last column shows the statistical difficulty: with 50 qualified applicants the 95 percent interval for first-generation women is about ±0.13, so the intersectional gap must be confirmed with an appropriate test or more data before it is acted on (Chapter 41).

Practical considerations

Implementation framework

Implementing fairness in a deep learning system involves four strands of work.

1. Architecture selection and modification. Choose architectures that allow intervention at the representation level; add fairness components such as an adversarial head; select activation functions and layer structures that support control over representations; and weigh model complexity against the need to inspect and constrain it.
2. Fair representation learning. Remove protected information from representations with adversarial or projection methods; learn disentangled representations that separate task features from demographic correlates (Creager et al., 2019; Locatello et al., 2019); use contrastive or counterfactual augmentation to make representations of matched individuals similar; and use domain adaptation for groups that are underrepresented in the training data.
3. Gradient-based fairness optimization. Design loss functions with explicit fairness terms; weight them dynamically during training; regularize specific layers where leakage arises; and use validation results to balance the objectives.
4. Transfer learning debiasing. Assess pretrained models for inherited bias; fine-tune with data aimed at problematic associations; transform embeddings where appropriate; and place adaptation layers between pretrained components and task-specific layers.

The framework enhances standard deep learning practice rather than replacing it: architecture selection becomes fairness-aware, training includes fairness objectives, and validation examines fairness alongside accuracy. Representation methods and constrained optimization complement each other, because one changes what the network learns and the other constrains what it does with it (Madras et al., 2018; Cotter et al., 2019). These techniques can be implemented with standard deep learning frameworks; they require care in tuning rather than new development paradigms.

Source: validant.ai.

Implementation challenges

Four pitfalls recur.

1. Gradient conflicts. Fairness and task objectives can produce gradients that point in opposing directions, which slows or prevents convergence. Remedies are careful loss design, progressive training that introduces fairness terms gradually, and gradient projection. Yu et al. (2020) proposed, for multi-task learning, to project a task's gradient onto the normal plane of any other task's gradient with which it conflicts; the Zhang et al. (2018) update applies the same idea to the predictor and the adversary.
2. Adversarial instability. The adversary can dominate or collapse, and training can diverge (Zhang et al., 2018). Mitigations include sizing the adversary appropriately, scheduling learning rates and the adversarial weight, and alternating updates so that neither player dominates.
3. Computational overhead. Adversaries, extra loss terms and additional validation increase training cost. Targeted regularization instead of a full adversarial structure, adapters instead of full fine-tuning, and reuse of pretrained components reduce it. The cost should be measured on the actual system and stated before deployment.
4. Evaluation complexity. Opacity makes validation difficult. Layered evaluation, which combines output metrics with representation probing, attribution comparison and counterfactual tests, addresses it (next section).

A specific caution concerns adversarial removal. Elazar and Goldberg (2018) showed that while the adversary in adversarial training reached chance-level accuracy on development data, a separate classifier trained afterwards on the frozen representations still recovered the demographic attribute with substantially higher accuracy, consistently across tasks, attributes and datasets. Their conclusion was not to rely on adversarial training alone to achieve representations invariant to sensitive features. The training adversary's accuracy is therefore not a valid measure of leakage; leakage must be measured with a fresh probe.

When communicating with stakeholders, the emphasis should be on concrete consequences rather than architectural detail. For executives, representation-level fairness reduces the risk of disparities that create regulatory and reputational exposure. For product teams, fairness interventions can often be introduced with a limited and measurable loss of performance, which should be reported rather than assumed. Implementation requires additional training time, validation data with demographic annotations (whose collection has legal limits; see the LEGAL box below), validation steps beyond standard performance testing, and tools for inspecting deep networks.

Evaluation approach

A deep model should be assessed on five dimensions.

1. Representation fairness: how well the protected attribute can be predicted from learned representations by a probe trained after the model is frozen.
2. Attribution equity: whether attribution patterns for key features differ substantially across groups.
3. Intersectional performance: standard fairness metrics disaggregated across intersections, with uncertainty.
4. Counterfactual consistency: how often predictions stay the same when only the protected attribute, or a reference to it, is changed.
5. Fairness and performance trade-off: the Pareto frontier of accuracy against fairness across model variants (Chapters 20 and 54).

Madras et al. (2018) motivate the first dimension directly: if representations are passed to third parties with unknown objectives, the fairness of the representation matters as well as the fairness of one model built on it. Evaluating both what the network has learned and what it does with that learning prevents bias from hiding in latent spaces.

Counterfactual consistency over a test set of n cases, where x_i^A←a' denotes case i with the protected attribute or its textual reference changed to another value, is:

For text, Garg et al. (2019) defined counterfactual token fairness by substituting identity terms, for example to detect a toxicity classifier that labels "Some people are gay" as toxic and "Some people are straight" as non-toxic, and found that blindness and counterfactual logit pairing improved it without harming classifier performance, with varying trade-offs against group fairness. Changing a token is a test of sensitivity to that token, not a full causal counterfactual: other features caused by the attribute are left unchanged (Chapter 32).

Indicative starting targets for these measures are: a probe accuracy no better than 60 percent when predicting protected attributes from representations; attribution differences across groups below 15 percent for key features; equal opportunity differences below 5 percentage points for all intersectional groups; counterfactual consistency above 90 percent for the primary task; and performance of at least 95 percent of the unconstrained baseline. The probe target needs correction. Raw probe accuracy must be compared with the accuracy of always predicting the majority group. If 70 percent of applicants belong to group a, a probe that ignores the representation already reaches 70 percent, so the 60 percent target is unattainable and meaningless. Leakage should be reported as balanced accuracy or AUC of the probe, where 0.5 means no recoverable information, with a confidence interval.

Worked example: reading a probe correctly

A frozen admissions encoder is probed on 1,000 held-out applicants, 700 in group a and 300 in group b. The probe identifies 180 of the 300 members of group b (60 percent) and 588 of the 700 members of group a (84 percent). Its accuracy is (180 + 588)/1,000 = 0.768. Against the majority baseline of 0.700 this is a modest gain; its balanced accuracy is (0.60 + 0.84)/2 = 0.72, well above the 0.5 of a representation with no group information. The representation leaks group membership: a probe that recovers it for six in ten members of group b gives every downstream user of the representation the means to treat that group differently.

Case study: a university admissions system

The following scenario is an illustrative composite that shows how the mechanisms and methods of this chapter combine. Its figures are illustrative, not empirical findings.

Context

A large public university plans an AI-based admissions system to bring consistency and efficiency to its process. The system analyzes academic records, essays, recommendation letters and extracurricular activities to predict the likelihood that an applicant will succeed in an undergraduate program. The task requires semantic understanding of unstructured text, pattern recognition across heterogeneous inputs and an assessment of potential. Stakeholders are the university administration, admissions officers, prospective students, faculty representatives, the diversity office and state education regulators. In the European Union such a system is high-risk under the AI Act (LEGAL box above); in the United States the use of race in admissions decisions is subject to strict legal limits (see the legal note in the admissions case study of Chapter 12, and Chapters 83 and 88), which is distinct from using demographic data to audit a model.

Initial deep learning models predicted well but raised fairness concerns. Although race and gender were excluded as inputs, Black and Hispanic applicants received systematically lower scores than applicants with similar academic records. First-generation applicants received lower predictions, particularly when their essays discussed financial hardship. International applicants with non-Western names received lower scores for recommendation letters of similar content. Standard fairness interventions that had worked for simpler models did not remove these patterns.

Problem analysis

The data science team found five architecture-specific problems.

1. Representation entanglement. The BERT-based text encoder produced representations in which socioeconomic and ethnic information was encoded; writing style and topic choice were strongly associated with background.
2. Layer-specific amplification. Probes showed demographic separability that was small in early layers and grew through the middle layers.
3. Transfer learning inheritance. The pretrained language models used for essays and recommendation letters carried biases from their pretraining corpora that persisted after fine-tuning on the university's data.
4. Attention disparity. Attention patterns differed by group: for advantaged groups the model focused on academic achievements, for underrepresented groups disproportionately on hardship narratives and non-academic content.
5. Intersectional blindness. Performance was especially poor at intersections such as first-generation students from rural areas and women applying to STEM programs, patterns invisible to single-attribute measures.

These findings match the mechanisms documented in the literature: contextual embeddings encode gender and other attributes and pass them to downstream systems (Zhao et al., 2019), and deep models amplify associations present in their training data (Zhao et al., 2017). The stakes were high because admissions decisions shape access to education, life opportunities and the diversity of the institution, under regulatory oversight and public scrutiny.

Solution

The university implemented a strategy matched to its architecture.

1. Architecture modification: explicit fairness components; an adversarial head competing with the primary task; disentangled representation learning separating task-relevant features from demographic correlates; and parallel attention mechanisms with weight averaging to reduce attention disparity.
2. Fair representation learning: adversarial debiasing of the embedding space; counterfactual regularization so that matched applicants receive consistent representations; contrastive learning for demographically balanced embeddings; and interpolation between demographic groups to create balanced training instances.
3. Gradient-based optimization: a multi-component loss combining the task loss (accuracy of the admission prediction), an adversarial loss (low predictability of demographics), a consistency loss (similar applicants receive similar embeddings) and an intersectional loss (penalties for underperformance at intersections); curriculum learning with progressively stronger fairness emphasis; and gradient projection to prevent oscillation between fairness and accuracy.
4. Transfer learning debiasing: evaluation of pretrained components for inherited bias; fine-tuning aimed at problematic associations; embedding transformation; and domain-specific adaptation layers between pretrained embeddings and task components.
5. Validation enhancement: representation probing for demographic encoding; attribution comparison across groups; counterfactual test suites that vary protected attributes while keeping qualifications fixed; and an intersectional evaluation framework.

In the notation of this chapter the training objective was:

with weights λ₁, λ₂ and λ₃ increased over the course of training. Rather than inventing a new architecture, the team augmented a standard one with fairness components, targeting specific network components instead of treating the model as a black box.

Outcomes and lessons

In the illustrative scenario, the demographic parity gap fell from 24 to 3 percentage points and the equal opportunity gap from 19 to 4; previously disadvantaged intersections reached comparable performance; and attention patterns became more consistent across groups. Overall accuracy was 97 percent of the unconstrained baseline, calibration stayed good across groups, the model generalized better to new application cycles, and attributions became more consistent. Admissions officers reported more trust in the recommendations, the diversity office confirmed improved outcomes for underrepresented applicants, regulators accepted the system on the basis of its fairness documentation, and student feedback indicated a more equitable experience. Any real deployment would need to verify such results with a held-out probe, confidence intervals and monitoring over several cycles.

Four lessons emerged:

1. Architecture matters for fairness. Generic approaches that worked for simpler models did not work for the deep network; strategies aimed at how the network processes information were needed.
2. Representation-level intervention complemented output correction. Changing how the model represented applicants produced more stable fairness than adjusting final decisions alone.
3. Transfer learning requires special attention. Pretrained components introduced biases that persisted after fine-tuning and had to be addressed explicitly.
4. Intersectional analysis reveals hidden patterns. Single-attribute disaggregation missed the problems at specific intersections; only explicit intersectional evaluation revealed them.

Competing interpretations and limitations

Removal versus use. Removing protected information from representations can conflict with legitimate uses of correlated information. A hardship narrative may be relevant evidence of resilience, and removing everything correlated with background can remove it. Whether a representation should be invariant to A, or only the decision should satisfy a criterion such as equal opportunity, is a normative choice that depends on the criterion selected (Chapter 22). Madras et al. (2018) show that different adversarial objectives correspond to different criteria; there is no single "fair representation".

Adversarial removal is incomplete. A training adversary at chance level does not imply invariance (Elazar and Goldberg, 2018), and linear projection methods remove linearly decodable information but not necessarily nonlinear encodings (Ravfogel et al., 2020 describe their guarantee in terms of linear separability). Claims of invariance should state the probe class against which they were tested.

Disentanglement needs evidence. Disentangled representations are attractive because they promise to separate task factors from demographic factors. Locatello et al. (2019) show that predictions from representations can be unfair even when the target and the sensitive variable are independent, and found several disentanglement scores correlated with increased fairness across more than 12,600 trained models, which suggests a useful property rather than a guarantee.

Representation versus output. A common claim is that representation-level methods outperform output correction. The evidence is mixed: output methods such as multiaccuracy post-processing can improve subgroup accuracy with black-box access only (Kim et al., 2019), and in-processing methods trade fairness and accuracy in ways that depend on the data. The defensible position is that representation evidence adds information that output metrics cannot provide, and that the choice of intervention should be made empirically on the system at hand.

Explanations as evidence. Attribution methods can be manipulated (Slack et al., 2020) and depend on choices such as baselines; they support but do not settle fairness claims.

Data about protected attributes. Most methods in this chapter need protected attributes at training or audit time. Beutel et al. (2017) show that small amounts of such data can suffice for adversarial training, but collection and processing remain legally constrained (Article 4a of the AI Act, formerly Article 10(5), in the European Union; Chapters 24 and 86).

Common questions

How can a deep model be made fairer without sacrificing performance?
By targeting interventions where bias arises instead of constraining the whole model uniformly. Start with an architecture that allows control over representations. Train the task first and introduce fairness components progressively, so that the model has learned useful features before fairness pressure is applied. Prefer multi-objective optimization with dynamic weighting to fixed penalties, and apply fairness at the representation level as well as at the output. Measure the cost rather than assume it: in Zhang et al. (2018), with a logistic regression predictor on the Adult data, adversarial debiasing for equality of odds reduced overall accuracy from 86.0 to 84.5 percent while bringing error rates close to equal. Fairness and performance are not inherently opposed, but a trade-off usually exists and should be quantified on a Pareto frontier (Chapter 20).

How should bias inherited from pretrained components be handled?
With a layered approach. First, assess candidate pretrained models before integration, by measuring demographic associations in their embeddings with association tests (Caliskan et al., 2017; Steed and Caliskan, 2021) and their predictions on benchmark datasets. Second, fine-tune with data aimed at the specific associations found, rather than relying on general adaptation. Third, consider adapters, small trainable modules inserted between frozen pretrained layers, which can change representations without disturbing pretrained knowledge and can carry debiasing that survives downstream training (Houlsby et al., 2019; Lauscher et al., 2021). Finally, monitor for re-emergence of inherited bias after updates. The principle is that inherited bias needs explicit measurement and targeted countermeasures; standard fine-tuning cannot be assumed to remove it.

Does removing the protected attribute from the inputs make a deep model fair?
No. Deep models can reconstruct the attribute from other inputs and encode it internally; amplification has been observed even with balanced data (Wang et al., 2019). Fairness must be measured on outputs and representations with the attribute available for audit.

If the adversary cannot predict the protected attribute, is the representation free of it?
Not necessarily. A separate probe trained after the model is frozen can recover information the training adversary missed (Elazar and Goldberg, 2018). Leakage should be measured with fresh probes, of more than one capacity, at several layers.

What should an organization new to deep learning fairness do first?
Three steps form a minimum: evaluate pretrained components for inherited bias before integration; implement basic adversarial or constrained fairness during fine-tuning, matched to the selected criterion; and probe representations for demographic information, reporting balanced accuracy against the baseline.

Conclusions

Deep learning moves the locus of unfairness from a visible feature space into learned representations. The consequence for assessment is that the classical output criteria remain necessary but are no longer sufficient as diagnostics: they show that outcomes differ but not where the difference is produced, and they can be met while a representation still carries protected information that any downstream user could exploit. A credible assessment of a deep system combines output metrics with representation probing, attribution comparisons, counterfactual tests and intersectional analysis, each reported with its limitations.

Methodologically, the main tools are fair representation learning, adversarial debiasing matched to the chosen criterion, constrained and worst-group training objectives, and targeted treatment of pretrained components. Each is effective under conditions and fragile outside them: adversarial training can be unstable and incomplete, explanations can be manipulated, and intersectional metrics are statistically noisy for small groups. Their effects should be verified on held-out data with probes the model was not trained against.

Normatively, the choice of what to remove from a representation and which criterion to enforce is not a technical detail. It determines whose information counts and how, and it must be justified for the context and documented (Chapter 22), particularly in high-risk domains such as admissions, where the law requires bias examination and mitigation.

Open questions

• How can invariance of a representation to a protected attribute be certified against all probes of a given capacity, not only the probes tried?
• How should fairness interventions be designed so that debiasing of a pretrained component survives repeated downstream fine-tuning and model updates?
• How can intersectional fairness be enforced and verified when many intersections have too few examples for reliable estimates?
• When is protected information in a representation legitimately task-relevant, and how should that be decided and documented?
• How can explanation-based fairness evidence be made robust to manipulation by the party that controls the model?

References

• Beutel, A., Chen, J., Zhao, Z., & Chi, E. H. (2017). Data decisions and theoretical implications when adversarially learning fair representations [Poster presentation]. Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2017). https://arxiv.org/abs/1707.00075
• Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems 29. https://proceedings.neurips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html
• Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (PMLR 81, pp. 77 - 91). https://proceedings.mlr.press/v81/buolamwini18a.html
• Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183 - 186. https://doi.org/10.1126/science.aal4230
• Cotter, A., Jiang, H., Gupta, M., Wang, S., Narayan, T., You, S., & Sridharan, K. (2019). Optimization with non-differentiable constraints with applications to fairness, recall, churn, and other goals. Journal of Machine Learning Research, 20(172), 1 - 59. https://jmlr.org/papers/v20/18-616.html
• Cotter, A., Jiang, H., & Sridharan, K. (2019). Two-player games for efficient non-convex constrained optimization. In Proceedings of the 30th International Conference on Algorithmic Learning Theory (PMLR 98, pp. 300 - 332). https://proceedings.mlr.press/v98/cotter19a.html
• Creager, E., Madras, D., Jacobsen, J.-H., Weis, M. A., Swersky, K., Pitassi, T., & Zemel, R. (2019). Flexibly fair representation learning by disentanglement. In Proceedings of the 36th International Conference on Machine Learning (PMLR 97, pp. 1436 - 1445). https://proceedings.mlr.press/v97/creager19a.html
• Crenshaw, K. (1989). Demarginalizing the intersection of race and sex: A Black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. University of Chicago Legal Forum, 1989(1), 139 - 167. https://chicagounbound.uchicago.edu/uclf/vol1989/iss1/8
• Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning (arXiv:1702.08608). arXiv. https://arxiv.org/abs/1702.08608
• Edwards, H., & Storkey, A. (2016). Censoring representations with an adversary [Conference paper]. International Conference on Learning Representations (ICLR 2016). https://arxiv.org/abs/1511.05897
• Elazar, Y., & Goldberg, Y. (2018). Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 11 - 21). Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1002
• European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act). Official Journal of the European Union, L series, 12 July 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
• European Parliament and Council of the European Union. (2026). Regulation (EU) 2026/1744 of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI). Official Journal of the European Union, L, 2026/1744. https://eur-lex.europa.eu/eli/reg/2026/1744/oj
• Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2016). Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59), 1 - 35. https://jmlr.org/papers/v17/15-239.html
• Garg, S., Perot, V., Limtiaco, N., Taly, A., Chi, E. H., & Beutel, A. (2019). Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (pp. 219 - 226). ACM. https://doi.org/10.1145/3306618.3317950
• Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (PMLR 97, pp. 2790 - 2799). https://proceedings.mlr.press/v97/houlsby19a.html
• Kim, M. P., Ghorbani, A., & Zou, J. (2019). Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (pp. 247 - 254). ACM. https://doi.org/10.1145/3306618.3314287
• Lauscher, A., Lüken, T., & Glavaš, G. (2021). Sustainable modular debiasing of language models. In Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 4782 - 4797). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.findings-emnlp.411
• Locatello, F., Abbati, G., Rainforth, T., Bauer, S., Schölkopf, B., & Bachem, O. (2019). On the fairness of disentangled representations. In Advances in Neural Information Processing Systems 32. https://arxiv.org/abs/1905.13662
• Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30. https://arxiv.org/abs/1705.07874
• Madras, D., Creager, E., Pitassi, T., & Zemel, R. (2018). Learning adversarially fair and transferable representations. In Proceedings of the 35th International Conference on Machine Learning (PMLR 80, pp. 3384 - 3393). https://proceedings.mlr.press/v80/madras18a.html
• Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020). Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 7237 - 7256). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.647
• Sagawa, S., Koh, P. W., Hashimoto, T. B., & Liang, P. (2020). Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization [Conference paper]. International Conference on Learning Representations (ICLR 2020). https://arxiv.org/abs/1911.08731
• Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020). Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 180 - 186). ACM. https://doi.org/10.1145/3375627.3375830
• Steed, R., & Caliskan, A. (2021). Image representations learned with unsupervised pre-training contain human-like biases. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 701 - 713). ACM. https://doi.org/10.1145/3442188.3445932
• Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning (PMLR 70, pp. 3319 - 3328). https://proceedings.mlr.press/v70/sundararajan17a.html
• Wang, T., Zhao, J., Yatskar, M., Chang, K.-W., & Ordonez, V. (2019). Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (pp. 5309 - 5318). IEEE. https://doi.org/10.1109/ICCV.2019.00541
• Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. (2020). Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems 33. https://arxiv.org/abs/2001.06782
• Zemel, R., Wu, Y., Swersky, K., Pitassi, T., & Dwork, C. (2013). Learning fair representations. In Proceedings of the 30th International Conference on Machine Learning (PMLR 28(3), pp. 325 - 333). https://proceedings.mlr.press/v28/zemel13.html
• Zhang, B. H., Lemoine, B., & Mitchell, M. (2018). Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (pp. 335 - 340). ACM. https://doi.org/10.1145/3278721.3278779
• Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., & Chang, K.-W. (2019). Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, pp. 629 - 634). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1064
• Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017). Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (pp. 2979 - 2989). Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1323