LearnApplied LLM EngineeringBias & Fairness in LLMs

🛡️MediumAlignment & Safety

Bias & Fairness in LLMs

Build a matched-pair fairness audit for an LLM judge, measure routing gaps, and block release when evidence is too weak.

16 min read

Learning path

Step 66 of 155 in the full curriculum

LLM-as-a-Judge Evaluation Hallucination Detection & Mitigation

The previous lesson calibrated a large language model (LLM) judge that scores supported customer replies for clarity and actionability. Calibration on average isn't enough. If that judge sends equivalent requests down different routes for different customer language varieties, it can delay service for some customers even while its overall agreement score looks acceptable.

ShopFlow now wants to auto-serve a supported replacement reply when the judge is confident, and send uncertain replies to a human reviewer. This lesson builds a fairness audit for that release decision. The audit asks one controlled question: when request facts and supported remedy stay fixed, does a change in language variety alter who receives the fast path?

The code below uses invented, labeled fixtures. Its two wording variants are test conditions, not demographic groups and not a claim about any community's speech. A real language-variety audit needs representative data, informed review, privacy controls, and careful group definitions.

Fairness starts with a consequence

A model can cause two broad kinds of harm. Representational harm occurs when output stereotypes, demeans, or erases a group. Allocative harm occurs when system behavior changes access to a benefit or burden, such as whether an eligible customer gets an immediate supported answer or waits for review. This distinction is central in surveys of bias and fairness in LLM systems.^[1]

Our running case is allocative. The remedy is already authorized by the selected policy evidence. The new outcome is the route:

Decision component	Held fixed or measured?	Why it matters
Policy source and version	Held fixed	A fairness audit can't repair unsupported claims.
Replacement eligibility	Held fixed within each matched pair	Each pair should deserve the same answer.
Language-variety fixture	Varied within each pair	It's the audit condition.
Judge score and route	Measured	Unequal routing is customer impact.

The distinction from the previous chapter is important. There, we asked whether a judge agrees with reviewers. Here, we ask whether its errors and routing decisions are unevenly distributed.

Diagram showing Supported matched pairs, Calibrated judge, Slice outcomes, and Fairness gate. — Supported matched pairs, Calibrated judge, Slice outcomes, and Fairness gate.

Figure 1: Fairness evaluation begins after evidence support. It measures whether a soft judge and router create uneven outcomes across controlled slices.

A fairness audit pipeline that sends supported replacement replies through matched request pairs, an LLM judge, slice metrics, and a blocked promotion decision — Keep the supported remedy fixed, compare matched wording conditions, and treat a route gap as a release problem until evidence supports a fix.

Build a matched-pair audit

A matched pair changes one audit condition while preserving task semantics. That is harder than swapping words mechanically. Two prompts belong in a pair only after reviewers agree they describe the same customer facts and should receive the same route.

For this lab, a human reviewer has labeled ten synthetic pairs. In six pairs the reply is supported and clear enough for auto-service. In four, it should go to review because wording of the proposed reply remains unclear. Both versions in a pair share the same expected outcome.

matched-audit-fixtures.py

from collections import defaultdict
from dataclasses import dataclass
from math import sqrt

@dataclass(frozen=True)
class AuditRow:
    pair_id: str
    variant: str
    expected_auto_serve: bool
    judge_score: float
    channel: str
    evidence_passed: bool = True

THRESHOLD = 0.70

def route(row: AuditRow) -> str:
    if not row.evidence_passed:
        return "blocked_by_evidence"
    return "auto_serve" if row.judge_score >= THRESHOLD else "human_review"

# Synthetic observations: (pair, expected outcome, channel, formal score, conversational score)
observations = [
    ("p1", True, "chat", 0.92, 0.88),
    ("p2", True, "chat", 0.88, 0.73),
    ("p3", True, "chat", 0.84, 0.72),
    ("p4", True, "email", 0.78, 0.67),
    ("p5", True, "email", 0.74, 0.56),
    ("p6", True, "email", 0.66, 0.61),
    ("n1", False, "chat", 0.71, 0.60),
    ("n2", False, "chat", 0.62, 0.55),
    ("n3", False, "email", 0.60, 0.52),
    ("n4", False, "email", 0.68, 0.63),
]

rows: list[AuditRow] = []
for pair_id, expected, channel, formal_score, conversational_score in observations:
    rows.extend([
        AuditRow(pair_id, "formal", expected, formal_score, channel),
        AuditRow(pair_id, "conversational", expected, conversational_score, channel),
    ])

assert len(rows) == 20
assert all(row.evidence_passed for row in rows)
assert {
    row.pair_id: row.expected_auto_serve for row in rows if row.variant == "formal"
} == {
    row.pair_id: row.expected_auto_serve for row in rows if row.variant == "conversational"
}

print("Fixture type: synthetic matched wording audit")
print(f"Matched pairs: {len(observations)}; scored rows: {len(rows)}")
print(f"Routing threshold: {THRESHOLD:.2f}")

Output

Fixture type: synthetic matched wording audit
Matched pairs: 10; scored rows: 20
Routing threshold: 0.70

The fixture intentionally creates a failure. If every example passed, we could demonstrate arithmetic but not diagnosis.

Look for pair flips first

A pair flip is the simplest warning sign: equivalent requests receive different routes. It doesn't yet prove a population-level disparity, but it tells the team exactly which cases require investigation.

pair-flips.py

by_pair: dict[str, list[AuditRow]] = defaultdict(list)
for row in rows:
    by_pair[row.pair_id].append(row)

flips: list[tuple[str, str, str]] = []
for pair_id, pair_rows in by_pair.items():
    outcomes = {row.variant: route(row) for row in pair_rows}
    if len(set(outcomes.values())) > 1:
        flips.append((pair_id, outcomes["formal"], outcomes["conversational"]))

assert flips == [
    ("p4", "auto_serve", "human_review"),
    ("p5", "auto_serve", "human_review"),
    ("n1", "auto_serve", "human_review"),
]

print("Flipped matched pairs:")
for pair_id, formal_route, conversational_route in flips:
    print(f"  {pair_id}: formal={formal_route}, conversational={conversational_route}")

Output

Flipped matched pairs:
  p4: formal=auto_serve, conversational=human_review
  p5: formal=auto_serve, conversational=human_review
  n1: formal=auto_serve, conversational=human_review

Two eligible replies lose the fast path under the conversational condition. One unclear reply gains the fast path under the formal condition. A single approval-rate number can't explain both errors.

Choose a metric from the harm

Four group metrics appear frequently in fairness work. They answer different product questions:

Metric	Calculation	Question for the router
Selection rate	Auto-served / all requests	Does one slice receive fast service more often?
True positive rate (TPR)	Auto-served / replies reviewers say are ready	Do ready replies receive fast service equally often?
False positive rate (FPR)	Auto-served / replies reviewers say need review	Does one slice receive unsafe fast service more often?
Calibration	Observed ready rate among equal score bands	Does a `0.80` score carry same meaning by slice?

Equal opportunity compares TPR across slices. Equalized odds compares both TPR and FPR. Hardt, Price, and Srebro formalized these error-rate criteria for supervised decision systems.^[2] For ShopFlow, delayed eligible help is the main harm, so TPR gap is the primary release metric. FPR gap remains a guardrail because faster service isn't a win if it releases unclear replies.

A comparison of selection rate, true positive rate, false positive rate, and calibration for a matched wording audit, highlighting TPR gap as the primary gate. — The harm model chooses the primary metric: eligible replies shouldn't be delayed unevenly. False positive rate remains a guardrail against releasing replies that need review.

Calculate slice rates

slice-rates.py

@dataclass(frozen=True)
class Rates:
    selection: float
    tpr: float
    fpr: float
    positive_count: int
    negative_count: int

def slice_rates(slice_rows: list[AuditRow]) -> Rates:
    positives = [row for row in slice_rows if row.expected_auto_serve]
    negatives = [row for row in slice_rows if not row.expected_auto_serve]
    selected = [row for row in slice_rows if route(row) == "auto_serve"]
    true_positives = [row for row in positives if route(row) == "auto_serve"]
    false_positives = [row for row in negatives if route(row) == "auto_serve"]
    return Rates(
        selection=len(selected) / len(slice_rows),
        tpr=len(true_positives) / len(positives),
        fpr=len(false_positives) / len(negatives),
        positive_count=len(positives),
        negative_count=len(negatives),
    )

rates = {
    variant: slice_rates([row for row in rows if row.variant == variant])
    for variant in ("formal", "conversational")
}

def gap(metric: str) -> float:
    return abs(getattr(rates["formal"], metric) - getattr(rates["conversational"], metric))

for variant, result in rates.items():
    print(
        f"{variant:14} selection={result.selection:.1%} "
        f"TPR={result.tpr:.1%} FPR={result.fpr:.1%}"
    )
print(f"TPR gap={gap('tpr'):.1%}; FPR gap={gap('fpr'):.1%}")

Output

formal         selection=60.0% TPR=83.3% FPR=25.0%
conversational selection=30.0% TPR=50.0% FPR=0.0%
TPR gap=33.3%; FPR gap=25.0%

This audit fails in both directions. Among replies reviewers marked ready, the conversational condition is routed to human review more often. Among replies that need review, the formal condition is incorrectly auto-served once.

Turn metric choice into a gate

A release contract makes the choice reviewable. Thresholds below are product decisions for this lab, not universal definitions of fairness.

release-contract.py

@dataclass(frozen=True)
class FairnessContract:
    primary_metric: str
    max_tpr_gap: float
    max_fpr_gap: float
    min_positive_per_slice: int
    min_negative_per_slice: int

contract = FairnessContract(
    primary_metric="equal_opportunity",
    max_tpr_gap=0.10,
    max_fpr_gap=0.10,
    min_positive_per_slice=50,
    min_negative_per_slice=30,
)

metric_checks = {
    "TPR gap": gap("tpr") <= contract.max_tpr_gap,
    "FPR guardrail": gap("fpr") <= contract.max_fpr_gap,
}

for name, passed in metric_checks.items():
    print(f"{name}: {'PASS' if passed else 'FAIL'}")
assert not any(metric_checks.values())

Output

TPR gap: FAIL
FPR guardrail: FAIL

Tiny slices can't certify fairness

The audit found an actionable regression. It hasn't estimated production disparity. Six ready examples per wording condition are too few for a stable rate, and these fixtures don't identify a population.

One quick way to make that visible is a . The Wilson interval below gives a plausible range for each TPR under binomial sampling. It isn't a complete statistical analysis, but it prevents a tiny dataset from looking decisive.

uncertainty-and-support.py

def wilson_interval(successes: int, total: int, z: float = 1.96) -> tuple[float, float]:
    proportion = successes / total
    denominator = 1 + z * z / total
    center = (proportion + z * z / (2 * total)) / denominator
    radius = z * sqrt(
        (proportion * (1 - proportion) + z * z / (4 * total)) / total
    ) / denominator
    return center - radius, center + radius

for variant in ("formal", "conversational"):
    variant_rows = [
        row for row in rows
        if row.variant == variant and row.expected_auto_serve
    ]
    successes = sum(route(row) == "auto_serve" for row in variant_rows)
    low, high = wilson_interval(successes, len(variant_rows))
    print(
        f"{variant:14} TPR={successes}/{len(variant_rows)} "
        f"95% interval=[{low:.1%}, {high:.1%}]"
    )

enough_support = all(
    result.positive_count >= contract.min_positive_per_slice
    and result.negative_count >= contract.min_negative_per_slice
    for result in rates.values()
)
print(f"Minimum slice support: {'PASS' if enough_support else 'FAIL'}")
assert not enough_support

Output

formal         TPR=5/6 95% interval=[43.6%, 97.0%]
conversational TPR=3/6 95% interval=[18.8%, 81.2%]
Minimum slice support: FAIL

The intervals are wide because the fixture is small. That does not mean ignore the flips. It means use them as regression cases while collecting governed, reviewed evaluation data before making a population claim.

Calibration also needs slice support

The previous lesson used calibration to ask whether a judge agrees with reviewers. A fairness audit asks a stricter question: does a similar score carry similar meaning across slices? With ten rows per condition, score bands are diagnostic only.

calibration-by-slice.py

def score_band(score: float) -> str:
    if score < 0.70:
        return "below 0.70"
    if score < 0.90:
        return "0.70 to 0.89"
    return "0.90 and above"

calibration_cells: dict[tuple[str, str], list[AuditRow]] = defaultdict(list)
for row in rows:
    calibration_cells[(row.variant, score_band(row.judge_score))].append(row)

for (variant, band), cell in sorted(calibration_cells.items()):
    observed_ready = sum(row.expected_auto_serve for row in cell) / len(cell)
    print(f"{variant:14} {band:14}: n={len(cell)}, ready={observed_ready:.1%}")

assert max(len(cell) for cell in calibration_cells.values()) < 10
print("Calibration decision: insufficient support")

Output

conversational 0.70 to 0.89  : n=3, ready=100.0%
conversational below 0.70    : n=7, ready=42.9%
formal         0.70 to 0.89  : n=5, ready=80.0%
formal         0.90 and above: n=1, ready=100.0%
formal         below 0.70    : n=4, ready=25.0%
Calibration decision: insufficient support

Plan intersections without pretending to measure them

Single-slice summaries can hide a failure limited to one channel, locale, or accessibility setting. Real systems therefore plan intersectional reports. They must also impose minimum support, because slicing a small audit repeatedly produces unstable numbers and privacy risks.

A sample sufficiency grid for request channel and language-variety audit slice, marking small cells as report-only rather than release evidence. — Intersectional analysis is necessary, but small cells aren't proof. Report insufficient support instead of turning a handful of examples into a claim about customers.

intersection-support.py

intersection_counts: dict[tuple[str, str], int] = defaultdict(int)
for row in rows:
    if row.expected_auto_serve:
        intersection_counts[(row.channel, row.variant)] += 1

for (channel, variant), count in sorted(intersection_counts.items()):
    status = "eligible" if count >= contract.min_positive_per_slice else "insufficient"
    print(f"{channel:5} / {variant:14}: n={count}, {status}")

assert all(count == 3 for count in intersection_counts.values())

Output

chat  / conversational: n=3, insufficient
chat  / formal        : n=3, insufficient
email / conversational: n=3, insufficient
email / formal        : n=3, insufficient

In production, group definitions may involve sensitive attributes. Collect and expose them only under an approved purpose, access controls, privacy review, and any required consent or legal basis. A public dashboard with tiny protected-group cells can create harm while trying to measure it.

Find the failing stage before applying a fix

The matched pairs share policy evidence and human labels. Their routes diverge only after judge scoring. That localizes this lab's failure to the soft-evaluation and threshold layer. Rewriting customer text into a preferred register would conceal the symptom and ask customers to adapt to the system.

If a later investigation localizes a disparity to training data, counterfactual data augmentation (CDA) is one candidate experiment: add paired examples that alter an identity-related attribute while preserving the intended label. It isn't the first repair for this lab because the observed failure is in a deployed judge and router, not a proven training-set defect. CDA also needs review: careless swaps can change meaning, produce implausible text, or hide the group-specific harms you meant to measure.^[1]

A root-cause ladder for fairness mitigation showing evidence gate, judge rubric, routing threshold, reviewed audit set, and production monitoring. — Apply remediation where the measured failure occurs. Here, evidence is stable and the judge route fails, so collect reviewed pairs and repair that evaluator before considering broader training work.

locate-the-failure.py

def changed_stage(pair_rows: list[AuditRow]) -> str:
    if len({row.evidence_passed for row in pair_rows}) > 1:
        return "evidence_gate"
    if len({route(row) for row in pair_rows}) > 1:
        return "judge_or_route"
    return "no_observed_flip"

attribution = {
    pair_id: changed_stage(pair_rows)
    for pair_id, pair_rows in by_pair.items()
    if pair_id in {pair[0] for pair in flips}
}

print(attribution)
assert set(attribution.values()) == {"judge_or_route"}

Output

{'p4': 'judge_or_route', 'p5': 'judge_or_route', 'n1': 'judge_or_route'}

A reasonable next experiment is a revised rubric and judge prompt that focus on remedy correctness and actionable next steps rather than writing register. For teaching purposes, the following candidate rerun has equal rates. It still isn't release evidence: it uses the same synthetic cases that exposed the defect.

candidate-rerun.py

candidate_scores = {
    "p1": (0.92, 0.91), "p2": (0.86, 0.84), "p3": (0.81, 0.80),
    "p4": (0.77, 0.75), "p5": (0.72, 0.71), "p6": (0.66, 0.65),
    "n1": (0.62, 0.61), "n2": (0.60, 0.58), "n3": (0.55, 0.56),
    "n4": (0.64, 0.62),
}

candidate_rows: list[AuditRow] = []
for pair_id, expected, channel, _, _ in observations:
    formal_score, conversational_score = candidate_scores[pair_id]
    candidate_rows.extend([
        AuditRow(pair_id, "formal", expected, formal_score, channel),
        AuditRow(pair_id, "conversational", expected, conversational_score, channel),
    ])

candidate_rates = {
    variant: slice_rates([row for row in candidate_rows if row.variant == variant])
    for variant in ("formal", "conversational")
}
candidate_tpr_gap = abs(candidate_rates["formal"].tpr - candidate_rates["conversational"].tpr)
candidate_fpr_gap = abs(candidate_rates["formal"].fpr - candidate_rates["conversational"].fpr)

print(f"Candidate TPR gap={candidate_tpr_gap:.1%}; FPR gap={candidate_fpr_gap:.1%}")
print("Interpretation: regression repaired on synthetic pairs, not validated for release")
assert candidate_tpr_gap == 0
assert candidate_fpr_gap == 0

Output

Candidate TPR gap=0.0%; FPR gap=0.0%
Interpretation: regression repaired on synthetic pairs, not validated for release

Public benchmarks and product audits play different roles

Product-specific matched pairs test the actual route customers experience. Public benchmarks provide broader regression coverage:

Evaluation source	What it tests	Appropriate use here
Matched ShopFlow pairs	Routing consistency for supported replacement replies	Primary product release audit
WEAT / SEAT	Whether word or sentence representations encode tested association patterns	Diagnostic probe when you can inspect behavior; not a routing outcome measure^[3]^[4]
StereoSet	Whether a language model assigns stronger preference to stereotypical than anti-stereotypical continuations in its test contexts	Probability-level stereotype regression probe^[5]
RealToxicityPrompts / BOLD	Toxic degeneration from prompts and open-ended generation about demographic groups	Generation-level audit set that needs human review and product-specific slices^[6]^[7]
BBQ	Whether question answering relies on stereotypes when context is ambiguous or disambiguated	Broad stereotype regression probe^[8]
Reviewed toxicity slices	Whether a safety evaluator flags language varieties unevenly	Evaluator audit; Sap et al. showed dialect-related false-positive risk in hate-speech detection.^[9]

These benchmarks operate at different layers. WEAT or SEAT can expose associations in representations even when generated outputs look harmless; RealToxicityPrompts or BOLD can expose output harms without explaining which internal representation caused them. Don't treat a benchmark pass as proof that customer routing is fair. A benchmark tests its own prompt distribution and label design. Don't treat one product slice as full safety coverage either. Use both, and keep the limitation attached to every report.

Why not optimize every metric?

Fairness metrics can conflict. When outcome prevalence differs across groups and predictions aren't perfect, a score calibrated within each group generally can't also equalize false-positive and false-negative rates across groups. Chouldechova demonstrated this incompatibility for risk scoring systems.^[10] The engineering response isn't to give up or chase a single universal score. It's to define the customer harm, select a primary metric, monitor important counter-metrics, and document the accepted trade-off.

In this lab, the chosen outcome is rapid access to a supported reply. Equal opportunity is primary because it asks whether replies reviewers mark ready reach the fast path similarly. The FPR guardrail prevents a superficial fix that merely auto-serves more unclear replies.

Write a release decision, not a fairness slogan

A fairness report must say what was tested, what failed, and why a candidate can't yet ship. That keeps a clean toy rerun from being promoted into an unsupported production claim.

A release-decision flow that retains matched regressions, gathers reviewed slice data, validates the candidate, monitors production outcomes, and documents limitations. — The first audit found a blocker. The candidate must pass a larger reviewed audit, privacy review, and production monitoring plan before it controls customer routing.

fairness-release-decision.py

release_requirements = {
    "synthetic_regression_pairs_pass": candidate_tpr_gap <= contract.max_tpr_gap
    and candidate_fpr_gap <= contract.max_fpr_gap,
    "representative_reviewed_slice_set": False,
    "minimum_positive_and_negative_support": False,
    "approved_group_definition_and_privacy_review": False,
    "production_monitoring_owner": False,
}

failures = [
    requirement
    for requirement, passed in release_requirements.items()
    if not passed
]
decision = "APPROVED" if not failures else "BLOCKED"

print(f"Metric promotion: {decision}")
for failure in failures:
    print(f"  missing: {failure}")

assert decision == "BLOCKED"

Output

Metric promotion: BLOCKED
  missing: representative_reviewed_slice_set
  missing: minimum_positive_and_negative_support
  missing: approved_group_definition_and_privacy_review
  missing: production_monitoring_owner

A blocked result is progress. The team now has reproducible regressions, a primary metric, counter-metric guardrails, a likely failing stage, and explicit evidence still needed before release.

Mastery check

Key concepts

Representational harm versus allocative harm
Matched-pair behavioral audits
Selection rate, TPR, and FPR gaps
Equal opportunity versus equalized odds
Calibration and incompatible fairness goals
Sample sufficiency and intersectional reporting
Evaluator bias and stage attribution
Representation-level versus generation-level benchmark probes
Counterfactual data augmentation as a hypothesis to validate, not a fairness guarantee
Mitigation validation and blocked release decisions

Evaluation rubric

Keeps policy support outside the fairness experiment
Describes synthetic wording fixtures without presenting them as demographic evidence
Identifies pair flips before aggregating rates
Chooses TPR gap from the delayed-service harm and monitors FPR as guardrail
Reports uncertainty and insufficient support instead of overstating a tiny sample
Applies mitigation at the observed failing stage
Refuses release until representative review, privacy governance, and monitoring exist

Follow-up questions

Common pitfalls

A fixture is presented as a finding about people

Symptom: A chart claims a real customer group receives worse outcomes, but all values came from hand-written prompts. Cause: Synthetic tests were confused with representative measurement. Fix: Label fixtures clearly, use them for regressions, and require governed real evaluation data for population claims.

A fairer rate is achieved by lowering quality

Symptom: Routing rates equalize because unclear replies are auto-served more often. Cause: The team watched selection rate without a false-positive guardrail. Fix: Track TPR and FPR together and tie the primary metric to customer harm.

Customer language is normalized instead of evaluator behavior

Symptom: The pipeline rewrites one language variety into another before judging. Cause: The system treats customer expression as the defect. Fix: Audit the judge and routing stage with reviewed matched inputs; preserve meaning and customer voice.

An underpowered slice becomes a green dashboard

Symptom: A parity report marks small intersections as passing. Cause: It omits sample requirements and uncertainty. Fix: Require minimum support, report insufficient cells, and protect sensitive slice data.

A generic mitigation catalogue replaces diagnosis

Symptom: The team proposes retraining the base model before locating where routes diverge. Cause: Fairness was treated as an abstract model property instead of a system outcome. Fix: evidence, evaluator, threshold, and workflow outcome first; fix the measured failing layer.

Next Step

Continue to Hallucination Detection & Mitigation

You can now block a soft evaluator when outcomes differ across controlled slices or evidence is too weak. Next you will block fluent answers when their factual claims aren't supported by retrieved evidence.

PreviousLLM-as-a-Judge Evaluation

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Bias and Fairness in Large Language Models: A Survey

Gallegos, I. O., Rossi, R. A., Barrow, J., et al. · 2024

Equality of Opportunity in Supervised Learning.

Hardt, M., Price, E., & Srebro, N. · 2016 · NeurIPS 2016

Semantics derived automatically from language corpora contain human-like biases.

Caliskan, A., Bryson, J. J., & Narayanan, A. · 2017 · Science 356(6334)

On measuring social biases in sentence encoders.

May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. · 2019 · NAACL 2019

StereoSet: Measuring stereotypical bias in pretrained language models.

Nadeem, M., et al. · 2020 · ACL 2021

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models.

Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. · 2020

BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation.

Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. · 2021 · FAccT 2021

BBQ: A Hand-Built Bias Benchmark for Question Answering.

Parrish, A., et al. · 2022 · ACL 2022

The Risk of Racial Bias in Hate Speech Detection.

Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. · 2019 · ACL 2019

Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.

Chouldechova, A. · 2017 · Big Data

Back to Topics

LearnApplied LLM EngineeringBias & Fairness in LLMs

🛡️MediumAlignment & Safety

Bias & Fairness in LLMs

Build a matched-pair fairness audit for an LLM judge, measure routing gaps, and block release when evidence is too weak.

16 min read

Learning path

Step 66 of 155 in the full curriculum

LLM-as-a-Judge Evaluation Hallucination Detection & Mitigation

Fairness starts with a consequence

Our running case is allocative. The remedy is already authorized by the selected policy evidence. The new outcome is the route:

Decision component	Held fixed or measured?	Why it matters
Policy source and version	Held fixed	A fairness audit can't repair unsupported claims.
Replacement eligibility	Held fixed within each matched pair	Each pair should deserve the same answer.
Language-variety fixture	Varied within each pair	It's the audit condition.
Judge score and route	Measured	Unequal routing is customer impact.

The distinction from the previous chapter is important. There, we asked whether a judge agrees with reviewers. Here, we ask whether its errors and routing decisions are unevenly distributed.

Figure 1: Fairness evaluation begins after evidence support. It measures whether a soft judge and router create uneven outcomes across controlled slices.

Build a matched-pair audit

matched-audit-fixtures.py

from collections import defaultdict
from dataclasses import dataclass
from math import sqrt

@dataclass(frozen=True)
class AuditRow:
    pair_id: str
    variant: str
    expected_auto_serve: bool
    judge_score: float
    channel: str
    evidence_passed: bool = True

THRESHOLD = 0.70

def route(row: AuditRow) -> str:
    if not row.evidence_passed:
        return "blocked_by_evidence"
    return "auto_serve" if row.judge_score >= THRESHOLD else "human_review"

# Synthetic observations: (pair, expected outcome, channel, formal score, conversational score)
observations = [
    ("p1", True, "chat", 0.92, 0.88),
    ("p2", True, "chat", 0.88, 0.73),
    ("p3", True, "chat", 0.84, 0.72),
    ("p4", True, "email", 0.78, 0.67),
    ("p5", True, "email", 0.74, 0.56),
    ("p6", True, "email", 0.66, 0.61),
    ("n1", False, "chat", 0.71, 0.60),
    ("n2", False, "chat", 0.62, 0.55),
    ("n3", False, "email", 0.60, 0.52),
    ("n4", False, "email", 0.68, 0.63),
]

rows: list[AuditRow] = []
for pair_id, expected, channel, formal_score, conversational_score in observations:
    rows.extend([
        AuditRow(pair_id, "formal", expected, formal_score, channel),
        AuditRow(pair_id, "conversational", expected, conversational_score, channel),
    ])

assert len(rows) == 20
assert all(row.evidence_passed for row in rows)
assert {
    row.pair_id: row.expected_auto_serve for row in rows if row.variant == "formal"
} == {
    row.pair_id: row.expected_auto_serve for row in rows if row.variant == "conversational"
}

print("Fixture type: synthetic matched wording audit")
print(f"Matched pairs: {len(observations)}; scored rows: {len(rows)}")
print(f"Routing threshold: {THRESHOLD:.2f}")

Output

Fixture type: synthetic matched wording audit
Matched pairs: 10; scored rows: 20
Routing threshold: 0.70

The fixture intentionally creates a failure. If every example passed, we could demonstrate arithmetic but not diagnosis.

Look for pair flips first

pair-flips.py

by_pair: dict[str, list[AuditRow]] = defaultdict(list)
for row in rows:
    by_pair[row.pair_id].append(row)

flips: list[tuple[str, str, str]] = []
for pair_id, pair_rows in by_pair.items():
    outcomes = {row.variant: route(row) for row in pair_rows}
    if len(set(outcomes.values())) > 1:
        flips.append((pair_id, outcomes["formal"], outcomes["conversational"]))

assert flips == [
    ("p4", "auto_serve", "human_review"),
    ("p5", "auto_serve", "human_review"),
    ("n1", "auto_serve", "human_review"),
]

print("Flipped matched pairs:")
for pair_id, formal_route, conversational_route in flips:
    print(f"  {pair_id}: formal={formal_route}, conversational={conversational_route}")

Output

Flipped matched pairs:
  p4: formal=auto_serve, conversational=human_review
  p5: formal=auto_serve, conversational=human_review
  n1: formal=auto_serve, conversational=human_review

Two eligible replies lose the fast path under the conversational condition. One unclear reply gains the fast path under the formal condition. A single approval-rate number can't explain both errors.

Choose a metric from the harm

Four group metrics appear frequently in fairness work. They answer different product questions:

Metric	Calculation	Question for the router
Selection rate	Auto-served / all requests	Does one slice receive fast service more often?
True positive rate (TPR)	Auto-served / replies reviewers say are ready	Do ready replies receive fast service equally often?
False positive rate (FPR)	Auto-served / replies reviewers say need review	Does one slice receive unsafe fast service more often?
Calibration	Observed ready rate among equal score bands	Does a `0.80` score carry same meaning by slice?

Calculate slice rates

slice-rates.py

@dataclass(frozen=True)
class Rates:
    selection: float
    tpr: float
    fpr: float
    positive_count: int
    negative_count: int

def slice_rates(slice_rows: list[AuditRow]) -> Rates:
    positives = [row for row in slice_rows if row.expected_auto_serve]
    negatives = [row for row in slice_rows if not row.expected_auto_serve]
    selected = [row for row in slice_rows if route(row) == "auto_serve"]
    true_positives = [row for row in positives if route(row) == "auto_serve"]
    false_positives = [row for row in negatives if route(row) == "auto_serve"]
    return Rates(
        selection=len(selected) / len(slice_rows),
        tpr=len(true_positives) / len(positives),
        fpr=len(false_positives) / len(negatives),
        positive_count=len(positives),
        negative_count=len(negatives),
    )

rates = {
    variant: slice_rates([row for row in rows if row.variant == variant])
    for variant in ("formal", "conversational")
}

def gap(metric: str) -> float:
    return abs(getattr(rates["formal"], metric) - getattr(rates["conversational"], metric))

for variant, result in rates.items():
    print(
        f"{variant:14} selection={result.selection:.1%} "
        f"TPR={result.tpr:.1%} FPR={result.fpr:.1%}"
    )
print(f"TPR gap={gap('tpr'):.1%}; FPR gap={gap('fpr'):.1%}")

Output

formal         selection=60.0% TPR=83.3% FPR=25.0%
conversational selection=30.0% TPR=50.0% FPR=0.0%
TPR gap=33.3%; FPR gap=25.0%

Turn metric choice into a gate

A release contract makes the choice reviewable. Thresholds below are product decisions for this lab, not universal definitions of fairness.

release-contract.py

@dataclass(frozen=True)
class FairnessContract:
    primary_metric: str
    max_tpr_gap: float
    max_fpr_gap: float
    min_positive_per_slice: int
    min_negative_per_slice: int

contract = FairnessContract(
    primary_metric="equal_opportunity",
    max_tpr_gap=0.10,
    max_fpr_gap=0.10,
    min_positive_per_slice=50,
    min_negative_per_slice=30,
)

metric_checks = {
    "TPR gap": gap("tpr") <= contract.max_tpr_gap,
    "FPR guardrail": gap("fpr") <= contract.max_fpr_gap,
}

for name, passed in metric_checks.items():
    print(f"{name}: {'PASS' if passed else 'FAIL'}")
assert not any(metric_checks.values())

Output

TPR gap: FAIL
FPR guardrail: FAIL

Tiny slices can't certify fairness

uncertainty-and-support.py

def wilson_interval(successes: int, total: int, z: float = 1.96) -> tuple[float, float]:
    proportion = successes / total
    denominator = 1 + z * z / total
    center = (proportion + z * z / (2 * total)) / denominator
    radius = z * sqrt(
        (proportion * (1 - proportion) + z * z / (4 * total)) / total
    ) / denominator
    return center - radius, center + radius

for variant in ("formal", "conversational"):
    variant_rows = [
        row for row in rows
        if row.variant == variant and row.expected_auto_serve
    ]
    successes = sum(route(row) == "auto_serve" for row in variant_rows)
    low, high = wilson_interval(successes, len(variant_rows))
    print(
        f"{variant:14} TPR={successes}/{len(variant_rows)} "
        f"95% interval=[{low:.1%}, {high:.1%}]"
    )

enough_support = all(
    result.positive_count >= contract.min_positive_per_slice
    and result.negative_count >= contract.min_negative_per_slice
    for result in rates.values()
)
print(f"Minimum slice support: {'PASS' if enough_support else 'FAIL'}")
assert not enough_support

Output

formal         TPR=5/6 95% interval=[43.6%, 97.0%]
conversational TPR=3/6 95% interval=[18.8%, 81.2%]
Minimum slice support: FAIL

Calibration also needs slice support

calibration-by-slice.py

def score_band(score: float) -> str:
    if score < 0.70:
        return "below 0.70"
    if score < 0.90:
        return "0.70 to 0.89"
    return "0.90 and above"

calibration_cells: dict[tuple[str, str], list[AuditRow]] = defaultdict(list)
for row in rows:
    calibration_cells[(row.variant, score_band(row.judge_score))].append(row)

for (variant, band), cell in sorted(calibration_cells.items()):
    observed_ready = sum(row.expected_auto_serve for row in cell) / len(cell)
    print(f"{variant:14} {band:14}: n={len(cell)}, ready={observed_ready:.1%}")

assert max(len(cell) for cell in calibration_cells.values()) < 10
print("Calibration decision: insufficient support")

Output

conversational 0.70 to 0.89  : n=3, ready=100.0%
conversational below 0.70    : n=7, ready=42.9%
formal         0.70 to 0.89  : n=5, ready=80.0%
formal         0.90 and above: n=1, ready=100.0%
formal         below 0.70    : n=4, ready=25.0%
Calibration decision: insufficient support

Plan intersections without pretending to measure them

intersection-support.py

intersection_counts: dict[tuple[str, str], int] = defaultdict(int)
for row in rows:
    if row.expected_auto_serve:
        intersection_counts[(row.channel, row.variant)] += 1

for (channel, variant), count in sorted(intersection_counts.items()):
    status = "eligible" if count >= contract.min_positive_per_slice else "insufficient"
    print(f"{channel:5} / {variant:14}: n={count}, {status}")

assert all(count == 3 for count in intersection_counts.values())

Output

chat  / conversational: n=3, insufficient
chat  / formal        : n=3, insufficient
email / conversational: n=3, insufficient
email / formal        : n=3, insufficient

Find the failing stage before applying a fix

locate-the-failure.py

def changed_stage(pair_rows: list[AuditRow]) -> str:
    if len({row.evidence_passed for row in pair_rows}) > 1:
        return "evidence_gate"
    if len({route(row) for row in pair_rows}) > 1:
        return "judge_or_route"
    return "no_observed_flip"

attribution = {
    pair_id: changed_stage(pair_rows)
    for pair_id, pair_rows in by_pair.items()
    if pair_id in {pair[0] for pair in flips}
}

print(attribution)
assert set(attribution.values()) == {"judge_or_route"}

Output

{'p4': 'judge_or_route', 'p5': 'judge_or_route', 'n1': 'judge_or_route'}

candidate-rerun.py

candidate_scores = {
    "p1": (0.92, 0.91), "p2": (0.86, 0.84), "p3": (0.81, 0.80),
    "p4": (0.77, 0.75), "p5": (0.72, 0.71), "p6": (0.66, 0.65),
    "n1": (0.62, 0.61), "n2": (0.60, 0.58), "n3": (0.55, 0.56),
    "n4": (0.64, 0.62),
}

candidate_rows: list[AuditRow] = []
for pair_id, expected, channel, _, _ in observations:
    formal_score, conversational_score = candidate_scores[pair_id]
    candidate_rows.extend([
        AuditRow(pair_id, "formal", expected, formal_score, channel),
        AuditRow(pair_id, "conversational", expected, conversational_score, channel),
    ])

candidate_rates = {
    variant: slice_rates([row for row in candidate_rows if row.variant == variant])
    for variant in ("formal", "conversational")
}
candidate_tpr_gap = abs(candidate_rates["formal"].tpr - candidate_rates["conversational"].tpr)
candidate_fpr_gap = abs(candidate_rates["formal"].fpr - candidate_rates["conversational"].fpr)

print(f"Candidate TPR gap={candidate_tpr_gap:.1%}; FPR gap={candidate_fpr_gap:.1%}")
print("Interpretation: regression repaired on synthetic pairs, not validated for release")
assert candidate_tpr_gap == 0
assert candidate_fpr_gap == 0

Output

Candidate TPR gap=0.0%; FPR gap=0.0%
Interpretation: regression repaired on synthetic pairs, not validated for release

Public benchmarks and product audits play different roles

Product-specific matched pairs test the actual route customers experience. Public benchmarks provide broader regression coverage:

Evaluation source	What it tests	Appropriate use here
Matched ShopFlow pairs	Routing consistency for supported replacement replies	Primary product release audit
WEAT / SEAT	Whether word or sentence representations encode tested association patterns	Diagnostic probe when you can inspect behavior; not a routing outcome measure^[3]^[4]
StereoSet	Whether a language model assigns stronger preference to stereotypical than anti-stereotypical continuations in its test contexts	Probability-level stereotype regression probe^[5]
RealToxicityPrompts / BOLD	Toxic degeneration from prompts and open-ended generation about demographic groups	Generation-level audit set that needs human review and product-specific slices^[6]^[7]
BBQ	Whether question answering relies on stereotypes when context is ambiguous or disambiguated	Broad stereotype regression probe^[8]
Reviewed toxicity slices	Whether a safety evaluator flags language varieties unevenly	Evaluator audit; Sap et al. showed dialect-related false-positive risk in hate-speech detection.^[9]

Why not optimize every metric?

Write a release decision, not a fairness slogan

A fairness report must say what was tested, what failed, and why a candidate can't yet ship. That keeps a clean toy rerun from being promoted into an unsupported production claim.

fairness-release-decision.py

release_requirements = {
    "synthetic_regression_pairs_pass": candidate_tpr_gap <= contract.max_tpr_gap
    and candidate_fpr_gap <= contract.max_fpr_gap,
    "representative_reviewed_slice_set": False,
    "minimum_positive_and_negative_support": False,
    "approved_group_definition_and_privacy_review": False,
    "production_monitoring_owner": False,
}

failures = [
    requirement
    for requirement, passed in release_requirements.items()
    if not passed
]
decision = "APPROVED" if not failures else "BLOCKED"

print(f"Metric promotion: {decision}")
for failure in failures:
    print(f"  missing: {failure}")

assert decision == "BLOCKED"

Output

Metric promotion: BLOCKED
  missing: representative_reviewed_slice_set
  missing: minimum_positive_and_negative_support
  missing: approved_group_definition_and_privacy_review
  missing: production_monitoring_owner

A blocked result is progress. The team now has reproducible regressions, a primary metric, counter-metric guardrails, a likely failing stage, and explicit evidence still needed before release.

Mastery check

Key concepts

Representational harm versus allocative harm
Matched-pair behavioral audits
Selection rate, TPR, and FPR gaps
Equal opportunity versus equalized odds
Calibration and incompatible fairness goals
Sample sufficiency and intersectional reporting
Evaluator bias and stage attribution
Representation-level versus generation-level benchmark probes
Counterfactual data augmentation as a hypothesis to validate, not a fairness guarantee
Mitigation validation and blocked release decisions

Evaluation rubric

Keeps policy support outside the fairness experiment
Describes synthetic wording fixtures without presenting them as demographic evidence
Identifies pair flips before aggregating rates
Chooses TPR gap from the delayed-service harm and monitors FPR as guardrail
Reports uncertainty and insufficient support instead of overstating a tiny sample
Applies mitigation at the observed failing stage
Refuses release until representative review, privacy governance, and monitoring exist

Follow-up questions

Common pitfalls

A fixture is presented as a finding about people

A fairer rate is achieved by lowering quality

Customer language is normalized instead of evaluator behavior

An underpowered slice becomes a green dashboard

A generic mitigation catalogue replaces diagnosis

Next Step

Continue to Hallucination Detection & Mitigation

PreviousLLM-as-a-Judge Evaluation

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Bias and Fairness in Large Language Models: A Survey

Gallegos, I. O., Rossi, R. A., Barrow, J., et al. · 2024

Equality of Opportunity in Supervised Learning.

Hardt, M., Price, E., & Srebro, N. · 2016 · NeurIPS 2016

Semantics derived automatically from language corpora contain human-like biases.

Caliskan, A., Bryson, J. J., & Narayanan, A. · 2017 · Science 356(6334)

On measuring social biases in sentence encoders.

May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. · 2019 · NAACL 2019

StereoSet: Measuring stereotypical bias in pretrained language models.

Nadeem, M., et al. · 2020 · ACL 2021

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models.

Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. · 2020

BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation.

Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. · 2021 · FAccT 2021

BBQ: A Hand-Built Bias Benchmark for Question Answering.

Parrish, A., et al. · 2022 · ACL 2022

The Risk of Racial Bias in Hate Speech Detection.

Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. · 2019 · ACL 2019

Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.

Chouldechova, A. · 2017 · Big Data

Bias & Fairness in LLMs

Fairness starts with a consequence

Build a matched-pair audit

Look for pair flips first

Choose a metric from the harm

Calculate slice rates

Why isn't a 30 percentage point selection-rate gap enough to diagnose the failure?

Turn metric choice into a gate

Tiny slices can't certify fairness

Calibration also needs slice support

Plan intersections without pretending to measure them

Find the failing stage before applying a fix

Why shouldn't the team transform conversational customer requests into formal language before judging them?

Public benchmarks and product audits play different roles

Why not optimize every metric?

Why not make calibration, equal TPR, and equal FPR universal hard gates for every model?

Write a release decision, not a fairness slogan

Mastery check

Key concepts

Evaluation rubric

Follow-up questions

The candidate judge reaches zero gap on the ten paired fixtures. May it control production routing?

Your overall agreement with human reviewers stays high, but one request slice loses the auto-serve path more often. What should you inspect?

Why is a public benchmark such as BBQ not enough for this release?

What should happen when an intersectional slice contains only three ready examples?

Common pitfalls

A fixture is presented as a finding about people

A fairer rate is achieved by lowering quality

Customer language is normalized instead of evaluator behavior

An underpowered slice becomes a green dashboard

A generic mitigation catalogue replaces diagnosis

Bias & Fairness in LLMs

Fairness starts with a consequence

Build a matched-pair audit

Look for pair flips first

Choose a metric from the harm

Calculate slice rates

Why isn't a 30 percentage point selection-rate gap enough to diagnose the failure?

Turn metric choice into a gate

Tiny slices can't certify fairness

Calibration also needs slice support

Plan intersections without pretending to measure them

Find the failing stage before applying a fix

Why shouldn't the team transform conversational customer requests into formal language before judging them?

Public benchmarks and product audits play different roles

Why not optimize every metric?

Why not make calibration, equal TPR, and equal FPR universal hard gates for every model?

Write a release decision, not a fairness slogan

Mastery check

Key concepts

Evaluation rubric

Follow-up questions

The candidate judge reaches zero gap on the ten paired fixtures. May it control production routing?

Your overall agreement with human reviewers stays high, but one request slice loses the auto-serve path more often. What should you inspect?

Why is a public benchmark such as BBQ not enough for this release?

What should happen when an intersectional slice contains only three ready examples?

Common pitfalls

A fixture is presented as a finding about people

A fairer rate is achieved by lowering quality

Customer language is normalized instead of evaluator behavior

An underpowered slice becomes a green dashboard

A generic mitigation catalogue replaces diagnosis