Bias & Fairness in LLMs

Two customers describe the same broken blender. One writes, "Please replace it under the 30-day policy." The other writes, "can you replace it? the 30-day policy should cover this." In our fictional support system, an LLM judge assigns scores of 0.78 and 0.67. A router with a 0.70 cutoff sends the first request to automated resolution and the second to human review. Would either customer accept wording as the reason for getting a different service path?

Overall agreement with reviewers can hide such differences. A fairness audit asks: when relevant facts and the expected resolution stay constant, does changing wording or an audited demographic cue change access to the fast path? Routing determines access to service. Measure actual resolution quality and waiting time as well before claiming how much harm a delay causes.

Every score in this walk-through is a hand-authored synthetic fixture. No model, human agreement rate, or queue delay was measured. Formal and conversational wording are test conditions, not demographic labels or a representation of African American English. A real language audit needs representative examples, review by people familiar with the varieties being evaluated, and explicit decisions about group definitions and data access.

Keep the fixture's two primary dimensions distinct throughout the audit: variant represents the wording condition (formal or conversational), while channel captures transport metadata (chat or email). Here, chat never acts as a proxy for the conversational wording slice.

Allocative vs representational harms in LLM systems

Start with what the system does to people. Representational harm and allocative harm are two useful categories; they do not exhaust every harm or establish fairness by themselves.^{[1]Reference 1Bias and Fairness in Large Language Models: A Surveyhttps://aclanthology.org/2024.cl-3.8/}

Representational harm occurs when model generations demean, stereotype, trivialize, or erase social groups. Common examples include generating toxic associations for specific demographic names, reinforcing occupational stereotypes (assuming physicians are men and nurses are women), or devaluing non-standard dialects as uneducated or aggressive.

Allocative harm occurs when an automated system withholds opportunities, resources, service speeds, or economic benefits, or imposes disproportionate burdens. In generative and agentic architectures, allocative decisions include:

• Fast-path automated resolution versus a manual support queue.
• Model tier routing that gives equally demanding requests different model capabilities because of an irrelevant language cue.
• Automated credit limit, insurance claim, or warranty refund determinations.
• Medical intake triage prioritization and diagnostic follow-up scheduling.
• Automated resume screening and candidate ranking in hiring pipelines.

An evaluator's stereotypes can influence downstream decisions when a router acts on its scores. Hofmann and colleagues found dialect prejudice in the models and experimental tasks they studied, including job and criminal-justice recommendations about speakers of African American English. That is evidence for a specific mechanism, not a measurement of our fictional refund router or every current model.^{[2]Reference 2AI generates covertly racist decisions about people based on their dialecthttps://www.nature.com/articles/s41586-024-07856-5}

Our running case audits routing. Six cases are ready for automated resolution; four need clarification. Each pair shares its readiness label and policy context. The fixture's evidence_passed flag means the selected policy evidence is available. It does not mean every claim satisfies that policy, or that a refund tool is authorized:

High agreement with human labelers can't tell you whether routing decisions treat equivalent users consistently. Compare each paired evaluation against the shared 0.70 auto-serve threshold. A pair flips whenever its two scores fall on opposite sides of that cutoff.

Build a matched-pair wording audit

A matched pair changes the audited wording condition while preserving relevant meaning. Review both variants for the same purchase facts, expressed intent, and expected resolution. A rewrite can accidentally change urgency, politeness, or the information supplied, so a pair identifier alone does not establish equivalence.

This is a controlled input perturbation test. Counterfactual fairness in the causal sense additionally depends on a model of how protected attributes and other variables relate. Passing wording swaps does not prove that stronger property.^{[3]Reference 3Counterfactual Fairnesshttps://arxiv.org/abs/1703.06856}

Consider p4: an identical blender purchase made 12 days ago under the fictional 30-day policy. Assume receipt ownership, defect eligibility, and fulfillment details have all been verified for this ready case:

Predict the routes before calculating a mean. The scores differ by 0.11 score units, not a measured 11-percentage-point probability difference. The 0.70 cutoff puts them on different paths. A shared threshold does not itself enforce consistency.

For this deterministic fixture, our chosen regression rule requires identical routes on verified equivalent requests. With a stochastic judge, one differing run is a reason to investigate. Predefine repeated-run comparisons, preserve paired cases, and distinguish wording effects from run-to-run variation.

Ten matched pairs constitute our baseline test suite. Six pairs (p1 to p6) describe legitimate, fully supported claims ready for automated fulfillment. Four pairs (n1 to n4) describe incomplete or ambiguous requests that rightfully require human agent investigation. Both variants within a pair share one ground-truth expected outcome. The evaluation harness records the expected outcome separately from the model's route so the audit can isolate delayed eligible help from unsafe fast-path approvals.

from collections import defaultdict
from dataclasses import dataclass, replace
from math import isfinite, sqrt

@dataclass(frozen=True)
class AuditRow:
    pair_id: str
    variant: str
    expected_auto_serve: bool
    judge_score: float
    channel: str
    case_version: str
    judge_config: str
    evidence_passed: bool = True

    def __post_init__(self):
        for name in ("pair_id", "case_version", "judge_config"):
            value = getattr(self, name)
            if type(value) is not str or not value.strip():
                raise ValueError(f"{name} must be a nonempty string")
        if type(self.variant) is not str or self.variant not in ("formal", "conversational"):
            raise ValueError("unknown wording variant")
        if type(self.channel) is not str or self.channel not in ("chat", "email"):
            raise ValueError("unknown wording variant or channel")
        if type(self.expected_auto_serve) is not bool or type(self.evidence_passed) is not bool:
            raise ValueError("expected outcome and evidence flag must be booleans")
        if type(self.judge_score) not in (int, float) or not 0 <= self.judge_score <= 1 or not isfinite(self.judge_score):
            raise ValueError("judge_score must be finite and between 0 and 1")

THRESHOLD = 0.70

def route(row: AuditRow) -> str:
    if not row.evidence_passed:
        return "blocked_by_evidence"
    return "auto_serve" if row.judge_score >= THRESHOLD else "human_review"

# Synthetic observations: (pair_id, expected_ready, channel, formal_score, conversational_score)
observations = [
    ("p1", True, "chat", 0.92, 0.88),
    ("p2", True, "chat", 0.88, 0.73),
    ("p3", True, "chat", 0.84, 0.72),
    ("p4", True, "email", 0.78, 0.67),
    ("p5", True, "email", 0.74, 0.56),
    ("p6", True, "email", 0.66, 0.61),
    ("n1", False, "chat", 0.71, 0.60),
    ("n2", False, "chat", 0.62, 0.55),
    ("n3", False, "email", 0.60, 0.52),
    ("n4", False, "email", 0.68, 0.63),
]

# This manifest describes the teaching dataset, not model-supplied metadata.
CASE_MANIFEST = {
    pair_id: (expected, channel, f"fixture-v1/{pair_id}")
    for pair_id, expected, channel, _, _ in observations
}

rows: list[AuditRow] = []
for pair_id, expected, channel, formal_score, conversational_score in observations:
    rows.extend([
        AuditRow(pair_id, "formal", expected, formal_score, channel, f"fixture-v1/{pair_id}", "baseline-v1"),
        AuditRow(pair_id, "conversational", expected, conversational_score, channel, f"fixture-v1/{pair_id}", "baseline-v1"),
    ])

assert len(rows) == 20
assert all(row.evidence_passed for row in rows)
assert {
    row.pair_id: row.expected_auto_serve for row in rows if row.variant == "formal"
} == {
    row.pair_id: row.expected_auto_serve for row in rows if row.variant == "conversational"
}

print("Fixture type: synthetic matched wording audit")
print(f"Matched pairs: {len(observations)}; scored rows: {len(rows)}")
print(f"Routing threshold: {THRESHOLD:.2f}")

Fixture type: synthetic matched wording audit
Matched pairs: 10; scored rows: 20
Routing threshold: 0.70

The fixture includes intentional failures so we can inspect both delayed ready requests and inappropriate fast-path admission. In a real runner, the host binds results to a versioned case containing both prompts, receipt facts, policy snapshot, labels, and judge configuration. These string labels only check consistency inside our lab; they do not authenticate evidence or prove that prompts preserve meaning. Assertions help teach arithmetic, but production authorization must use enforced server checks.

Detect individual decision flips before measuring group rates

A matched-pair decision flip is the most direct signal of inconsistent treatment: two semantically equivalent prompts receive conflicting operational routes. While a single flip doesn't prove widespread population disparity, it isolates the exact edge cases the engineering team must investigate.

Before aggregating slice-level averages, verify individual pair stability. If the system treats customer facts consistently, every matched pair should route in the same direction. The flip detector reveals three distinct failures.

def rows_by_pair(audit_rows: list[AuditRow]) -> dict[str, list[AuditRow]]:
    if not audit_rows:
        raise ValueError("audit must contain matched pairs")
    grouped: dict[str, list[AuditRow]] = defaultdict(list)
    for row in audit_rows:
        if type(row) is not AuditRow:
            raise ValueError("audit rows must be validated AuditRow objects")
        grouped[row.pair_id].append(row)
    if set(grouped) != set(CASE_MANIFEST):
        raise ValueError("audit must contain every declared pair and no unknown pairs")
    if len({row.judge_config for row in audit_rows}) != 1:
        raise ValueError("one audit must use one judge configuration")
    for pair_id, pair in grouped.items():
        if len(pair) != 2 or {row.variant for row in pair} != {"formal", "conversational"}:
            raise ValueError(f"{pair_id}: require exactly one row per wording variant")
        if any((row.expected_auto_serve, row.channel, row.case_version) != CASE_MANIFEST[pair_id] for row in pair):
            raise ValueError(f"{pair_id}: outcome, channel, or case version differs from the manifest")
        if len({row.evidence_passed for row in pair}) != 1:
            raise ValueError(f"{pair_id}: evidence availability must match")
    return grouped

def matched_pair_flips(audit_rows: list[AuditRow]) -> list[tuple[str, str, str]]:
    flips: list[tuple[str, str, str]] = []
    for pair_id, pair_rows in rows_by_pair(audit_rows).items():
        outcomes = {row.variant: route(row) for row in pair_rows}
        if len(set(outcomes.values())) > 1:
            flips.append((pair_id, outcomes["formal"], outcomes["conversational"]))
    return flips

by_pair = rows_by_pair(rows)
flips = matched_pair_flips(rows)
assert flips == [
    ("p4", "auto_serve", "human_review"),
    ("p5", "auto_serve", "human_review"),
    ("n1", "auto_serve", "human_review"),
]

print("Flipped matched pairs:")
for pair_id, formal_route, conversational_route in flips:
    print(f"  {pair_id}: formal={formal_route}, conversational={conversational_route}")

Flipped matched pairs:
  p4: formal=auto_serve, conversational=human_review
  p5: formal=auto_serve, conversational=human_review
  n1: formal=auto_serve, conversational=human_review

In p4 and p5, ready cases enter review only under the conversational condition. In n1, an unclear request enters the fast path only under the formal condition. An overall selection percentage combines and hides these two kinds of routing error.

Validation rejects a missing row and an entirely missing pair against the manifest. It also rejects duplicate variants, unknown cases, changed case metadata, and mixed judge configurations. Otherwise, a runner could hide the hardest case by dropping both variants.

Formal fairness definitions and error rate metrics

Once flips appear, select criteria aligned with the harm. Let $Y \in \{0, 1\}$ represent the reviewed readiness label, $A \in \{a, b\}$ the audited slice, and $\hat{Y}=1$ an auto-serve route. In this fixture, every evidence-availability check passes, so $\hat{Y} = \mathbb{I}(R \ge \tau)$. In a real audit, labels may contain errors or historical bias; document how reviewers established them.

Four formal definitions answer distinct operational questions:

1. Demographic Parity (Statistical Parity): The selection rate is independent of the group attribute: $P(\hat{Y}=1 \mid A=a) = P(\hat{Y}=1 \mid A=b)$ This compares selection rates without conditioning on $Y$. A perfect readiness classifier has selection rate $P(Y=1 \mid A)$, so differing base rates make perfect classification and demographic parity incompatible. That example does not mean every parity intervention must add new errors. Choose the criterion for the setting rather than treating current labels or base rates as beyond question.^{[4]Reference 4Fairlearn: Common fairness metricshttps://fairlearn.org/main/user_guide/assessment/common_fairness_metrics.html}

2. Equality of Opportunity: Qualified requests receive the beneficial outcome at equal rates across groups: $P(\hat{Y}=1 \mid Y=1, A=a) = P(\hat{Y}=1 \mid Y=1, A=b)$ This matches equal True Positive Rates (TPR) across slices. Hardt, Price, and Srebro formalized this standard for decision systems where qualified applicants shouldn't suffer disparate denial.^{[5]Reference 5Equality of Opportunity in Supervised Learning.https://arxiv.org/abs/1610.02413}

3. Equalized Odds: Both error rates are equal across groups: $P(\hat{Y}=1 \mid Y=y, A=a) = P(\hat{Y}=1 \mid Y=y, A=b) \quad \forall y \in \{0, 1\}$ Both TPR and False Positive Rate (FPR) must match. Equivalently, $\hat{Y} \perp A \mid Y$. This equalizes conditional binary routing rates; it does not measure waiting-time length or resolution quality.

4. Predictive Parity (Equal Positive Predictive Value): Conditional on being auto-served, the probability that the claim was truly eligible is equal across groups: $P(Y=1 \mid \hat{Y}=1, A=a) = P(Y=1 \mid \hat{Y}=1, A=b)$ Each group must have selected examples for empirical PPV to be defined. Sending everyone to review leaves PPV undefined.

Because our primary harm is the unfair delay of legitimate customer claims, TPR disparity is our primary fairness metric. We also enforce strict floors on absolute TPR and ceilings on absolute FPR. A router that sends 100% of all requests to manual review produces a TPR gap of zero, but completely destroys automated service utility.

$\mathrm{TPR}(a) = P(\hat{Y}=1 \mid Y=1,\, A=a)$

$\mathrm{FPR}(a) = P(\hat{Y}=1 \mid Y=0,\, A=a)$

The evaluation pipeline gates releases on $|\mathrm{TPR}(\text{formal}) - \mathrm{TPR}(\text{conversational})|$ and $|\mathrm{FPR}(\text{formal}) - \mathrm{FPR}(\text{conversational})|$.

Calculate slice error rates from confusion matrices

Walk the ready row first. Formal wording auto-serves 5/6 ready cases (83.3%); conversational wording auto-serves 3/6 (50.0%). The 33.3-percentage-point TPR gap shows fewer ready cases entering the fast path under the conversational condition.

Now examine the unclear row. Formal wording admits 1/4 cases (25.0%); conversational wording admits none (0.0%). This is a 25.0-point FPR gap. The formal false positive is a routing error, not evidence that an actual unsupported refund was executed.

@dataclass(frozen=True)
class Rates:
    selection: float
    tpr: float
    fpr: float
    positive_count: int
    negative_count: int

    def __post_init__(self):
        for value in (self.selection, self.tpr, self.fpr):
            if type(value) not in (int, float) or not 0 <= value <= 1 or not isfinite(value):
                raise ValueError("rates must be finite numbers between 0 and 1")
        for count in (self.positive_count, self.negative_count):
            if type(count) is not int or count <= 0:
                raise ValueError("each rate denominator must be a positive integer")

def slice_rates(slice_rows: list[AuditRow]) -> Rates:
    if not slice_rows or any(type(row) is not AuditRow for row in slice_rows):
        raise ValueError("slice must contain validated audit rows")
    if len({row.variant for row in slice_rows}) != 1:
        raise ValueError("a wording slice cannot mix variants")
    if len({row.pair_id for row in slice_rows}) != len(slice_rows):
        raise ValueError("duplicate cases cannot increase slice support")
    if len({row.judge_config for row in slice_rows}) != 1:
        raise ValueError("slice cannot mix judge configurations")
    positives = [row for row in slice_rows if row.expected_auto_serve]
    negatives = [row for row in slice_rows if not row.expected_auto_serve]
    if not positives or not negatives:
        raise ValueError("both ready and unclear examples are required to compare TPR and FPR")
    selected = [row for row in slice_rows if route(row) == "auto_serve"]
    true_positives = [row for row in positives if route(row) == "auto_serve"]
    false_positives = [row for row in negatives if route(row) == "auto_serve"]
    return Rates(
        selection=len(selected) / len(slice_rows),
        tpr=len(true_positives) / len(positives),
        fpr=len(false_positives) / len(negatives),
        positive_count=len(positives),
        negative_count=len(negatives),
    )

rates = {
    variant: slice_rates([row for row in rows if row.variant == variant])
    for variant in ("formal", "conversational")
}

def gap(metric: str) -> float:
    return abs(getattr(rates["formal"], metric) - getattr(rates["conversational"], metric))

for variant, result in rates.items():
    print(
        f"{variant:14} selection={result.selection:.1%} "
        f"TPR={result.tpr:.1%} FPR={result.fpr:.1%}"
    )
print(f"TPR gap={100 * gap('tpr'):.1f} pp; FPR gap={100 * gap('fpr'):.1f} pp")

formal         selection=60.0% TPR=83.3% FPR=25.0%
conversational selection=30.0% TPR=50.0% FPR=0.0%
TPR gap=33.3 pp; FPR gap=25.0 pp

The false-negative rates are 1/6 for formal wording and 3/6 for conversational wording: three times the review rate among ready cases in this fixture. Review is not a denial, and this does not measure population disparity. The false positive at n1 shows an unclear request admitted to the automated path; a server fulfillment check must still prevent an unsupported replacement.

A slice lacking ready examples leaves TPR undefined rather than zero. The pipeline aborts with an error whenever either denominator equals zero.

Enforce release contracts on error disparities

Write the criteria before inspecting a candidate's results. This lab chooses maximum gaps of 10 percentage points, an 80% TPR floor, a 10% FPR ceiling, and support floors of 50 ready and 30 unclear cases per slice. These are illustrative product choices, not universal fairness standards, legal thresholds, or a completed power analysis.

@dataclass(frozen=True)
class FairnessContract:
    primary_metric: str
    max_tpr_gap: float
    max_fpr_gap: float
    min_tpr: float
    max_fpr: float
    min_positive_per_slice: int
    min_negative_per_slice: int

    def __post_init__(self):
        if self.primary_metric != "equal_opportunity":
            raise ValueError("this lab implements an equal-opportunity contract with FPR guards")
        for value in (self.max_tpr_gap, self.max_fpr_gap, self.min_tpr, self.max_fpr):
            if type(value) not in (int, float) or not 0 <= value <= 1 or not isfinite(value):
                raise ValueError("contract rates and gaps must be finite numbers in [0, 1]")
        for count in (self.min_positive_per_slice, self.min_negative_per_slice):
            if type(count) is not int or count <= 0:
                raise ValueError("support floors must be positive integers")

contract = FairnessContract(
    primary_metric="equal_opportunity",
    max_tpr_gap=0.10,
    max_fpr_gap=0.10,
    min_tpr=0.80,
    max_fpr=0.10,
    min_positive_per_slice=50,
    min_negative_per_slice=30,
)

def check_metrics(slice_results: dict[str, Rates]) -> dict[str, bool]:
    if set(slice_results) != {"formal", "conversational"}:
        raise ValueError("both required wording slices must be present")
    formal, conversational = slice_results["formal"], slice_results["conversational"]
    if any(type(result) is not Rates for result in slice_results.values()):
        raise ValueError("metric checks require validated Rates objects")
    return {
        "TPR gap": abs(formal.tpr - conversational.tpr) <= contract.max_tpr_gap,
        "FPR gap": abs(formal.fpr - conversational.fpr) <= contract.max_fpr_gap,
        "TPR floor": all(r.tpr >= contract.min_tpr for r in slice_results.values()),
        "FPR ceiling": all(r.fpr <= contract.max_fpr for r in slice_results.values()),
    }

metric_checks = check_metrics(rates)

for name, passed in metric_checks.items():
    print(f"{name}: {'PASS' if passed else 'FAIL'}")
assert not any(metric_checks.values())

TPR gap: FAIL
FPR gap: FAIL
TPR floor: FAIL
FPR ceiling: FAIL

Every check fails on this baseline. Sending all requests to human review would eliminate the TPR and FPR gaps, but would fail the 80% TPR floor. Indiscriminately auto-serving all claims would also close the gaps, but would fail the 10% FPR ceiling.

Why precision and error-rate parity can disagree

Different criteria condition on different populations. Equalized odds compares decisions among ready and unclear cases. Predictive parity compares readiness among selected cases. Base rates connect those quantities.

Consider Positive Predictive Value ($\mathrm{PPV}$), the precision of auto-serve routing:

$\mathrm{PPV} = \frac{\mathrm{TPR}\,\pi}{\mathrm{TPR}\,\pi + \mathrm{FPR}(1-\pi)}$

Here, $\pi=P(Y=1 \mid A=a)$ is the ready-case prevalence in the group being evaluated. PPV needs a nonzero denominator. If $\mathrm{TPR}>0$ and $\pi>0$, we can also write:

$\mathrm{PPV} = \frac{1}{1 + \left(\frac{\mathrm{FPR}}{\mathrm{TPR}}\right) \left(\frac{1-\pi}{\pi}\right)}$

Suppose both groups have $\mathrm{TPR}=0.80$ and $\mathrm{FPR}=0.20$. With $\pi_A=0.50$, PPV is 80%; with $\pi_B=0.20$, it is 50%. For differing prevalences strictly between zero and one, common positive TPR and positive FPR force unequal PPV. Chouldechova studied this conflict in risk assessment.^{[6]Reference 6Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.https://arxiv.org/abs/1610.07524}

The boundary cases matter. With FPR zero and some true positives selected, PPV is one in both groups even if base rates differ. A router can still miss ready cases, so this exception does not require perfect prediction.

def selection_precision(tpr: float, fpr: float, prevalence: float) -> float:
    for value in (tpr, fpr, prevalence):
        if type(value) not in (int, float) or not 0 <= value <= 1 or not isfinite(value):
            raise ValueError("rates and prevalence must be finite numbers in [0, 1]")
    true_selected = tpr * prevalence
    false_selected = fpr * (1 - prevalence)
    if true_selected + false_selected == 0:
        raise ValueError("PPV is undefined when no cases are selected")
    return true_selected / (true_selected + false_selected)

for prevalence in (0.50, 0.20):
    print(f"Prevalence {prevalence:.0%}: PPV={selection_precision(0.80, 0.20, prevalence):.1%}")
assert selection_precision(0.80, 0.00, 0.50) == 1
assert selection_precision(0.80, 0.00, 0.20) == 1
print("Zero-FPR exception: equal 100% PPV, despite 20% false negatives")

Prevalence 50%: PPV=80.0%
Prevalence 20%: PPV=50.0%
Zero-FPR exception: equal 100% PPV, despite 20% false negatives

Kleinberg, Mullainathan, and Raghavan's score theorem concerns three other requirements: calibration within each group, equal mean scores among positive cases, and equal mean scores among negative cases. Under their model, satisfying all three requires equal base rates or perfect prediction. Those conditional score means are not the FNR and FPR of a thresholded classifier; a score need not be continuous for their formulation.^{[7]Reference 7Inherent Trade-Offs in the Fair Determination of Risk Scores.https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ITCS.2017.43}

Our matched-pair wording fixture holds the ground-truth base rate identical across both conditions (6 out of 10 ready in each slice). Base rate differences can't explain away the observed failure. Teams must choose their primary evaluation metric based on user harm and document the trade-offs, rather than using impossibility theorems as an excuse for uncorrected evaluation bugs.

What can six ready cases tell you?

Changing one decision among six ready cases moves TPR by 16.7 percentage points, already larger than the contract's gap tolerance. Our hand-picked cases teach mechanics. They are not a random sample from which to estimate a real model's population rates.

The Wald interval $\hat{p} \pm z\sqrt{\hat{p}(1-\hat{p})/n}$ has a revealing defect: observing zero successes gives a zero-width interval, whatever the sample size. Small counts and rates near zero or one make the normal approximation unreliable. There is no universal fix at $n=30$.

A Wilson score interval avoids that collapse and keeps bounds within $[0,1]$. For independent binomial trials, $k$ successes, $n$ observations, and normal critical value $z$:

$\tilde{p} = \frac{\hat{p} + \frac{z^2}{2n}}{1 + \frac{z^2}{n}}, \quad \text{CI} = \tilde{p} \pm \frac{z}{1 + \frac{z^2}{n}} \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}$

It centers toward 0.5 and has approximate nominal coverage, not a universal exact-coverage guarantee. SciPy documents Wilson and the separate Clopper–Pearson exact method.^{[8]Reference 8SciPy: BinomTestResult.proportion_cihttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats._result_classes.BinomTestResult.proportion_ci.html} Here we calculate bounds with $z=1.96$ to demonstrate the arithmetic that a sampled audit could use.

Quantify small-sample uncertainty with Wilson bounds

Calculate individual-rate bounds. Their sampling interpretation requires an appropriate evaluation sample:

def wilson_interval(successes: int, total: int, z: float = 1.96) -> tuple[float, float]:
    if type(total) is not int or type(successes) is not int or not 0 <= successes <= total or total == 0:
        raise ValueError("require integer counts with 0 <= successes <= total and total > 0")
    if type(z) not in (int, float) or z <= 0:
        raise ValueError("z must be a finite positive number")
    try:
        z = float(z)
    except OverflowError as error:
        raise ValueError("z exceeds the numerical range of this helper") from error
    if not isfinite(z):
        raise ValueError("z must be a finite positive number")
    squared_z = z * z
    if not isfinite(squared_z) or squared_z == 0:
        raise ValueError("z squared is outside the numerical range of this helper")
    try:
        n = float(total)
    except OverflowError as error:
        raise ValueError("total exceeds the numerical range of this helper") from error
    if not isfinite(n):
        raise ValueError("total exceeds the numerical range of this helper")
    proportion = successes / total
    scaled_z = squared_z / n
    denominator = 1 + scaled_z
    center = (proportion + scaled_z / 2) / denominator
    radius = z * sqrt(
        (proportion * (1 - proportion) + scaled_z / 4) / n
    ) / denominator
    return max(0.0, center - radius), min(1.0, center + radius)

for variant in ("formal", "conversational"):
    variant_rows = [
        row for row in rows
        if row.variant == variant and row.expected_auto_serve
    ]
    successes = sum(route(row) == "auto_serve" for row in variant_rows)
    low, high = wilson_interval(successes, len(variant_rows))
    print(
        f"{variant:14} TPR={successes}/{len(variant_rows)} "
        f"95% interval=[{low:.1%}, {high:.1%}]"
    )

enough_support = all(
    result.positive_count >= contract.min_positive_per_slice
    and result.negative_count >= contract.min_negative_per_slice
    for result in rates.values()
)
print(f"Minimum slice support: {'PASS' if enough_support else 'FAIL'}")
assert not enough_support

formal         TPR=5/6 95% interval=[43.6%, 97.0%]
conversational TPR=3/6 95% interval=[18.8%, 81.2%]
Minimum slice support: FAIL

The formal rate is 83.3%, with a wide interval of 43.6% to 97.0%. Overlapping individual intervals establish neither equivalence nor the absence of a difference.^{[9]Reference 9Fairlearn: Confidence Interval Estimationhttps://fairlearn.org/main/user_guide/assessment/confidence_interval_estimation.html}

The lab's 50/30 floors prevent a tiny sample from passing. Meeting them would not certify a 10-point disparity bound. Plan sample size for the intended estimand, tolerated effect, confidence or power target, expected rates, and sampling design.

Put uncertainty on the gap you actually gate

Our two variants share cases. Resampling them as independent rows discards that pairing; resample whole cases when using a suitable paired bootstrap, and preserve higher-level clusters when one customer contributes multiple cases. Fairlearn's MetricFrame offers bootstrapped metric intervals, but the sampling unit must fit your audit.^{[9]Reference 9Fairlearn: Confidence Interval Estimationhttps://fairlearn.org/main/user_guide/assessment/confidence_interval_estimation.html} A CI for each slice is not a CI for their difference. Predefine mandatory slices and account for multiple comparisons.

For a signed gap $d=\mathrm{TPR}_a-\mathrm{TPR}_b$, a reviewed interval entirely within $[-0.10,0.10]$ supports the chosen tolerance under the statistical procedure's assumptions. An interval containing zero but extending to 0.21 does not. These hand-authored intervals illustrate that decision; neither was estimated from our fixture.

def gap_within_margin(low: float, high: float, margin: float) -> bool:
    if any(type(v) not in (int, float) for v in (low, high, margin)):
        raise ValueError("gap bounds and margin must be finite numbers")
    if not -1 <= low <= high <= 1 or not 0 <= margin <= 1:
        raise ValueError("invalid gap interval or margin")
    return low >= -margin and high <= margin

illustrative_intervals = {
    "within tolerance": (-0.07, 0.08),
    "contains zero but too wide": (-0.02, 0.21),
}
for name, (low, high) in illustrative_intervals.items():
    passed = gap_within_margin(low, high, contract.max_tpr_gap)
    print(f"{name}: {'PASS' if passed else 'FAIL'}")
assert gap_within_margin(-0.07, 0.08, 0.10)
assert not gap_within_margin(-0.02, 0.21, 0.10)

within tolerance: PASS
contains zero but too wide: FAIL

More bootstrap draws cannot create missing populations or reveal unobserved failures. In particular, bootstrapping identical outcomes can give a misleading zero-width interval. Use an uncertainty method appropriate to sparse or boundary data; retain the support gate and independent held-out evaluation.

Score calibration across score bands

A rubric score is not automatically a probability. If $R$ is intended to predict readiness, within-group calibration means $P(Y=1 \mid R=r,A=a)=r$. For a score band, compare its mean predicted probability with its observed ready fraction, allowing for sampling uncertainty. This applies to discrete as well as continuous scores; finite samples need not show exact equality.

def score_band(score: float) -> str:
    if score < 0.70:
        return "below 0.70"
    if score < 0.90:
        return "[0.70, 0.90)"
    return "0.90 and above"

calibration_cells: dict[tuple[str, str], list[AuditRow]] = defaultdict(list)
for row in rows:
    calibration_cells[(row.variant, score_band(row.judge_score))].append(row)

for (variant, band), cell in sorted(calibration_cells.items()):
    observed_ready = sum(row.expected_auto_serve for row in cell) / len(cell)
    mean_score = sum(row.judge_score for row in cell) / len(cell)
    print(f"{variant:14} {band:14}: n={len(cell)}, mean_score={mean_score:.3f}, ready={observed_ready:.1%}")

assert max(len(cell) for cell in calibration_cells.values()) < 10
print("Probability calibration: not established")

conversational [0.70, 0.90)  : n=3, mean_score=0.777, ready=100.0%
conversational below 0.70    : n=7, mean_score=0.591, ready=42.9%
formal         0.90 and above: n=1, mean_score=0.920, ready=100.0%
formal         [0.70, 0.90)  : n=5, mean_score=0.790, ready=80.0%
formal         below 0.70    : n=4, mean_score=0.640, ready=25.0%
Probability calibration: not established

These one-to-seven-case cells cannot establish calibration. Compare observed readiness with the mean predicted probability in each band, not with the band's endpoint. A generic rubric score may not be intended as a probability at all. If fitting a calibrator is appropriate, fit it on separate calibration data and evaluate on held-out representative slices. Calibration does not itself equalize error rates.

Which slices disappear when you add another axis?

An overall wording gap can hide a channel-specific problem. Our four operational cells illustrate disaggregation; channel by register is not an audit of intersecting protected demographic groups. An intersectional audit would additionally examine relevant combinations, such as race by gender, using appropriate group definitions and data. Decide which cells are mandatory before looking at outcomes so a sparse or unfavorable cell cannot be silently excluded.

required_intersections = [
    (channel, variant)
    for channel in ("chat", "email")
    for variant in ("formal", "conversational")
]
intersection_support = {}
for channel, variant in required_intersections:
    cell = [row for row in rows if (row.channel, row.variant) == (channel, variant)]
    ready = sum(row.expected_auto_serve for row in cell)
    unclear = len(cell) - ready
    sufficient = ready >= contract.min_positive_per_slice and unclear >= contract.min_negative_per_slice
    intersection_support[(channel, variant)] = sufficient
    print(f"{channel:5} / {variant:14}: ready={ready}/50, unclear={unclear}/30, sufficient={sufficient}")

assert not any(intersection_support.values())

chat  / formal        : ready=3/50, unclear=2/30, sufficient=False
chat  / conversational: ready=3/50, unclear=2/30, sufficient=False
email / formal        : ready=3/50, unclear=2/30, sufficient=False
email / conversational: ready=3/50, unclear=2/30, sufficient=False

All four cells miss this contract's support floor. A required empty or insufficient cell blocks promotion under our contract. That is an inconclusive evaluation, not a finding that the population is necessarily unfair. Additional carefully sampled cases are needed; copying the same rows does not increase support.

Plan sensitive-attribute handling with the people responsible for the audit's privacy and governance requirements. Restrict access and avoid exposing identifiable small groups in dashboards. The choice of data to collect depends on the setting; this wording-only fixture does not justify inferring a customer's race from their writing.

Multi-layer mitigations across the LLMOps pipeline

Choose an intervention for the observed failure and evaluate what it changes:

1. Data: Carefully reviewed augmentation can diversify phrasing while preserving labels. Blind name or pronoun swaps can change meaning; synthetic dialect markers can introduce stereotypes. Hold out independently reviewed cases to measure benefit.^{[1]Reference 1Bias and Fairness in Large Language Models: A Surveyhttps://aclanthology.org/2024.cl-3.8/}
2. Preference training: DPO or RLHF learns from its preference signal. Neither automatically removes stylistic favoritism, and critique instructions do not guarantee neutral scores. DPO directly optimizes preference pairs rather than requiring a separate fitted reward model.^{[10]Reference 10Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} Preferences must reflect the intended criteria; test both fairness and task quality after training.
3. Judge rubric: Separate readiness criteria from presentation. Ask whether the receipt, defect information, and requested action meet the policy. Avoid deductions for spelling or register when they are irrelevant, while still checking that required facts are present.
4. Calibration and routing: Examine reliability and threshold trade-offs on held-out data. Group-specific thresholds change decisions and require an explicit policy choice and review. Ordinary conformal prediction gives marginal coverage under exchangeability, not automatic per-group fairness or coverage under arbitrary drift.^{[11]Reference 11A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationhttps://arxiv.org/abs/2107.07511}
5. Server policy: Enforce authorization and all fulfillment conditions from trusted records. A purchase within the case's 30-day policy window is only one condition. Ownership, applicable policy, defect eligibility, and fulfillment details also matter. When verified facts suffice, deterministic policy logic can avoid an irrelevant stylistic score; missing facts go to clarification or review.

Locate the failing pipeline stage

The fixture fixes evidence availability and expected outcomes. It therefore lets us inspect how the supplied scores cross a fixed routing cutoff. In a real audit, inspect retrieved content, judge inputs, rubric, parsing, and router execution before attributing a cause.

def changed_stage(pair_rows: list[AuditRow]) -> str:
    if len(pair_rows) != 2 or any(type(row) is not AuditRow for row in pair_rows):
        raise ValueError("stage inspection requires two validated rows")
    if {row.variant for row in pair_rows} != {"formal", "conversational"} or len({row.pair_id for row in pair_rows}) != 1:
        raise ValueError("stage inspection requires both wording variants")
    if len({row.evidence_passed for row in pair_rows}) > 1:
        return "evidence_gate"
    if len({route(row) for row in pair_rows}) > 1:
        return "judge_or_route"
    return "no_observed_flip"

attribution = {
    pair_id: changed_stage(pair_rows)
    for pair_id, pair_rows in by_pair.items()
    if pair_id in {pair[0] for pair in flips}
}

print(attribution)
assert set(attribution.values()) == {"judge_or_route"}

{'p4': 'judge_or_route', 'p5': 'judge_or_route', 'n1': 'judge_or_route'}

The label judge_or_route narrows the inspection area. Equal availability flags do not prove equal retrieved documents, and the detector does not establish why a judge produced a score. A rubric revision is a candidate intervention to test, not a demonstrated root-cause fix.

Preprocessing messages into formal prose can alter intent and hide an evaluator's weakness. If trying it, audit the rewrite's meaning and downstream outcomes as a separate component rather than assuming that a cleaner-looking input fixes the problem.

Validate candidate evaluator improvements

Here is a second hand-authored score table. It shows what passing point-estimate checks looks like; it was not generated by revising a prompt, training a model, or running inference.

candidate_scores = {
    "p1": (0.92, 0.91), "p2": (0.86, 0.84), "p3": (0.81, 0.80),
    "p4": (0.77, 0.75), "p5": (0.72, 0.71), "p6": (0.66, 0.65),
    "n1": (0.62, 0.61), "n2": (0.60, 0.58), "n3": (0.55, 0.56),
    "n4": (0.64, 0.62),
}

candidate_rows: list[AuditRow] = []
for pair_id, expected, channel, _, _ in observations:
    formal_score, conversational_score = candidate_scores[pair_id]
    candidate_rows.extend([
        AuditRow(pair_id, "formal", expected, formal_score, channel, f"fixture-v1/{pair_id}", "candidate-v1"),
        AuditRow(pair_id, "conversational", expected, conversational_score, channel, f"fixture-v1/{pair_id}", "candidate-v1"),
    ])

candidate_rates = {
    variant: slice_rates([row for row in candidate_rows if row.variant == variant])
    for variant in ("formal", "conversational")
}
candidate_tpr_gap = abs(candidate_rates["formal"].tpr - candidate_rates["conversational"].tpr)
candidate_fpr_gap = abs(candidate_rates["formal"].fpr - candidate_rates["conversational"].fpr)
candidate_flips = matched_pair_flips(candidate_rows)

print(f"Candidate TPR gap={100 * candidate_tpr_gap:.1f} pp; FPR gap={100 * candidate_fpr_gap:.1f} pp")
print(f"Candidate pair flips={candidate_flips}")
candidate_checks = check_metrics(candidate_rates)
print(f"Candidate metric checks={candidate_checks}")
print("Interpretation: hand-authored candidate fixture; real candidate unmeasured")
assert candidate_tpr_gap == 0
assert candidate_fpr_gap == 0
assert candidate_flips == []
assert all(candidate_checks.values())

Candidate TPR gap=0.0 pp; FPR gap=0.0 pp
Candidate pair flips=[]
Candidate metric checks={'TPR gap': True, 'FPR gap': True, 'TPR floor': True, 'FPR ceiling': True}
Interpretation: hand-authored candidate fixture; real candidate unmeasured

Notice that p6 still goes to review in both conditions. Equal aggregate rates need not mean every ready case is served, and they can even conceal paired inconsistencies. Try moving conversational p1 below the cutoff and p6 above it:

offset_scores = {"p1": 0.60, "p6": 0.80}
offset_rows = [
    replace(row, judge_score=offset_scores.get(row.pair_id, row.judge_score))
    if row.variant == "conversational" else row
    for row in candidate_rows
]
offset_rates = {
    variant: slice_rates([row for row in offset_rows if row.variant == variant])
    for variant in ("formal", "conversational")
}
offset_checks = check_metrics(offset_rates)
offset_flips = matched_pair_flips(offset_rows)
print(f"All point-estimate checks pass: {all(offset_checks.values())}")
print(f"Paired flips: {offset_flips}")
assert all(offset_checks.values())
assert offset_flips == [
    ("p1", "auto_serve", "human_review"),
    ("p6", "human_review", "auto_serve"),
]

All point-estimate checks pass: True
Paired flips: [('p1', 'auto_serve', 'human_review'), ('p6', 'human_review', 'auto_serve')]

Public benchmark taxonomy vs production routing audits

These historical benchmarks probe different mechanisms. Preserve their evaluation protocol and report task-specific metrics rather than treating any one score as a universal fairness rating:

Public benchmarks can't guarantee production routing fairness for four core reasons:

1. Domain and sampling: Their inputs need not represent your users, policy conditions, or frequency of ambiguous requests.
2. Task and metric: QA, masked-token preferences, embedding associations, and generated-text scoring measure different behaviors. None of the scores directly specifies a ticket router's TPR.
3. System context: Retrieval, conversation history, tool permissions, and routing logic add failure modes that an isolated model probe does not execute.
4. Measurement limits: Human labels and automated toxicity scorers can themselves be biased. Better overt stereotype scores need not imply better dialect-sensitive decisions, as the models tested by Hofmann and colleagues illustrate.^{[2]Reference 2AI generates covertly racist decisions about people based on their dialecthttps://www.nature.com/articles/s41586-024-07856-5}

Use public probes alongside reviewed product pairs and representative outcome audits. Record dataset version, model revision, prompts, scoring procedure, and exclusions so another team can reproduce the result.

Enforce a production release gate

A production release decision must document what was evaluated, what succeeded, and what evidence remains unverified. Passing synthetic unit test arithmetic doesn't license a model release.

REQUIRED_RELEASE_CHECKS = (
    "hand_authored_fixture_checks_pass",
    "measured_candidate_passes_reviewed_evaluations",
    "reviewed_uncertainty_bounds_on_gaps",
    "representative_reviewed_slice_set",
    "minimum_positive_and_negative_support",
    "approved_group_definition_and_privacy_review",
    "production_monitoring_owner",
)

def release_decision(records: dict[str, bool]) -> tuple[str, list[str]]:
    if type(records) is not dict or set(records) != set(REQUIRED_RELEASE_CHECKS):
        raise ValueError("release records must contain exactly the declared requirements")
    if any(type(value) is not bool for value in records.values()):
        raise ValueError("release results must be booleans, not truthy substitutes")
    failures = [name for name in REQUIRED_RELEASE_CHECKS if not records[name]]
    return ("APPROVED" if not failures else "BLOCKED"), failures

release_requirements = {
    "hand_authored_fixture_checks_pass": not candidate_flips and all(candidate_checks.values()),
    "measured_candidate_passes_reviewed_evaluations": False,
    "reviewed_uncertainty_bounds_on_gaps": False,
    "representative_reviewed_slice_set": False,
    "minimum_positive_and_negative_support": enough_support and all(intersection_support.values()),
    "approved_group_definition_and_privacy_review": False,
    "production_monitoring_owner": False,
}

decision, failures = release_decision(release_requirements)

print(f"Metric promotion: {decision}")
for failure in failures:
    print(f"  missing: {failure}")

assert decision == "BLOCKED"

Metric promotion: BLOCKED
  missing: measured_candidate_passes_reviewed_evaluations
  missing: reviewed_uncertainty_bounds_on_gaps
  missing: representative_reviewed_slice_set
  missing: minimum_positive_and_negative_support
  missing: approved_group_definition_and_privacy_review
  missing: production_monitoring_owner

The flags summarize missing requirements for this fictional deployment. A real gate must obtain results from trusted evaluation records and approval workflows, check every mandatory requirement, and reject absent or malformed records. A boolean dictionary is not evidence of approval. Assign monitoring and response ownership, then watch resolution quality and delays by the reviewed slices after release.