Bias & Fairness in LLMs

A large language model (LLM) judge can agree with reviewers on average and still behave unevenly. If that judge sends equivalent requests down different routes for different customer language varieties, it can delay service for some customers even while its overall agreement score looks acceptable.

FairReply now wants to auto-serve a supported policy reply when the judge is confident, and send uncertain replies to a human reviewer. A fairness audit for that release decision asks one controlled question: when request facts and supported remedy stay fixed, does a change in language variety alter who receives the fast path?

These fixtures are invented and labeled. The two wording variants are test conditions, not demographic groups and not a claim about any community's speech. A real language-variety audit needs representative data, informed review, privacy controls, and careful group definitions.

Fairness starts with a consequence

A model can cause two broad kinds of harm. Representational harm occurs when output stereotypes, demeans, or erases a group. Allocative harm occurs when system behavior changes access to a benefit or burden, such as whether an eligible customer gets an immediate supported answer or waits for review. This distinction is central in surveys of bias and fairness in LLM systems.^{[1]Reference 1Bias and Fairness in Large Language Models: A Surveyhttps://aclanthology.org/2024.cl-3.8/}

Our running case is allocative. The remedy is already authorized by the selected policy evidence. The new outcome is the route:

The distinction from the previous chapter is important. There, we asked whether a judge agrees with reviewers. Here, we ask whether its errors and routing decisions are unevenly distributed. The overview below plots every paired score against the actual route threshold, then separates measured gaps from the evidence still required for release.

Build a matched-pair audit

A matched pair changes one audit condition while preserving task semantics. That's harder than swapping words mechanically. Two prompts belong in a pair only after reviewers agree they describe the same customer facts and should receive the same route.

For this lab, a human reviewer has labeled ten synthetic pairs. In six pairs the reply is supported and clear enough for auto-service. In four, it should go to review because wording of the proposed reply remains unclear. Both versions in a pair share the same expected outcome.

The fixture intentionally creates a failure. If every example passed, we could demonstrate arithmetic but not diagnosis.

Look for pair flips first

A pair flip is the simplest warning sign: equivalent requests receive different routes. It doesn't yet prove a population-level disparity, but it tells the team exactly which cases require investigation.

Two eligible replies lose the fast path under the conversational condition. One unclear reply gains the fast path under the formal condition. A single approval-rate number can't explain both errors.

Choose a metric from the harm

Four group metrics appear frequently in fairness work. They answer different product questions:

Equal opportunity compares TPR across slices. Equalized odds compares both TPR and FPR. Hardt, Price, and Srebro formalized these error-rate criteria for supervised decision systems.^{[2]Reference 2Equality of Opportunity in Supervised Learning.https://arxiv.org/abs/1610.02413} For FairReply, delayed eligible help is the main harm, so TPR gap is the primary release metric. FPR gap remains a guardrail because faster service isn't a win if it releases unclear replies.

A confusion matrix makes those denominators visible. TPR reads only the ready-reply row. FPR reads only the row of replies that reviewers say need review.

Calculate slice rates

This audit fails in both directions. Among replies reviewers marked ready, the conversational condition is routed to human review more often. Among replies that need review, the formal condition is incorrectly auto-served once.

Why isn't a 30 percentage point selection-rate gap enough to diagnose the failure?

Answer

Selection rate mixes correct and incorrect decisions. Here, one condition loses ready replies while the other incorrectly gains an unclear reply. TPR and FPR separate delayed service from unsafe fast service.

Turn metric choice into a gate

A release contract makes the choice reviewable. Thresholds below are product decisions for this lab, not universal definitions of fairness.

Tiny slices can't certify fairness

The audit found an actionable regression. It hasn't estimated production disparity. Six ready examples per wording condition are too few for a stable rate, and these fixtures don't identify a population.

One quick way to make that visible is a confidence interval. The Wilson interval below gives a plausible range for each TPR under binomial sampling. It isn't a complete statistical analysis, but it prevents a tiny dataset from looking decisive.

The intervals are wide because the fixture is small. That does not mean ignore the flips. It means use them as regression cases while collecting governed, reviewed evaluation data before making a population claim.

Calibration also needs slice support

The previous lesson used calibration to ask whether a judge agrees with reviewers. A fairness audit asks a stricter question: does a similar score carry similar meaning across slices? With ten rows per condition, score bands are diagnostic only.

Plan intersections without pretending to measure them

Single-slice summaries can hide a failure limited to one channel, locale, or accessibility setting. Real systems therefore plan intersectional reports. They must also impose minimum support, because slicing a small audit repeatedly produces unstable numbers and privacy risks.

In production, group definitions may involve sensitive attributes. Collect and expose them only under an approved purpose, access controls, privacy review, and any required consent or legal basis. A public dashboard with tiny protected-group cells can create harm while trying to measure it.

Find the failing stage before applying a fix

The matched pairs share policy evidence and human labels. Their routes diverge only after judge scoring. That localizes this lab's failure to the soft-evaluation and threshold layer. Rewriting customer text into a preferred register would conceal the symptom and ask customers to adapt to the system.

If a later investigation localizes a disparity to training data, counterfactual data augmentation (CDA) is one candidate experiment: add paired examples that alter an identity-related attribute while preserving the intended label. It isn't the first repair for this lab because the observed failure is in a deployed judge and router, not a proven training-set defect. CDA also needs review: careless swaps can change meaning, produce implausible text, or hide the group-specific harms you meant to measure.^{[1]Reference 1Bias and Fairness in Large Language Models: A Surveyhttps://aclanthology.org/2024.cl-3.8/}

A reasonable next experiment is a revised rubric and judge prompt that focus on remedy correctness and actionable next steps rather than writing register. For teaching purposes, the following candidate rerun has equal rates and no matched-pair flips. Both checks matter: offsetting flips can cancel out in aggregate. The rerun still isn't release evidence because it uses the same synthetic cases that exposed the defect.

Why shouldn't the team transform conversational customer requests into formal language before judging them?

Answer

That hides a measured failure instead of establishing equal treatment, may change meaning, and treats one register as the product default. Repair and validate the evaluator on representative, reviewed inputs rather than asking users to conform to it.

A candidate rerun has zero TPR and FPR gaps, but two matched pairs still flip in opposite directions. Has the synthetic regression suite passed?

Answer

No. Aggregate rates can cancel out while equivalent requests still receive different routes. Require both acceptable slice gaps and no matched-pair flips on retained regression cases.

Public benchmarks and product audits play different roles

Product-specific matched pairs test the actual route customers experience. Public benchmarks provide broader regression coverage:

These benchmarks operate at different layers. WEAT or SEAT can expose associations in representations even when generated outputs look harmless; RealToxicityPrompts or BOLD can expose output harms without explaining which internal representation caused them. Don't treat a benchmark pass as proof that customer routing is fair. A benchmark tests its own prompt distribution and label design. Don't treat one product slice as full safety coverage either. Use both, and keep the limitation attached to every report.

Why not optimize every metric?

Fairness metrics can conflict. When outcome prevalence differs across groups and predictions aren't perfect, a score calibrated within each group generally can't also equalize false-positive and false-negative rates across groups. Chouldechova demonstrated this incompatibility for risk scoring systems.^{[10]Reference 10Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.https://arxiv.org/abs/1610.07524} The engineering response isn't to give up or chase a single universal score. It's to define the customer harm, select a primary metric, monitor important counter-metrics, and document the accepted trade-off.

In this lab, the chosen outcome is rapid access to a supported reply. Equal opportunity is primary because it asks whether replies reviewers mark ready reach the fast path similarly. The FPR guardrail prevents a superficial fix that merely auto-serves more unclear replies.

Why not make calibration, equal TPR, and equal FPR universal hard gates for every model?

Answer

When outcome prevalence differs across slices and predictions aren't perfect, those criteria can conflict. Name the customer harm, choose a primary metric, keep important counter-metrics visible, and document the accepted trade-off instead of pretending one configuration satisfies every goal.

Write a release decision, not a fairness slogan

A fairness report must say what was tested, what failed, and why a candidate can't yet ship. That keeps a clean toy rerun from being promoted into an unsupported production claim.

A blocked result is progress. The team now has reproducible regressions, a primary metric, counter-metric guardrails, a likely failing stage, and explicit evidence still needed before release.