Hallucination Detection & Mitigation

An on-call engineer asks a simple question: is the embedding API fully recovered? The runbook update accepted as evidence for INC-48291 says the database failover completed on May 26 at 08:14 UTC and the status is still investigating. It has no recovery time. Yet a draft adds, "Full recovery is expected by 14:30 UTC." The sentence sounds useful, but the system has no source for that promise.

Would you serve that draft? The fairness lesson blocked a judge that routed equivalent requests differently. This lesson applies the same hard stop to incident facts: a fluent large language model (LLM) answer can't turn an absent field into a fact just because someone wants certainty.

We'll build incident-answerer-v1 as a small serving gate. It splits an answer into atomic claims, checks each claim against a versioned record, routes unsupported details to abstention or a bounded rewrite, and records why release stays blocked. For this product, "not present in admitted evidence" is enough reason not to serve a factual claim. It doesn't prove the claim is false in the wider world.

Why overlap and answer-level scores miss incident failures

Start with the smallest failure. Two status lines can share nearly every word while demanding opposite action.

Bilingual Evaluation Understudy (BLEU) and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) compare surface overlap, not whether admitted evidence supports a claim.^{[1]Reference 1BLEU: a Method for Automatic Evaluation of Machine Translation.https://aclanthology.org/P02-1040[2]Reference 2ROUGE: A Package for Automatic Evaluation of Summaries.https://aclanthology.org/W04-1013/} "Database failover failed" and "Database failover succeeded" share most of their words, but the first calls for investigation while the second reports progress.

An answer-level score has the opposite blind spot. A long incident summary can be mostly correct while one invented action, time, or status makes it unsafe. Split the answer into atomic claims and verify each one before computing a summary-level quality score.

That leaves an open question: if overlap isn't the contract, what relationship to the source is?

Faithfulness to the record is the product contract

Before choosing a detector, decide what "true" means for this product. Natural language generation work calls a claim intrinsically hallucinated when it conflicts with supplied context, and extrinsically hallucinated when it adds an assertion the context can't verify. Huang et al. survey that split.^{[3]Reference 3A Survey on Hallucination in Large Language Modelshttps://arxiv.org/abs/2311.05232}

That distinction separates factuality (agreement with verifiable world knowledge) from faithfulness (staying inside the user's instructions and provided context). An incident gate can enforce the second contract even when it can't establish the first.

This product promises faithfulness to an admitted incident record. In an open-world setting, an unsupported recovery time might later turn out true. Here it's still an extrinsic hallucination and an unsafe serve because the product promised evidence-grounded updates.

A generator can also ignore the retrieved record and fall back to a stronger parametric habit. Suppose a log states that Redis uses port 6380 while the draft substitutes the common default, 6379. That's an intrinsic hallucination: the model replaced an admitted fact with its learned prior.

An independent verifier can compare each atomic claim with admitted evidence. A Natural Language Inference (NLI) classifier is one candidate: it predicts whether a premise entails, contradicts, or is neutral toward a hypothesis. Those labels are predictions, not mathematical guarantees. Calibrate the verifier on domain examples, keep an abstain or review path, and use deterministic checks for fields such as timestamps and status codes.

For a retrieved incident record, three verdicts are enough to start:

The gate is fail-closed on facts: if verification can't license a clause, that clause doesn't ship. A missing field therefore changes the route, not just the score.

Put the evidence in code

The incident record isn't decorative text. Its exact source ID and version form the admitted record the answer must obey. Before running the first cell, predict its output: four drafted claims, but no recovery_eta among the available fields. The cell records that source ID and version alongside the claims.

An LLM can write the recovery-time sentence because it has seen incident-update patterns. That doesn't make it admissible. The admitted evidence has no recovery_eta field, so the next check should reject that claim while keeping the three claims backed by fields.

Verify each atomic claim

FActScore makes the key move explicit: decompose a long generation into atomic facts, then measure how many a knowledge source supports. Its original Wikipedia evaluation used Supported, Not-supported, and Irrelevant labels.^{[4]Reference 4FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.https://arxiv.org/abs/2305.14251}

Ragas Faithfulness applies the same decompose-and-check idea to retrieved context: extract statements, then ask whether each one can be inferred from that context.^{[5]Reference 5RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217}

The verifier below applies that reasoning to our operational record. Predict the verdicts first: service, event, and event time should be supported; the missing recovery estimate (ETA) should be not_supported; a later resolved claim should be contradicted. A claim can also lack an admitted source entirely, which is why this product adds no_source to the usual labels.

The distinction matters. Recovery time isn't proven false; it's absent from this source. A resolved statement is stronger evidence of failure because it conflicts with status=investigating.

⚠️ Common mistake: A supported verdict is faithfulness to the admitted version, not a freshness check. If this runbook is an hour stale and the live service already recovered, the gate can still pass status=investigating. That's a source-admission problem, not a license to invent a new ETA.

Route the response as well as the score

The verifier caught the ETA, but a warning beside the original response would still leak it. Routing is part of the safety contract. Once any factual claim fails, incident-answerer-v1 keeps supported information when that's still safe and replaces the failed detail with a bounded statement. If no safe factual clause remains, it abstains.

Why doesn't not_supported mean the recovery time is false?

Answer

The verifier has scoped authority: the admitted incident record. It can show that the system has no evidence for a recovery time, which is enough to withhold the claim, but it can't claim to know every fact in the outside world.

Measure failure before choosing mitigation

A single blocked recovery time gives one regression case, not release evidence. Before expanding it, predict what a useful suite must separate: a clean update, an unsupported estimate, a contradiction, and a missing source. Each failure should lead to a different route or repair.

The suite gives us three different denominators. For a grounded incident-status product, track:

Claim support can improve while unsafe serves remain unacceptable. One contradicted incident status shown to an on-call engineer is still a serious failure, even if every other clause passed.

Consistency is an alarm, not evidence

Suppose four samples all repeat "Full recovery is expected by 14:30 UTC." Does zero disagreement make that sentence safe? No. White-box uncertainty methods look at token log-probabilities or token entropy, but token-level logit uncertainty measures vocabulary dispersion across surface forms. If an LLM varies between "14:30 UTC" and "2:30 PM UTC", high token entropy flags lexical variety rather than factual doubt.

Black-box methods bypass token logits by evaluating multiple stochastic generations. SelfCheckGPT queries consistency across sampled passages using NLI, LLM prompting, or BERTScore to detect sentence-level fabrications without external databases.^{[6]Reference 6SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models.https://arxiv.org/abs/2303.08896} Semantic entropy takes a meaning-level view: it groups sampled answers into semantic equivalence classes using bidirectional entailment, then computes entropy over those meaning clusters:

$SE(x) = -\sum_{c \in C} p(c) \log p(c)$

Farquhar et al. designed semantic entropy specifically for confabulations: arbitrary, sample-unstable generations where the model's answers fluctuate across stochastic draws.^{[7]Reference 7Detecting Hallucinations in Large Language Models Using Semantic Entropyhttps://www.nature.com/articles/s41586-024-07421-0}

Use these consistency probes when no admitted source is available, or to prioritize expensive verification paths. They don't authorize a recovery-time claim. If a model has a strong parametric bias, it can repeat the exact same unsupported estimate across every sample ($SE \approx 0$). Setting temperature to 0 doesn't fix that either; it only makes the ungrounded guess deterministic.

The next cell compares unstable samples with a repeated false estimate, then checks both against the admitted record. Predict which signal should decide the repeated ETA.

Low disagreement means "the model repeats itself," not "the fact is true." Evidence still has to license the claim.

Combine signals with the right authority

Evidence wins even when samples agree. When every claim is supported but generation is unstable, consistency can send the case to review instead of serving an answer that changes from run to run. The router below applies that order to the four fixtures: a mixed not_supported draft shortens, a conflict or missing source abstains, and only a fully supported answer consults disagreement.

Predict the three routes before running it: clean stable should serve, clean unstable should go to review, and a repeated unsupported ETA should still shorten.

Five sampled answers repeat the same unsupported claim. Does that consistency make the claim grounded?

Answer

No. Consistency can reveal instability when outputs disagree, but agreement can still repeat one unsupported model belief. Grounding requires comparison with trusted evidence.

Citations must be checked, not decorated

A response that prints [incident-INC-48291] isn't necessarily grounded. The citation must resolve to the admitted version and support the nearby claim. Otherwise a model can attach a real-looking source marker to an invented recovery promise.

Claim-level factual evaluation here follows a narrower loop: decompose, retrieve or admit evidence, verify, and retain the provenance for failed and passed claims. It extends the retrieval-augmented generation (RAG) evaluation lesson by separating citation presence from citation support.

An answer includes a valid-looking citation URL. What must a citation checker verify beyond its presence?

Answer

It must resolve the cited source and test whether the source supports the nearby claim. A real URL can still be irrelevant, stale, or contradicted by the answer.

Attribute failures before adding complexity

The same blocked answer can have different causes. Before adding another mitigation, locate the first failed stage:

Chain-of-Verification (CoVe) is a generator-side control: the model drafts an answer, plans verification questions, answers them independently, and produces a revised response. The paper reports fewer hallucinations across list questions, closed-book question answering, and long-form generation.^{[8]Reference 8Chain-of-Verification Reduces Hallucination in Large Language Modelshttps://arxiv.org/abs/2309.11495}

CoVe can tighten generation, but it doesn't replace an authoritative incident record or the final claim gate in this product. If the source is missing or stale, a more careful draft still needs the same boundary.

Read the traces before running the attribution cell. req-eta should fail during claim generation, req-missing during evidence admission, and req-vary during generation stability.

Evaluate a candidate gate

The traces identify a likely repair, but they don't prove it works outside these fixtures. Research benchmarks help compare methods, but they don't execute this incident feed, prompt, citation format, or serving route. Use external datasets for breadth and product regressions for release:

Before reading the candidate metrics, predict the trade-off: the baseline serves all four cases and is unsafe on three, while the candidate should remove unsafe serves without losing the clean update.

The candidate is a useful repair on four deliberately small cases, not production factuality evidence. It matches the product router taught earlier: the clean update serves, the invented ETA shortens (supported clauses stay, failed claim drops), and the contradicted status plus missing source abstain.

The suite is failure-heavy by design, so two of four requests withhold entirely. It's the safe cost of refusing unsupported or conflicting facts, not a reason to loosen the gate.

Keep the mitigation stack small and testable

For this workflow, extra decoding tricks aren't the next priority. The failure is local: the draft adds a fact absent from a live source. Start with controls you can test directly against that source, then add complexity only when a trace points elsewhere:

Hand the next lesson a trace

The next chapter can't monitor a correctness property that the trace never records. Store enough to reconstruct the answer decision without retaining unnecessary incident text: source versions, verdict counts, route, and failure stage.

Predict the promotion result from the contract: known regressions should pass, but missing holdout coverage and alert ownership must keep the candidate blocked.

The candidate gate fixes its known fabricated-recovery-time regressions, but promotion remains blocked. It still needs a representative labeled holdout and an operational owner for alerts. The code has now produced the facts an observability system should aggregate.