Hallucination Detection & Mitigation

The fairness lesson blocked a soft evaluator when it could route equivalent user requests differently. Factual reliability needs an equally strict boundary: a fluent large language model (LLM) answer can't invent an incident event just because the on-call engineer wants certainty.

The next incident question asks, "Is the embedding API fully recovered?" The admitted runbook update says the database failover completed on May 26 at 08:14 UTC and service remains investigating. It contains no recovery time. A draft answer that adds "Full recovery is expected by 14:30 UTC" may sound helpful, but the system has no source for that promise.

incident-answerer-v1 is a small serving gate. It turns an answer into atomic claims, checks each claim against versioned evidence, routes unsupported details to abstention, and records why release remains blocked. For a grounded incident answer, "not present in admitted evidence" is enough reason not to serve a factual claim. It doesn't prove the claim is false in the wider world.

Why overlap and answer-level scores miss incident failures

Sequence-overlap metrics and one answer-level judgment don't establish claim support:

• Lexical overlap misses operational polarity: BLEU and ROUGE compare surface overlap rather than whether admitted evidence supports a claim.^{[1]Reference 1ROUGE: A Package for Automatic Evaluation of Summaries.https://aclanthology.org/W04-1013/} "Database failover failed" and "Database failover succeeded" share most words but imply opposite actions.
• One score hides a critical false claim: A long incident summary can be mostly correct while one invented action, time, or status makes it unsafe. Split the answer into atomic claims and verify each one before computing a summary-level quality score.

Parametric override and the role of NLI verifiers

A generative model may ignore retrieved context and fall back to a stronger parametric association. For example, a log can state that Redis uses port 6380 while the draft substitutes the common default, 6379. Treat this contextual disregard as one failure mode, not a complete explanation for hallucinations.

An independent verifier can compare each atomic claim with admitted evidence. A Natural Language Inference (NLI) classifier is one candidate: it predicts whether a premise entails, contradicts, or is neutral toward a hypothesis. Those labels are model predictions, not mathematical guarantees. Calibrate the verifier on domain examples, preserve an abstain or review path, and keep deterministic checks for fields such as timestamps and status codes.

Three verdicts are enough to start

Hallucination taxonomies can become abstract quickly. For a retrieved incident record, begin with the source relationship:

Prior hallucination taxonomies distinguish outputs that conflict with supplied context from outputs that can't be verified from that context.^{[2]Reference 2A Survey on Hallucination in Large Language Modelshttps://arxiv.org/abs/2311.05232} In an open-world setting, an unsupported statement might later turn out true. In this answer path it's still unsafe, because the product promised evidence-grounded incident updates.

Put the evidence in code

The incident record isn't decorative text. It's the authority the answer must obey. Our first cell records its source ID and version alongside the candidate claims.

An LLM can write the recovery-time sentence because it has seen incident-update patterns. That doesn't make the sentence admissible. The admitted evidence has no recovery_eta field.

Verify each atomic claim

FActScore evaluates long-form generations by decomposing them into atomic facts and measuring how many a reliable source supports.^{[3]Reference 3FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.https://arxiv.org/abs/2305.14251} The verifier below uses the same reasoning on a small operational record: a claim is either supported by its cited source, missing from that source, contradicted by a different value, or missing an admitted source entirely.

The distinction matters. Recovery time isn't proven false; it's absent from this source. The resolved statement is worse because it conflicts with status=investigating.

Route the response as well as the score

A detector that writes a warning beside an unsafe response has not protected the reader. Once any factual claim fails, incident-answerer-v1 keeps supported information when possible and replaces the failed detail with a bounded statement. If no safe factual clause remains, it abstains.

Measure failure before choosing mitigation

A single blocked recovery time gives one regression case, not release evidence. Build a small suite that separates four failure modes: clean update, unsupported estimate, contradiction, and missing source.

For a grounded incident-status product, three simple metrics are useful:

Claim support can improve while unsafe serves remain unacceptable. A single contradicted incident status shown to an on-call engineer is still a serious failure.

Consistency is an alarm, not evidence

Black-box methods such as SelfCheckGPT compare an answer with additional sampled generations; disagreement can flag content the model may be inventing without an external database.^{[4]Reference 4SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models.https://arxiv.org/abs/2303.08896} Semantic entropy groups sampled answers by meaning and measures uncertainty over those clusters.^{[5]Reference 5Detecting Hallucinations in Large Language Models Using Semantic Entropyhttps://www.nature.com/articles/s41586-024-07421-0} Use these signals when no admitted source is available or when you need to prioritize expensive checks.

They don't authorize a recovery-time claim. A model can repeat the same unsupported estimate every time.

This fixes a common misconception: low uncertainty means "the model repeats itself," not "the fact is true."

Combine signals with the right authority

Evidence failure blocks claim even when samples agree. When evidence passes but generation is unstable, consistency can route case to review instead of serving answer that changes run to run.

Citations must be checked, not decorated

A response that prints [incident-INC-48291] isn't necessarily grounded. The citation must resolve to the admitted version and support the nearby claim. Otherwise a model can attach a real-looking source marker to an invented recovery promise.

This is a narrower version of claim-level factual evaluation: decompose, retrieve or admit evidence, verify, and retain the provenance for failed and passed claims. It also extends the RAG evaluation lesson: citation presence is no longer confused with citation support.

Attribute failures before adding complexity

An unsupported answer may begin with retrieval or generation:

Chain-of-Verification has a model draft an answer, plan verification questions, answer them independently, and produce a revised response. The paper reports fewer hallucinations across list questions, closed-book question answering, and long-form generation.^{[6]Reference 6Chain-of-Verification Reduces Hallucination in Large Language Modelshttps://arxiv.org/abs/2309.11495} That's a possible additional generator control. It doesn't replace an authoritative incident record or the final claim gate in this product.

Evaluate a candidate gate

Research benchmarks help compare methods, but they don't execute this incident feed, prompt, citation format, or serving route. Use external datasets for breadth and product regressions for release:

This is a useful repair on four deliberately small cases. It isn't enough to claim production factuality. The candidate abstains on three of four requests here because the test suite is failure-heavy by design.

Keep the mitigation stack small and testable

For this workflow, extra decoding tricks aren't next priority. Failure is already local: the draft adds a fact absent from a live source. Start with controls you can test directly against that source:

Hand the next lesson a trace

The next chapter can't monitor a correctness property that the trace never records. Store enough information to reconstruct the answer decision without retaining unnecessary incident text: source versions, verdict counts, route, and failure stage.

The candidate gate fixes its known fabricated-recovery-time regressions, but promotion remains blocked. It still needs a representative labeled holdout and an operational owner for alerts. The code has now produced exactly the facts an observability system should aggregate.

Mastery check

Mastery outcomes

Evaluation rubric

• Treats absence of source support as a serving failure without claiming outside-world falsity
• Decomposes a draft answer into testable factual claims
• Blocks both invented recovery times and claims contradicted by admitted state
• Demonstrates why stable repeated output isn't proof of truth
• Keeps citations coupled to verified source versions
• Locates first failure before proposing new mitigation layers
• Evaluates safe routing with both unsafe-serve rate and supported coverage
• Produces a trace that the monitoring lesson can aggregate

Follow-up questions

Common pitfalls

Repeated answers are mistaken for true answers

• Symptom: A stable sampled recovery time is served despite no source field establishing it.
• Cause: Consistency was treated as evidence.
• Fix: Use consistency to prioritize verification or review; require admitted source support for served facts.

Citations are generated but not verified

• Symptom: The answer has plausible source tags beside an invented recovery promise.
• Cause: Citation formatting was checked, but claim-to-source support was not.
• Fix: Verify each cited factual claim against its source version before serving it.

The retriever is blamed for generator overreach

• Symptom: More documents are indexed even though correct incident evidence was already retrieved.
• Cause: The team didn't identify the first failed stage.
• Fix: Separate evidence-admission failure from claim-generation failure in every trace.

Abstention is declared a release success on tiny fixtures

• Symptom: Four regression cases are used to claim production factual reliability.
• Cause: A focused regression suite was confused with a representative holdout.
• Fix: Keep the regressions, then collect labeled workflow slices and assign monitoring ownership before promotion.