Capstone: Document QA

The earlier support-agent design chapter ended with a precise engineering request: retrieve a published policy record, cite it, reject private notes as authority, and abstain when approved evidence is missing. The predictive-ML capstones since then established the same production discipline around data lineage, promotion gates, monitoring, and rollback.

This capstone ships that request as a small product. You'll build a document question-answering (QA) service for merchant policies. Its first release is deliberately extractive: when the service has approved evidence that directly supports a question, it returns the policy text and its source identifier. When it doesn't, it returns an abstention. A later language model can make that answer friendlier only after it preserves the same evidence contract.

Start with the contract your caller already needs

Document QA isn't useful because it can chat about a PDF. It's useful when another system can depend on its answer. Here, that caller is the support agent you designed earlier.

Its exported brief looked like this:

That JSON is more valuable than a vague requirement such as "build RAG." It gives you one supported question and three failure conditions. You can turn each one into an executable test before choosing a vector database or a model provider.

Retrieval-augmented generation (RAG) combines generation with retrieved external memory so responses can use information outside a model's parameters.^[1] This capstone begins one step earlier: prove that the retrieved memory is authorized and sufficient. If that boundary fails, adding a generator only makes the failure sound smoother.

Define the artifact before the implementation

A capstone is a project another engineer can run and review. Before you write retrieval code, write its contract:

The implementation below is small enough to understand line by line, but the boundaries are production-shaped. You can replace its retrieval baseline with embeddings and reranking later without changing what callers or evaluators expect.

The first cell recreates the prior chapter's brief and supplies three candidate records: the authoritative return policy, an unrelated published policy, and a private note that attempts to authorize an immediate refund. Approval lives in a separate registry snapshot. A record can't declare itself authoritative by carrying a convenient Boolean.

Ingest approved evidence, not every available string

Ingestion is where many document QA demos quietly become unsafe. A naive implementation embeds every text field it can access. Then a private note, customer message, or obsolete draft may be retrieved beside policy text and look equally authoritative to the answering step.

The 2025 OWASP Top 10 for LLM Applications includes prompt injection and excessive agency. A support workflow with documents and tools must therefore distinguish text the system may read from evidence the system may use as policy authority.^[2]

Our ingestion rule is simple:

1. Read authority from a controlled registry snapshot, not from a candidate record.
2. Require a published, effective policy grant for the requested region.
3. Verify the parsed text against the registry's content hash and reject duplicate identifiers.
4. Keep a reason for every admission decision so an engineer can explain why a record never reached retrieval.

In a larger product, PDF parsing and chunk splitting happen before or during this step. The important design remains the same: every emitted chunk inherits a stable document identity and corpus snapshot from a controlled registry. A user upload doesn't become an approved merchant policy merely because parsing succeeded. A changed document also needs a new reviewed hash before it can enter the index.

Ship a transparent retrieval baseline first

You already learned dense and hybrid retrieval in the Applied LLM Engineering phase. A portfolio capstone doesn't improve by hiding its first test behind an opaque service call. Start with a deterministic baseline you can inspect, then demand that any embedding or reranking upgrade beats it on frozen fixtures.

The baseline below normalizes a few word forms and ranks approved chunks by meaningful term overlap. It isn't a claim that token overlap is enough for production. Retrieval only finds candidates. The answering step still needs to prove support before it cites one.

• if the required fixture fails, the corpus or contract is broken before a model enters the picture
• if a paraphrased fixture fails, you have evidence for adding dense retrieval
• if an unsupported question retrieves a nearby policy, the answer gate must still abstain

The important result isn't that a tiny scorer found the answer. It's that you now have a visible candidate attached to the same document the caller expects. Candidate retrieval is necessary, but it isn't permission to answer. Upgrade retrieval when a failing fixture proves why, not because "vector database" sounds more impressive in a README.

An honest baseline should also expose its limitations. The next check paraphrases damaged electronics as broken device. The overlap retriever abstains because it can't bridge that vocabulary change. That isn't a production success, but it's a useful test: a later dense or hybrid retriever must turn this specific gap into a cited answer without breaking the safety cases.

Make the first answer verifiably boring

A generative answer can summarize or rephrase a policy well, but it can also introduce a word the cited source never supported. The first shipped candidate uses an extractive answer: return an approved passage only when its normalized terms cover the question's normalized terms. That rule is intentionally conservative. It will abstain on valid paraphrases, but it won't confuse a nearby retrieved policy with proof.

That choice gives the project a trustworthy baseline. Once an LLM synthesis layer is added, it must match or improve answer usefulness while continuing to cite the same support and abstain on the same failures. A production support verifier will need richer semantics than term containment, but it still belongs after retrieval.

Treat abstention and injection resistance as product features

The happy path proves almost nothing by itself. The service becomes useful when it refuses questions the corpus doesn't support and ignores text that isn't authorized policy.

Here are the two failure cases exported by the support agent. The first one is deliberately close to the valid policy. Retrieval should find the return-policy chunk, but the answer gate must notice that the passage doesn't support a five-year warranty.

An instruction inside unapproved context is a distinct failure mode, so it deserves its own named fixture:

Notice the two distinct gates. The warranty question retrieves a nearby approved policy but still abstains because retrieval isn't answer support. The private note never enters the approved evidence collection at all. Source authority is application policy, not a model opinion.

Keep generation and long context behind an evaluation gate

At this point you could add an LLM that changes:

Refunds at or above 500 USD require specialist approval before a refund is queued.

into:

Your 900 USD damaged-item refund needs specialist approval before it can be queued.

That can improve user experience, but it also introduces a new failure mode: the generated sentence may claim more than its evidence. Preserve versioned citations, retain the extractive baseline in your tests, and add a claim-support grade before promoting synthesis.

Similarly, don't dump an entire policy binder into the prompt to avoid retrieval work. Liu et al. measured large changes in answer quality when relevant information moved within long contexts, with performance often falling when that information appeared in the middle.^[3] Their result is a reason to test context construction, not a promise that one fixed top_k works for every model and corpus.

Put the contract behind an API

The core logic is framework independent. Packaging it as an HTTP service gives the support agent a stable interface and gives another engineer something they can run. FastAPI can validate Pydantic request bodies and response models, which makes the evidence contract explicit in generated API documentation.^[4]

This adapter is intentionally short. Keep the tested retrieval and answer functions in a service module; let the web layer translate JSON into typed calls. Use a nested response model for citations rather than dict[str, str]. Otherwise, generated API documentation can say only that each citation is a string-valued object, not that corpus_version, document_id, chunk_id, and section are required.

Before wiring a framework, test the payload the endpoint is allowed to expose. It should contain the corpus snapshot, citation identifiers for an approved answer, and no rejected record identifiers. Snapshot identity matters because a document ID alone can't prove which approved index answered an old request.

For the required fixture, POST /answer returns the same evidence contract the support agent can consume:

A reviewable repository needs more than app.py:

Docker's Python guide demonstrates the ordinary packaging path: declare a Python image and dependencies, copy the service, expose its port, and run it in a container.^[5] Your README should contain the exact commands that build the image, call /answer, and run evals from a fresh checkout.

Export dashboard-ready rows, not a success anecdote

The next capstone is an evaluation dashboard. Give it real rows rather than a screenshot of one passing query. Each row should say which frozen dataset, implementation run, and corpus snapshot produced it; what question ran; which result was expected; what was cited; and whether the contract passed.

The untrusted_instruction row is important even though it abstains. A future retrieval rewrite could accidentally index private notes. The unsupported_question row tests a different boundary: retrieval finds a nearby approved policy, but the answer gate still refuses to invent a warranty promise. A dashboard should report both slices separately before the support agent is allowed to rely on the service.

Gate the baseline before you extend it

The baseline doesn't claim to handle every way a customer may phrase a policy question. Its gate is narrower and honest: it satisfies the required consumer fixture, fails closed on two required safety cases, exports rows for the next capstone to extend, and refuses to pass when row coverage is ambiguous. Passing this gate proves the core contract, not that a fixture-only script is ready for deployment.

This is a genuine capstone milestone, not a deployment approval or a finished universal QA engine. Add real document loading and endpoint tests to submit the project. Add paraphrase fixtures before adding dense retrieval, synthesis fixtures before allowing generated wording, and policy-version and region slices before putting more merchant corpora behind the API. Keep exact row coverage and corpus identity in the gate so missing or duplicated evidence can't look like a clean run.

Practice: Break the evidence contract

Run the cells again after each mutation. Revert one mutation before trying the next.

1. Add a RegistryGrant for seller-note-48291 with source_kind="published_policy", published=True, effective=True, region="US", and the note's SHA-256 hash. Which eval row fails? Why is this an authorization failure rather than a ranking problem?
2. Remove the supports_extractive_answer(...) check from answer_question(...) and return the first retrieved candidate. Which safety fixture fails? What does that prove about confusing retrieval with answer support?
3. Change the approved return-policy text without changing its registry hash. Which admission reason appears? Why should a content edit require a reviewed registry update?
4. Run baseline_report([row for row in rows if row.slice != "untrusted_instruction"]), then baseline_report(rows + [rows[0]]). Which coverage errors appear? Why should omitted or duplicated rows block promotion?
5. Assume you add dense retrieval to fix paraphrase_gap. Which existing rows and known miss must remain frozen during the comparison?

Submission checklist

A portfolio-ready submission should let a reviewer answer each question with a file or command:

Evaluation rubric

Use this rubric to review the artifact, not only its demo output:

• Evidence boundary: Only registry-approved, hash-matched policy records enter a versioned evidence index, and rejected records carry reasons.
• Answer contract: Candidate retrieval and answer support are separate gates. Supported answers carry versioned document citations; unsupported or adversarial queries abstain without citations.
• Upgrade honesty: A known paraphrase miss is recorded as an improvement target rather than presented as solved.
• Product packaging: A typed endpoint, real corpus input, automated contract tests, and repeatable run commands exist in the submitted repository.
• Capstone handoff: Row-level outputs are ready for slice aggregation in the evaluation dashboard.

Common failures

Treating every parsed document as evidence

Symptom: A private seller note appears as the citation for a customer-facing answer. Cause: Parsing and evidence admission were collapsed into one operation. Fix: Require a controlled registry grant, publication state, effective version, region, and content hash before indexing chunks.

Treating retrieval as answer support

Symptom: A warranty question cites a return-policy passage that never mentions a warranty. Cause: The service returns its highest-ranked chunk without checking whether that chunk supports the requested claim. Fix: Keep candidate retrieval and answer support as separate gates. Abstain when no candidate supports the question.

Hiding an untested generator behind a polished UI

Symptom: Answers read naturally, but no test checks whether their claims occur in approved support text. Cause: Synthesis was added before a supported and unsupported baseline existed. Fix: Release an extractive contract first, then grade any generated candidate against its cited evidence.

Calling one passing example an evaluation

Symptom: A retrieval change improves the demo question and silently breaks abstention or injection handling. Cause: The project saved a success screenshot rather than frozen, versioned evaluation rows. Fix: Export result rows by slice and block promotion when a required row is missing, duplicated, or failed.

Self-check questions