Capstone: Document QA

The support-agent design left a precise engineering request: retrieve a published policy record, cite it, reject private workspace notes as authority, and abstain when approved evidence is missing. The predictive-ML capstones since then established the same production discipline around data lineage, promotion gates, monitoring, and rollback.

This capstone ships that request as a small product. You'll build a document question-answering (QA) service for access policies. Its first release is deliberately extractive: when the service has approved evidence that directly supports a question, it returns the policy text and its source identifier. When it doesn't, it returns an abstention. A later language model can make that answer friendlier only after it preserves the same evidence contract.

Start with the contract your caller already needs

Document QA isn't useful because it can chat about a PDF. It's useful when another system can depend on its answer. Here, that caller is the access agent you designed earlier.

Its exported brief looked like this:

That JSON is more useful than a vague requirement such as "build RAG." It names one supported question and three failure conditions. You can turn each one into an executable test before choosing a vector database or a model provider.

Retrieval-augmented generation (RAG) combines generation with retrieved external memory so responses can use information outside a model's parameters.^{[1]Reference 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401} This capstone begins one step earlier: prove that the retrieved memory is authorized and sufficient. If that boundary fails, adding a generator only makes the failure sound smoother.

Define the artifact before the implementation

A capstone is a project another engineer can run and review. Before you write retrieval code, write its contract:

This implementation is small enough to understand line by line, but the boundaries are production-shaped. You can replace its retrieval baseline with embeddings and reranking later without changing what callers or evaluators expect.

The first cell recreates the prior chapter's brief and supplies three candidate records: the authoritative access policy, an unrelated published incident policy, and a private workspace note that attempts to authorize a privileged-access grant. Approval lives in a separate registry snapshot. A record can't declare itself authoritative by carrying a convenient Boolean.

Ingest approved evidence, not every available string

Ingestion is where many document QA demos quietly become unsafe. A naive implementation embeds every text field it can access. Then a private workspace note or obsolete draft may be retrieved beside policy text and look equally authoritative to the answering step.

The 2025 OWASP Top 10 for LLM Applications includes prompt injection and excessive agency. A support workflow with documents and tools must therefore distinguish text the system may read from evidence the system may use as policy authority.^{[2]Reference 2OWASP Top 10 for Large Language Model Applicationshttps://genai.owasp.org/llm-top-10/}

Our ingestion rule is simple:

1. Read authority from a controlled registry snapshot, not from a candidate record.
2. Require a published, effective policy grant for the requested region.
3. Verify the parsed text against the registry's content hash and reject duplicate identifiers.
4. Keep a reason for every admission decision so an engineer can explain why a record never reached retrieval.

In a larger product, PDF parsing and chunk splitting happen before or during this step. The important design remains the same: every emitted chunk inherits a stable document identity and corpus snapshot from a controlled registry. A workspace upload doesn't become an approved access policy merely because parsing succeeded. A changed document also needs a new reviewed hash before it can enter the index.

Ship a transparent retrieval baseline first

You already learned dense and hybrid retrieval in the Applied LLM Engineering phase. A portfolio capstone doesn't improve by hiding its first test behind an opaque service call. Start with a deterministic baseline you can inspect, then demand that any embedding or reranking upgrade beats it on frozen fixtures.

The baseline below normalizes a few word forms and ranks approved chunks by meaningful term overlap. It isn't a claim that token overlap is enough for production. Retrieval only finds candidates. The answering step still needs to prove support before it cites one.

• if the required fixture fails, the corpus or contract is broken before a model enters the picture
• if a paraphrased fixture fails, you have evidence for adding dense retrieval
• if an unsupported question retrieves a nearby policy, the answer gate must still abstain

The important result isn't that a tiny scorer found the answer. It's that a visible candidate is attached to the same document the caller expects. Candidate retrieval is necessary, but it isn't permission to answer. Upgrade retrieval when a failing fixture proves why, not because "vector database" sounds more impressive in a README.

An honest baseline should also expose its limitations. The next check paraphrases production API key access as locked credential. The overlap retriever abstains because it can't bridge that vocabulary change. That isn't a production success, but it's a useful test: a later dense or hybrid retriever must turn this specific gap into a cited answer without breaking the safety cases.

Make the first answer verifiably boring

A generative answer can summarize or rephrase a policy well, but it can also introduce a claim the cited source never supported. The first shipped candidate uses an extractive answer: return an approved passage only when a bound support receipt labels the question-passage relation entailed. A validated verifier or human reviewer writes that receipt. contradicted, unknown, missing, or stale adjudication always abstains.

Term containment remains useful as a smoke test for obvious retrieval misses. It never authorizes an answer because negation can preserve nearly every term while reversing meaning. This separation gives the project a trustworthy baseline. Once an LLM synthesis layer is added, it must improve usefulness while preserving citation, adjudication, and abstention behavior.

Treat abstention and injection resistance as product features

The happy path proves almost nothing by itself. The service becomes useful when it refuses questions the corpus doesn't support and ignores text that isn't authorized policy.

The access agent exports two failure cases. The first one is deliberately close to the valid policy. Retrieval should find the access-policy chunk, but the answer gate must notice that the passage doesn't support a permanent admin access.

An instruction inside unapproved context is a distinct failure mode, so it deserves its own named fixture:

Notice the distinct gates. Corpus admission decides which sources may act as authority. Retrieval finds candidates. A bound semantic adjudication decides whether a candidate entails, contradicts, or leaves the proposed answer unknown. The admin-access question retrieves a nearby approved policy but still abstains on unknown; the private workspace note never enters approved evidence at all.

Keep generation and long context behind an evaluation gate

A tempting next step is to add an LLM that changes:

Privileged scopes require manager approval before access is queued.

into:

Your privileged access restore needs manager approval before it can be queued.

That can improve user experience, but it also introduces a new failure mode: the generated sentence may claim more than its evidence. Preserve versioned citations, retain the extractive baseline in your tests, and require source-bound entailed, contradicted, or unknown adjudication before promoting synthesis.

Similarly, don't dump an entire policy binder into the prompt to avoid retrieval work. Liu et al. measured large changes in answer quality when relevant information moved within long contexts, with performance often falling when that information appeared in the middle.^{[3]Reference 3Lost in the Middle: How Language Models Use Long Contextshttps://arxiv.org/abs/2307.03172} Their result is a reason to test context construction, not a promise that one fixed top_k works for every model and corpus.

Put the contract behind an API

The core logic is framework independent. Packaging it as an HTTP service gives the access agent a stable interface and gives another engineer something they can run. FastAPI can validate Pydantic request bodies and response models, which makes the evidence contract explicit in generated API documentation.^{[4]Reference 4FastAPI Documentation.https://fastapi.tiangolo.com/}

This adapter is intentionally short. Keep the tested retrieval and answer functions in a service module; let the web layer translate JSON into typed calls. Use a nested response model for citations rather than dict[str, str]. Otherwise, generated API documentation can say only that each citation is a string-valued object, not that corpus_version, document_id, chunk_id, and section are required.

Before wiring a framework, test the payload the endpoint is allowed to expose. It should contain the corpus snapshot, citation identifiers, semantic label, and support-receipt ID for an approved answer, with no rejected record identifiers. Snapshot identity matters because a document ID alone can't prove which approved index answered an old request.

For the required fixture, POST /answer returns the same evidence contract the access agent can consume:

A reviewable repository needs more than app.py:

Docker's Python guide demonstrates the ordinary packaging path: declare a Python image and dependencies, copy the service, expose its port, and run it in a container.^{[5]Reference 5Docker Documentation.https://docs.docker.com/} Your README should contain the exact commands that build the image, call /answer, and run evals from a fresh checkout.

Export dashboard-ready rows, not a success anecdote

The next capstone is an evaluation dashboard. Give it real rows rather than a screenshot of one passing query. Each row should say which frozen dataset, implementation run, and corpus snapshot produced it; what question ran; which result was expected; what was cited; and whether the contract passed.

The untrusted_instruction row is important even though it abstains. A future retrieval rewrite could accidentally index private workspace notes. The unsupported_question row tests a different boundary: retrieval finds a nearby approved policy, but its validated-verifier receipt says unknown. A dashboard should report both slices separately before the access agent is allowed to rely on the service.

These rows are dashboard-compatible: they share fixture ids (missing_admin_policy, not a hyphenated variant), stamp grader_version, and export semantic labels, receipt IDs, and failure_codes. The next capstone freezes policy-qa-v2 by adding a policy_paraphrase fixture while keeping the same grader contract and the three baseline ids above.

Gate the baseline before you extend it

The baseline doesn't claim to handle every way an operator may phrase a policy question. Its gate is narrower and honest: it satisfies the required consumer fixture, fails closed on two required safety cases, exports rows for the next capstone to extend, and refuses to pass when row coverage is ambiguous. Passing this gate proves the core contract, not that a fixture-only script is ready for deployment.

This is a genuine capstone milestone, not a deployment approval or a finished universal QA engine. Add real document loading and endpoint tests to submit the project. Add paraphrase fixtures before adding dense retrieval, synthesis fixtures before allowing generated wording, and policy-version and region slices before putting more access-policy corpora behind the API. Keep exact row coverage and corpus identity in the gate so missing or duplicated evidence can't look like a clean run.

Extend the service into a deep-research agent

The baseline answers one question from one directly supporting policy chunk. Some requests require a longer loop: decompose a comparison, search several approved corpora, read competing evidence, reconcile conflicts, and write a report whose claims carry citations. That workload is deep research, not a larger call to /answer.

Current provider APIs expose this distinction directly. Google's Deep Research documentation describes an agent that plans, searches, reads, iterates, and produces a cited report. Because a run can take minutes, its API requires background execution and exposes status for polling or streaming.^{[6]Reference 6Gemini Deep Research Agenthttps://ai.google.dev/gemini-api/docs/deep-research} Treat that product as one current implementation. Your capstone contract should stay provider-neutral so model or provider changes don't rewrite job state, source authority, or evaluation rows.

The extractive service remains a building block. Each research step may call retrieval many times, but every source still passes admission and every report claim needs a source-bound semantic adjudication. Orchestration adds duration and breadth; it doesn't weaken the evidence boundary.

Define bounded request before starting work

An open-ended prompt such as "research access policy" gives the agent no stopping rule. The request should name question, allowed source classes, expected output, and hard budgets. For this product, keep private workspace notes excluded from policy authority even when they could help the agent discover search terms.

Budgets are product controls, not prompt suggestions. Harness code should reject a new search after max_steps, avoid admitting a thirteenth source, and move job to a named timeout or partial state when deadline expires. Store requested policy beside job so resumed worker can't silently broaden source scope.

Run research as resumable background job

A synchronous HTTP request ties work lifetime to client connection. Research may outlive load-balancer timeout or browser tab, so create job, checkpoint each phase, and let callers poll, stream, cancel, or resume. Google's current Deep Research API uses this background pattern for the same reason.^{[6]Reference 6Gemini Deep Research Agenthttps://ai.google.dev/gemini-api/docs/deep-research}

Use explicit states instead of one running Boolean:

The mini-lab below enforces allowed transitions and persists checkpoint name with step count. A new worker can reconstruct job from durable record and continue from source-batch-2 without repeating completed searches.

Cancellation stops future tool calls and preserves completed receipts. Resume first verifies checkpoint and leases job to one worker. If a failed source fetch is safe to retry, reuse stable operation key so restart can't duplicate uploads, writes, or paid searches.

Admit sources, then verify claims

Research agents read web pages, uploaded files, search results, and tool output. Google's documentation warns that supplied files and public web pages may contain prompt injection and that mixing sensitive data with web access creates exfiltration risk.^{[6]Reference 6Gemini Deep Research Agenthttps://ai.google.dev/gemini-api/docs/deep-research} Treat fetched content as data. It can't grant itself authority, expand allowed tools, or change report instructions.

Each admitted source needs a receipt such as source_id, source class, access scope, retrieval time, parser version, and content hash. Each factual claim needs exact source spans plus an adjudication receipt bound to claim text and source snapshots. Citation presence isn't enough: a polished report can cite a real policy that never states its nearby claim.

The second mini-lab keeps lexical overlap as a diagnostic only. Its contradiction deliberately passes that smoke test because the three-letter word not reverses the meaning while every longer term still appears. Report completion depends instead on entailed, contradicted, or unknown output from an approved human review path or a verifier with a recorded validation run.

The overlap smoke test returns True for the contradicted manager-approval claim, yet the report still fails. Completion authority comes from the bound semantic receipt, not shared words. Human review and validated verifiers can both write receipts, but their identities and validation records belong to application policy. Preserve exact claim text, source spans, and source hashes so a disputed adjudication can be replayed. contradicted and unknown remain distinct failure states; conflicting sources should produce an explicit unresolved-conflict section rather than a forced synthesis.

Expose lifecycle through API

Keep /answer for fast supported lookups. Add separate resource-oriented endpoints for research:

Return 202 Accepted when work is queued, not a pretend completed answer. Status responses should expose phase and budget use without leaking raw chain-of-thought or private source content. A completed report returns source, span, claim, and adjudication receipts; a failed or partial result names which gate stopped promotion.

Add these files to capstone repository:

Test restart after every phase, cancellation during a tool call, stale worker lease, budget exhaustion, source hash change, missing citation, citation mismatch, conflicting sources, and private-note injection. Those cases turn deep research from a long prompt into an auditable product.

Practice: break the evidence contract

Run the cells again after each mutation. Revert one mutation before trying the next.

1. Add a RegistryGrant for workspace-note-48291 with source_kind="published_policy", published=True, effective=True, region="US", and the note's SHA-256 hash. Which admission assertion fails? Why is this an authorization failure rather than a ranking problem?
2. Remove the verified_support_receipt(...) check from answer_question(...) and return the first retrieved candidate. Which safety fixture fails? What does that prove about confusing retrieval with authorization?
3. Change the approved access-policy text without changing its registry hash. Which admission reason appears? Why should a content edit require a reviewed registry update?
4. Run baseline_report([row for row in rows if row.slice != "untrusted_instruction"]), then baseline_report(rows + [rows[0]]). Which coverage errors appear? Why should omitted or duplicated rows block promotion?
5. Assume you add dense retrieval to fix paraphrase_gap. Which existing rows and known miss must remain frozen during the comparison?
6. Change report_gate(...) so lexical_overlap_smoke=True authorizes a claim without reading its adjudication receipt. Which contradicted claim now passes?

Submission checklist

A portfolio-ready submission should let a reviewer answer each question with a file or command: