Production RAG Pipelines

Your ShopFlow support agent can now be evaluated as an agent. It still can't answer a policy question safely unless it receives the right evidence. A resolution rule may change by region, merchant, product condition, or policy revision. An answer that sounds right but cites last year's rule can authorize a costly mistake.

Retrieval-augmented generation (RAG) gives a language model retrieved evidence at answer time instead of expecting its weights to contain current private facts.^[1] In production, that simple idea becomes a system contract: index traceable evidence, retrieve only evidence the user may see, generate from that evidence, abstain when it isn't enough, and retain a trace that a reviewer can inspect.

This lesson builds that contract for policy-answerer-v1. You won't implement BM25, dense embeddings, fusion, or reranking here. Those retrieval algorithms belong in the next lessons. Here, you'll make the pipeline around any retriever trustworthy.

The promise the service must keep

Suppose Luna, an EU support specialist, asks:

Can a customer get a replacement for a damaged refurbished laptop delivered 10 days ago?

The answer isn't just text. A release-worthy response must satisfy four properties:

The data path has an offline side and an online side:

The indexer produces evidence records when documents change. The request path filters those records by identity and policy state, asks a retriever for candidates, packs source-labelled context, and returns either a supported answer or an abstention. The release path replays frozen questions before a new index, prompt, retriever, or model version serves users.

Build the evidence record

Earlier chunking lessons showed how to cut a document into searchable spans. A production service adds the fields needed to use those spans later: a stable document identifier, a parent section for citations, a version, an effective date range, a region, and access tags.

Our tiny corpus has two current policies and one superseded policy. Notice that the US and EU rules deliberately differ. That difference turns an access-control bug into a visible wrong answer.

The record is deliberately more boring than a model call. That's good. Every later stage can now prove which policy revision it used. The fixed EVAL_DATE also makes this replay reproducible instead of changing behavior with the wall clock.

Retrieve small, cite enough context

Indexing whole policy pages gives a retriever too much irrelevant prose. Indexing one sentence can lose surrounding exceptions. Parent-child indexing stores a compact child span for search and a parent section for final evidence. The retriever can match the child ID, then context assembly can fetch the parent section and its stable citation metadata.

The compact lab keeps child text inline and carries document_id plus parent_id. A full parent-child implementation resolves parent_id to a version-matched, permitted parent section before packing. Keep those fields separate instead of parsing meaning out of an ID string.

Chunk overlap remains useful when a sentence straddles a boundary, but it isn't a magic setting. Treat it as an indexing candidate that must survive retrieval tests on your own policy questions.

The next fragment checks a basic index invariant: at most one current record for the same region and parent section. Two active revisions would allow the request path to retrieve contradictory promises.

Put authorization before similarity

An embedding index doesn't know whether Luna can read a document. A highly similar restricted chunk is still forbidden. The safe order is:

1. Determine the caller's tenant, role, region, and request date from trusted application state.
2. Select admissible evidence by those fields.
3. Search only within that admissible set, or use a store that enforces the filter inside retrieval.
4. Pass only returned permitted text to context assembly and logs visible to the caller.

Filtering after text has already reached the model is too late. The model, request trace, cache, or error report may already contain restricted content.

The lab uses a simple term-overlap search so its authorization behavior is obvious. Its retrieve() interface is the part you'll replace with hybrid search in the next chapter.

This retriever isn't production search: its two-term threshold rejects weak hits, but it misses paraphrases such as "reconditioned notebook." It is a clean test double for the surrounding pipeline. Once the authorization and trace contract work, you can improve recall without weakening the boundary.

Failure test: a tempting but forbidden result

A useful test shouldn't only prove success. It should include a result that would rank well if the permission filter were missing.

Pack evidence for a grounded answer

Retrieval produces candidate records, not an answer. Context assembly must give the generator source labels, version information, and a clear instruction to abstain when the evidence doesn't establish the requested promise.

Don't stuff every near-match into the prompt. Even when a context window fits a large amount of text, models can use relevant information less reliably when it sits among long distractors, especially in the middle of a long input.^[2] Pack the strongest permitted evidence first, keep the set small, and evaluate this policy rather than assuming it works.

Answer or abstain

In an actual service, a language model would receive the packed context and an instruction to cite it. For the lab, a deterministic answerer makes the core contract inspectable: it emits the rule only when the required evidence is present and otherwise refuses to promise a resolution outcome.

The lab uses string checks only to make the invariant runnable. A real candidate may use a model, structured citations, and claim verification. The release rule remains: if permitted current evidence doesn't support a material policy claim, the system must abstain or escalate.

Record a reproducible request trace

The agent evaluation lesson treated traces as observable release evidence. RAG needs the same discipline. Record versions and decisions needed to reproduce an answer, but don't copy restricted source text into broad logs.

Budget latency by stage

RAG adds work before the first generated token: authorization, retrieval, and context packing. Track that work separately from time to first token (TTFT). The fixture records generate as time after the first token, so its stages add up to one request timeline. If TTFT rises after a corpus change while retrieval stays fast, prompt size may be the issue.

Use frozen cases as a release gate

An appealing demo question doesn't establish reliability. Create frozen cases from policy questions, authorization attacks, outdated revisions, and missing-evidence requests. Keep the expected evidence IDs with each case. This separates retrieval failure from generation failure before users see the candidate.

RAG evaluation research also separates retrieval evidence quality from answer faithfulness and relevance rather than hiding all failures inside one final score.^[3] The dedicated RAG evaluation lesson will implement those metrics. This lesson starts with hard release assertions that catch expensive mistakes immediately.

The minimal suite already checks three high-impact failures: a forbidden chunk, a superseded chunk, and an unsupported answer. A serious deployment adds paraphrases, policy conflicts, index deletion cases, model-judge calibration, human reviews, and latency distributions.

What to block before launch

Ship the policy-answerer-v1 artifact

You now have the bones of a production RAG service. Turn the fragments into a small portfolio artifact:

1. Store three versions of an electronics-return policy with chunk_id, document_id, parent_id, effective dates, region, and ACL tags.
2. Add at least four frozen questions: a supported EU request, a US-only request, a restricted merchant-policy attack, and a question whose answer is absent.
3. Implement a retriever behind the retrieve() contract. Keep the simple overlap baseline first.
4. Pack evidence with stable source IDs and return a cited answer or a documented abstention.
5. Write one trace JSON row per request without logging restricted text.
6. Produce a release report listing authorization, freshness, evidence, grounding, and latency gates.

Mastery check

You are ready to design a production RAG pipeline when you can:

• Explain why a RAG answer must be treated as an evidence-backed operation, not merely generated prose.
• Define a versioned chunk record with stable citation identity, effective dates, region, and ACL metadata.
• Enforce authorization and policy freshness before any retrieved text reaches the model.
• Pack small, cited context and require an abstention when permitted evidence can't support the answer.
• Record a reproducible request trace without storing restricted text in unsafe logs.
• Gate a candidate on authorization, freshness, grounding, abstention, and latency behavior.
• Preserve that contract while a later retrieval implementation replaces the simple search baseline.

Evaluation rubric

Common pitfalls