Production RAG Pipelines

Your internal security-policy support agent can now be evaluated as an agent. It still can't answer a policy question safely unless it receives the right evidence. A rotation rule may change by region, account type, risk state, or policy revision. An answer that sounds right but cites last year's rule can authorize a costly mistake.

Retrieval-augmented generation (RAG) gives a language model retrieved evidence at answer time instead of expecting its weights to contain current private facts.^{[1]Reference 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401} In production, that simple idea becomes a system contract: index traceable evidence, retrieve only evidence the user may see, generate from that evidence, abstain when it isn't enough, and retain a trace that a reviewer can inspect.

For policy-answerer-v1, that contract is the main deliverable. You won't implement BM25, dense embeddings, fusion, or reranking here. Those retrieval algorithms belong in the next lessons. Here, you'll make the pipeline around any retriever trustworthy.

The promise the service must keep

Suppose Luna, an EU support specialist, asks:

Can a stale service-account key be rotated automatically if the risk signal arrived 10 days ago?

The answer isn't just text. A release-worthy response must satisfy four properties:

The data path has an offline side and an online side. When documents change, the indexer produces evidence records. The request path filters those records by identity and policy state, asks a retriever for candidates, packs source-labelled context, and returns either a supported answer or an abstention. Before a new index, prompt, retriever, or model version serves users, the release path replays frozen questions.

Build the evidence record

Earlier chunking lessons showed how to cut a document into searchable spans. A production service adds the fields needed to use those spans later: a stable document identifier, a parent section for citations, a version, an effective date range, a region, and an access control list (ACL) tag.

Our tiny corpus has three current policies and one superseded policy. Notice that the US and EU rules deliberately differ. That difference turns an access-control bug into a visible wrong answer.

The record is deliberately more boring than a model call. That's good. Every later stage can now prove which policy revision it used. The fixed EVAL_DATE also makes this replay reproducible instead of changing behavior with the wall clock.

Retrieve small, cite enough context

Indexing whole policy pages gives a retriever too much irrelevant text. Indexing one sentence can lose surrounding exceptions. Parent-child indexing stores a compact child span for search and a parent section for final evidence. The retriever can match the child ID, then context assembly can fetch the parent section and its stable citation metadata.

The compact lab keeps child text inline and carries document_id plus parent_id. A full parent-child implementation resolves parent_id to a version-matched, permitted parent section before packing. Keep those fields separate instead of parsing meaning out of an ID string.

Chunk overlap remains useful when a sentence straddles a boundary, but it isn't a default setting to trust blindly. Treat it as an indexing candidate that must survive retrieval tests on your own policy questions.

The next fragment checks a basic index invariant: at most one current record for the same region and policy document. Two active revisions would allow the request path to retrieve contradictory promises.

Put authorization before similarity

An embedding index doesn't know whether Luna can read a document. A highly similar restricted chunk is still forbidden. The safe order is:

1. Determine the caller's tenant, role, region, and request date from trusted application state.
2. Select admissible evidence by those fields.
3. Search only within that admissible set, or use a store that enforces the filter inside retrieval.
4. Pass only returned permitted text to context assembly and logs visible to the caller.

Filtering after text has already reached the model is too late. The model, request trace, cache, or error report may already contain restricted content.

The fixture below has one internal security-policy tenant, so it models region and ACL tags directly. A multi-tenant service must enforce tenant isolation inside the same permission boundary; tenant identity can't depend on model instructions.

The lab uses a simple term-overlap search so its authorization behavior is obvious. Its retrieve() interface is the part you'll replace with hybrid search in the next chapter.

This retriever isn't production search: its two-term threshold rejects weak hits, but it misses paraphrases such as "refresh expired machine credential." It's a clean test double for the surrounding pipeline. Once the authorization and trace contract work, you can improve recall without weakening the boundary.

Failure test: a tempting but forbidden result

A useful test shouldn't only prove success. It should include a result that would rank well if the permission filter were missing.

Pack evidence for a grounded answer

Retrieval produces candidate records, not an answer. Context assembly must give the generator source labels, version information, and a clear instruction to abstain when the evidence doesn't establish the requested promise.

Don't stuff every near-match into the prompt. Even when a context window fits a large amount of text, models can use relevant information less reliably when it sits among long distractors, especially in the middle of a long input.^{[2]Reference 2Lost in the Middle: How Language Models Use Long Contextshttps://arxiv.org/abs/2307.03172} Pack the strongest permitted evidence first, keep the set small, and evaluate this policy rather than assuming it works.

Answer or abstain

In an actual service, a language model would receive the packed context and an instruction to cite it. For the lab, a deterministic answerer makes the core contract inspectable: it emits the rule only when the required evidence is present and otherwise refuses to promise a resolution outcome.

The lab uses string checks only to make the invariant runnable. A real candidate may use a model, structured citations, and claim verification. The release rule remains: if permitted current evidence doesn't support a material policy claim, the system must abstain or escalate.

Record a reproducible request trace

The agent evaluation lesson treated traces as observable release evidence. RAG needs the same discipline. Record versions and decisions needed to reproduce an answer, but don't copy restricted source text into broad logs.

Setting temperature = 0 doesn't make an answer reproducible. Greedy decoding removes sampling randomness, but outputs may still change when model weights behind an alias, prompt template, retriever configuration, or index changes. Pin a model version or weight hash instead of latest, hash exact prompt template, record retriever and reranker configuration, and retain an index snapshot identifier. Extend this trace's index_version with model_version and prompt_hash. Even with those pins, provider-hosted generation may be only approximately reproducible because hardware and batching can perturb low-probability tokens. Store produced answer as audit evidence instead of assuming byte-for-byte regeneration.

Budget latency by stage

RAG adds work before the first generated token: authorization, retrieval, and context packing. Keep two measurements separate:

• End-to-end time to first token (TTFT) is what the caller feels from request arrival until the first generated token.
• Model TTFT starts when the service sends packed context to the model and ends when the first generated token arrives.

The fixture records model_ttft plus generate, where generate is time after the first token. That makes each stage additive while preserving the caller-visible TTFT. If model TTFT rises after a corpus change while retrieval stays fast, packed prompt size may be the issue.

Use frozen cases as a release gate

An appealing demo question doesn't establish reliability. Create frozen cases from policy questions, authorization attacks, outdated revisions, and missing-evidence requests. Keep the expected evidence IDs with each case. This turns the suite into an eval gate and separates retrieval failure from generation failure before users see the candidate.

RAG evaluation research also separates retrieval evidence quality from answer faithfulness and relevance rather than hiding all failures inside one final score.^{[3]Reference 3RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217} The dedicated RAG evaluation lesson will implement those metrics. Start with hard release assertions that catch expensive mistakes immediately.

The minimal suite already checks three high-impact failures: a forbidden chunk, a superseded chunk, and an unsupported answer. A serious deployment adds paraphrases, policy conflicts, index deletion cases, model-judge calibration, human reviews, and latency distributions.

What to block before launch

Ship the policy-answerer-v1 artifact

The production RAG service has enough structure for a small portfolio artifact:

1. Store three versions of an access-control policy with chunk_id, document_id, parent_id, effective dates, region, and ACL tags.
2. Add at least four frozen questions: a supported EU request, a US-only request, a restricted admin-only policy attack, and a question whose answer is absent.
3. Implement a retriever behind the retrieve() contract. Keep the simple overlap baseline first.
4. Pack evidence with stable source IDs and return a cited answer or a documented abstention.
5. Write one trace JSON row per request without logging restricted text.
6. Produce a release report listing authorization, freshness, evidence, grounding, and latency gates.

Mastery check

You're ready to design a production RAG pipeline when you can:

• Explain why a RAG answer must be treated as an evidence-backed operation instead of generated text alone.
• Define a versioned chunk record with stable citation identity, effective dates, region, and ACL metadata.
• Enforce authorization and policy freshness before any retrieved text reaches the model.
• Pack small, cited context and require an abstention when permitted evidence can't support the answer.
• Record a reproducible request trace without storing restricted text in unsafe logs.
• Separate caller-visible TTFT from model TTFT so latency regressions point to the right stage.
• Gate a candidate on authorization, freshness, grounding, abstention, and latency behavior.
• Preserve that contract while a later retrieval implementation replaces the simple search baseline.

Evaluation rubric

Common pitfalls