Chunking Strategies

A clean incident-runbook record gives retrieval a concrete starting point:

incident-runbook-v3.pdf, page 7: Rollback failure: page on-call within 15 minutes with deploy ID.

That sentence is trustworthy evidence only while its pieces stay together. Split it into one chunk containing "Rollback failure" and another containing "15 minutes with deploy ID," and a search result can retrieve the condition without the deadline or the deadline without its required identifier.

Chunking turns normalized records into searchable evidence units. In a retrieval-augmented generation (RAG) system, a retriever selects passages from an external index and the generator answers from those passages.^{[1]Reference 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401} Your chunk boundaries decide what a retrieved passage can prove.

What a useful chunk must preserve

Suppose an engineer asks, "A rollback failed after deploy pay-742. When do I page, and what ID do I include?"

The first engineering requirement isn't "make chunks small." It's "make each retrieved chunk a defensible piece of evidence."

Fixed windows show the boundary failure

The simplest splitter takes windows of tokens. If a window has size $N$ and overlap $O$, the next window starts after $N - O$ tokens. Overlap repeats boundary text, which can help continuity, but it can't guarantee that a complete policy rule survives.

The lab uses whitespace-separated words as visible token stand-ins. A production pipeline should measure with the tokenizer used by its embedding model.

The late notes archive within 14 days. window and final days. fragment expose a second failure: an orphan tail can lose the condition that gives a value meaning. A production fallback should merge an undersized tail into the previous window or send it for review instead of indexing it blindly.

Now measure what overlap bought you. More repeated words increase indexed text, but the rule becomes useful only when a window carries all three required pieces.

This is the right posture for overlap: a configuration to evaluate, not a ritual to apply to every document.

Prefer policy structure when you have it

File ingestion preserved headings and source locations. Use them. A heading boundary is more meaningful than an arbitrary word count when the source already says which rule belongs together.

LangChain's RecursiveCharacterTextSplitter is a common generic-text baseline. Its documentation describes trying separators in order, with the default sequence ["\n\n", "\n", " ", ""], so paragraphs are preferred before smaller cuts.^{[2]Reference 2RecursiveCharacterTextSplitterhttps://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter} Start with explicit heading boundaries when the source provides them. Add the smaller-cut fallback only for sections that still exceed your size limit.

Each chunk's indexed_text includes its heading, so the searchable text keeps the condition attached to the deadline. The locator separately preserves the path back to original evidence.

A policy table needs an equally deliberate rule. Splitting a row away from its column names turns exact information into ambiguous fragments.

For a long section, split inside it while carrying its heading and original locator into every child chunk. Merge a tiny final window back into the preceding child so the fallback doesn't emit an orphan fragment.

Evaluate boundaries with labeled questions

A chunking strategy isn't good because it sounds fancy. It works when labeled questions retrieve complete supporting evidence.

Start with a tiny, transparent score before involving a vector database. For the query terms {rollback, failure, deploy, minutes}, the complete rollback-runbook chunk matches all four. A fragment that contains only {rollback, failure} may rank, but it can't answer the question.

This lexical scorer is deliberately simple. It isolates the effect of boundaries; replace its score with real embeddings after the fixture and expected evidence are stable.

Now compare a broken fixed-window configuration against a section-aware configuration using the same question and the same expected evidence phrase.

One question proves little. Ship a small labeled set containing policy exceptions, tables, and boundary failures, then measure each candidate splitter on exactly that set.

When you replace this lexical baseline with embeddings, the assertions stay useful: retrieve the correct source and retain text sufficient to answer.

Search small, answer with enough context

Some questions match a narrow sentence, while a faithful answer needs its surrounding section. A parent-child design indexes small children for search and stores a pointer to the larger source section returned for generation.^{[3]Reference 3ParentDocumentRetrieverhttps://reference.langchain.com/python/langchain-classic/retrievers/parent_document_retriever/ParentDocumentRetriever}

Child windows inside a longer section also benefit from their section label. Embedding a child with a contextual header is a cheap hypothesis to test against your labeled set, not a promise of improvement.

Escalate only when the baseline exposes a gap

Structure-aware chunks handle many handbooks and policy pages. Some corpora force different choices:

Semantic chunking places candidate boundaries where neighboring sentence representations change sharply. The next small lab uses transparent topic vectors, so you can see the boundary without trusting an external embedding service.

Late chunking is a different escalation. Günther et al. encode the longer text first and apply chunk pooling after the transformer's contextual token representations have been computed.^{[4]Reference 4Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models.https://arxiv.org/abs/2409.04701} That means a child representation can carry information from surrounding document text, but it requires a compatible long-context embedding stack and must earn its additional cost in evaluation.

Ship a chunking decision, not a guess

For the incident-runbook corpus, a credible first release is straightforward:

Your release gate can be encoded as an executable manifest check.

Mastery check

Clean evidence records from ingestion can now become retrieval units that can be evaluated.

Key concepts

• A chunk is a searchable evidence unit, not an arbitrary slice of text.
• Fixed windows expose boundary and overlap costs directly.
• Structure-aware splitting is a strong baseline when headings or tables carry meaning.
• Parent-child expansion separates precise search from sufficient answer context.
• Semantic or late chunking should be escalations justified by measured failures.
• A labeled retrieval set must check both correct selection and answerable evidence.

Evaluation rubric

• Foundational: Identifies why a broken chunk can't answer the rollback-failure question
• Foundational: Implements fixed windows and explains what overlap repeats
• Intermediate: Preserves headings, tables, source IDs, and locators in chunk records
• Intermediate: Compares candidate boundaries on labeled retrieval cases
• Intermediate: Uses parent-child expansion when narrow retrieval lacks sufficient context
• Advanced: Chooses semantic or late chunking only after measuring a baseline failure

Follow-up questions

Common pitfalls

• A deadline loses its condition: A window retrieves 15 minutes without Rollback failure. Fix: split at section boundaries and assert answerability.
• A fallback emits an orphan tail: A tiny final window retrieves 14 days. without Routine deploy notes. Fix: merge undersized tails into the preceding child or block them for review.
• A table row loses its header: A paging or archive window becomes ambiguous. Fix: keep column names with rows and test exact-value questions.
• Overlap creates duplicates without fixing answers: Repeated chunks dominate top results. Fix: measure overlap against both complete evidence and index cost.
• Chunks can't be cited: Search looks plausible but support can't defend the answer. Fix: retain source, locator, heading, and checksum metadata.
• An advanced splitter is adopted by reputation: Complexity rises without better results. Fix: preserve a simple baseline and evaluate every escalation on the same labeled set.