Chunking Strategies

In File Ingestion for AI, you already turned incident-runbook-v3.pdf into a located, checksummed record:

incident-runbook-v3.pdf, page 7: Rollback failure: page on-call within 15 minutes with deploy ID.

That sentence is evidence only while its pieces stay together. Split it so one chunk has "Rollback failure" and another has "15 minutes with deploy ID," and a search hit can retrieve the condition without the deadline, or the deadline without the identifier the on-call must include.

Chunking is the cut that turns those records into searchable evidence units. In a retrieval-augmented generation (RAG) system, a retriever selects passages from an external index and the generator answers from those passages.^{[1]Reference 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401} Your chunk boundaries decide what a retrieved passage can prove.

What a useful chunk must preserve

Before comparing splitters, define success in answer terms. Suppose an engineer asks, "A rollback failed after deploy pay-742. When do I page, and what ID do I include?"

The first engineering requirement isn't "make chunks small." It's "make each retrieved chunk a defensible piece of evidence."

A substring check is only a stand-in for a harder test. If this chunk is the only passage the generator sees, can it cite the 15-minute paging rule without guessing?

So what happens if you ignore the heading ingestion already preserved and cut the page by word count instead?

Fixed windows show the boundary failure

Start with a mechanical baseline: windows of tokens. If a window has size $N$ and overlap $O$, the next window starts after $N - O$ tokens. Overlap repeats boundary text, which can help continuity, but it can't guarantee that a complete policy rule survives.

The labs below use whitespace-separated words as visible token stand-ins. A production pipeline should measure length with the tokenizer used by its embedding model and respect that model's max_seq_length. Word counts aren't embed tokens. Overflow past the model limit truncates meaning silently at encode time.

Predict the first window before running the splitter. The runbook line has 17 words. With $N = 6$, that first window reads Rollback failure page on-call within 15. The deadline starts at word 7 (minutes), so this cut already loses the paging rule.

Window 3 (notes archive within 14 days.) exposes a second failure: an orphan tail can lose the condition that gives a value meaning. The final days. fragment makes that loss especially obvious. A production fallback should merge an undersized tail into the previous window or send it for review instead of indexing it blindly.

Why shouldn't the trailing notes archive within 14 days. window enter the index by itself?

Answer

It no longer says that the archive window belongs to routine deploy notes. A retriever could surface the number without the runbook condition needed to cite or apply it correctly.

Now measure what overlap bought you. Predict the outcome first: more repeated words should increase indexed text, but a rule becomes useful only when one window carries all three required pieces. Size 7 looks generous on a 17-word page. It still never holds the full rollback rule.

Treat overlap as a knob you measure, not a default you copy from a tutorial.

🔬 Research insight: On Natural Questions with a sparse retriever, Bennani and Moslonka found that 10-20% overlap didn't move BERTScore or exact match, while chunk count grew by about $1/(1-r)$.^{[2]Reference 2A Systematic Analysis of Chunking Strategies for Reliable Question Answering.https://arxiv.org/abs/2601.14123} That's Wikipedia plus SPLADE, not your headed runbook. The lab above is the matching local test: more overlap, same missing rule.

Ingestion already kept headings and source locations. If word windows can't reconstruct the rule, those headings give us a better boundary to test.

Prefer policy structure when you have it

Overlap made more copies but never restored the rule. Use the source's own structure when it tells you which text belongs together.

LangChain's RecursiveCharacterTextSplitter is a common generic-text baseline. It tries separators in order, defaulting to ["\n\n", "\n", " ", ""], so paragraphs stay together before sentences, then words.^{[3]Reference 3RecursiveCharacterTextSplitterhttps://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter}

That splitter measures size with length_function, which defaults to Python len (characters), not embedding tokens. Start with explicit heading boundaries when the source provides them. Use the smaller-cut fallback only for sections that still exceed your size limit, and measure that limit with the embedding tokenizer.

Each chunk's indexed_text includes its heading, so searchable text keeps the condition attached to the deadline. The locator separately preserves the path back to original evidence.

Tables create the next boundary test. Splitting a row away from its column names turns exact information into ambiguous fragments.

For a long section, split inside it while carrying its heading and original locator into every child chunk. Merge a tiny final window back into the preceding child so the fallback doesn't emit an orphan fragment.

The heading and locator on a child record are provenance, not a substitute for answerable text. The first child's text still severs deploy from ID, so it fails the earlier answerability check if you embed that field alone. Before indexing children:

1. Prepend the section heading (and any required condition) into the string you embed, as in the prepend-section-context lab below, and/or
2. Treat incomplete children as search candidates only when parent expansion is mandatory on every hit.

Never ship orphan spans that fail answerability alone and rely on metadata fields the retriever never sees.

The headed chunks look right on the page. That still doesn't prove a retriever will select them or that selected text can answer.

Evaluate boundaries with labeled questions

Once a chunk looks coherent, test whether retrieval can find it. A chunking strategy isn't good because it sounds fancy. It works when labeled questions retrieve complete supporting evidence.

Start with a tiny, transparent score before involving a vector database. For the query terms {rollback, failure, deploy, minutes}, the complete rollback-runbook chunk matches all four. A fragment that contains only {rollback, failure} may rank, but it can't answer the question.

This lexical scorer is deliberately simple. It isolates the effect of boundaries. Replace its score with real embeddings after the fixture and expected evidence are stable.

Now compare a broken fixed-window configuration against a section-aware configuration using the same question and the same expected evidence phrase.

One question proves little. Ship a small labeled set containing policy exceptions, tables, and boundary failures, then measure each candidate splitter on exactly that set.

When you replace this lexical baseline with embeddings, the assertions stay useful: retrieve the correct source and retain text sufficient to answer.

Sometimes the right search unit is smaller than the unit a faithful answer needs. That's a different failure from a broken boundary, and it needs a different design.

Search small, answer with enough context

Some questions match a narrow sentence, while a faithful answer needs its surrounding section. A parent-child design indexes small children for search and stores a pointer to the larger source section returned for generation.^{[4]Reference 4ParentDocumentRetrieverhttps://reference.langchain.com/python/langchain-classic/retrievers/parent_document_retriever/ParentDocumentRetriever}

The parent can be a section or the whole document. For this runbook, expand to the section, not the entire handbook.

Parent expansion can over-pack. In the same Natural Questions study, generation quality dropped once the filled context went past about 2.5k tokens on that QA setup.^{[2]Reference 2A Systematic Analysis of Chunking Strategies for Reliable Question Answering.https://arxiv.org/abs/2601.14123} Keep a generation budget in the regression suite, and don't expand a child hit into the whole PDF.

Child windows inside a longer section also benefit from their section label. Embedding a child with a contextual header is a cheap hypothesis to test against your labeled set, not a promise of improvement.

Some leftover failures aren't a size problem. The paragraph switches topic, or a local span is unreadable without a name that appeared pages earlier. Those are the cases that justify a more expensive splitter, and only after the baseline has been measured.

Escalate only when the baseline exposes a gap

Structure-aware chunks handle many handbooks and policy pages. Some corpora force different choices:

On unstructured Wikipedia pages, sentence-preserving splits matched semantic chunking up to roughly 5k generation tokens.^{[2]Reference 2A Systematic Analysis of Chunking Strategies for Reliable Question Answering.https://arxiv.org/abs/2601.14123} A headed runbook is a different document. Use the headings first, and treat semantic splits as an experiment for leftover topic shifts.

Semantic chunking proposes boundaries where neighboring sentence representations change sharply. The next lab uses transparent topic vectors, so you can see the boundary without trusting an external embedding service.

Count whole words, not substrings. Otherwise id matches incident, and the cosine stops meaning what you think.

The first two sentences share the rollback/deploy topic axis, so cosine stays 1.00. The commander sentence shares none of those words, so cosine drops to 0.00 and the lab marks a boundary. You'd still cap chunk size and re-score the labeled queries. A prettier boundary that empties the answer out of both sides is still a failed split.

Late chunking is a different escalation. Günther et al. still run a chunker to get boundary cues. They encode the longer text through the transformer first, then mean-pool the token embeddings that fall inside each span.^{[5]Reference 5Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models.https://arxiv.org/abs/2409.04701}

The pooled ranges stay the same. What changes is the context inside each child vector: a later sentence that only says "the city" can still sit near "Berlin" in embedding space because the full document was encoded first.

You need a long-context embedder that exposes those token states, typically through mean pooling. If the document is longer than the embedder window, use the paper's long-document variant rather than silently truncating. Keep late chunking only when it improves the labeled set enough to pay for its extra compute.

Ship a chunking decision, not a guess

For this runbook, a first release can look like this:

Your release gate can be encoded as an executable manifest check.

At this point, a policy page can be split into answerable chunks, retain ingestion locators, and block fragments at the release gate. The remaining question isn't another splitter. Does a model use that retrieved evidence, and can a public benchmark stand in for that proof?

Mastery check

A clean evidence record is only the starting point. The checks above turn it into a retrieval unit that can be evaluated.

Key concepts

• A chunk is a searchable evidence unit, not an arbitrary slice of text.
• Fixed windows expose boundary and overlap costs directly.
• Structure-aware splitting is a strong baseline when headings or tables carry meaning.
• Parent-child expansion separates precise search from sufficient answer context.
• Semantic or late chunking should be escalations justified by measured failures.
• A labeled retrieval set must check both correct selection and answerable evidence.

Evaluation rubric

• Foundational: Identifies why a broken chunk can't answer the rollback-failure question
• Foundational: Implements fixed windows and explains what overlap repeats
• Intermediate: Preserves headings, tables, source IDs, and locators in chunk records
• Intermediate: Compares candidate boundaries on labeled retrieval cases
• Intermediate: Uses parent-child expansion when narrow retrieval lacks sufficient context
• Advanced: Chooses semantic or late chunking only after measuring a baseline failure

Follow-up questions

Why is a chunk containing 15 minutes with deploy ID insufficient by itself?

Answer

It omits the condition that the deadline applies to rollback failures. Without that condition, the model can't cite the paging window as the answer to a failed-rollback question.

When should overlap be set to zero?

Answer

Start with zero when a chunk already follows a complete boundary, such as a policy section or intact table row group. Add overlap only when a measured boundary failure shows that nearby text must be repeated.

Why retain the locator from the ingestion record in every chunk?

Answer

The locator lets a retrieved answer cite original evidence, compare re-ingestion changes, and debug which boundary rule created a failed result.

What problem does parent-child retrieval solve?

Answer

It lets the search index use a small, specific match while the generator receives the larger section needed for a complete, cited answer.

What evidence justifies switching from structure-aware splitting to late chunking?

Answer

A labeled evaluation set must show failures caused by missing broader document context, and a late-chunking implementation must improve retrieval or answer quality enough to justify its latency and infrastructure cost.

Common pitfalls

• A deadline loses its condition: A window retrieves 15 minutes without Rollback failure. Fix: split at section boundaries and assert answerability.
• A pronoun loses its antecedent: A chunk says This process reduces wear without naming the process. Fix: keep the referred entity in the same unit, prepend section context, or expand to the parent when coreference would leave the span ambiguous.
• A fallback emits an orphan tail: A tiny final window retrieves 14 days. without Routine deploy notes. Fix: merge undersized tails into the preceding child or block them for review.
• A table row loses its header: A paging or archive window becomes ambiguous. Fix: keep column names with rows and test exact-value questions.
• Overlap creates duplicates without fixing answers: Repeated chunks dominate top results. Fix: measure overlap against both complete evidence and index cost.
• Word windows overflow the embedder: A chunk fits a word budget but exceeds max_seq_length, so the model silently drops the end of the rule. Fix: size and gate with the embedding tokenizer.
• Chunks can't be cited: Search looks plausible but support can't defend the answer. Fix: retain source, locator, heading, and checksum metadata.
• An advanced splitter is adopted by reputation: Complexity rises without better results. Fix: preserve a simple baseline and evaluate every escalation on the same labeled set.