Chunking Strategies

A clean incident-runbook record gives retrieval a concrete starting point:

incident-runbook-v3.pdf, page 7: Rollback failure: page on-call within 15 minutes with deploy ID.

That sentence is trustworthy evidence only while its pieces stay together. Split it into one chunk containing "Rollback failure" and another containing "15 minutes with deploy ID," and a search result can retrieve the condition without the deadline or the deadline without its required identifier.

Chunking turns normalized records into searchable evidence units. In a retrieval-augmented generation (RAG) system, a retriever selects passages from an external index and the generator answers from those passages.^{[1]Reference 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401} Your chunk boundaries decide what a retrieved passage can prove.

What a useful chunk must preserve

Suppose an engineer asks, "A rollback failed after deploy pay-742. When do I page, and what ID do I include?"

The first engineering requirement isn't "make chunks small." It's "make each retrieved chunk a defensible piece of evidence."

Fixed windows show the boundary failure

The simplest splitter takes windows of tokens. If a window has size $N$ and overlap $O$, the next window starts after $N - O$ tokens. Overlap repeats boundary text, which can help continuity, but it can't guarantee that a complete policy rule survives.

The lab uses whitespace-separated words as visible token stand-ins. A production pipeline should measure length with the tokenizer used by its embedding model and respect that model's max_seq_length. Whitespace word counts are not embed tokens; overflow past the model limit truncates meaning silently at encode time.

The late notes archive within 14 days. window and final days. fragment expose a second failure: an orphan tail can lose the condition that gives a value meaning. A production fallback should merge an undersized tail into the previous window or send it for review instead of indexing it blindly.

Why shouldn't the trailing notes archive within 14 days. window enter the index by itself?

Answer

It no longer says that the archive window belongs to routine deploy notes. A retriever could surface the number without the runbook condition needed to cite or apply it correctly.

Now measure what overlap bought you. More repeated words increase indexed text, but the rule becomes useful only when a window carries all three required pieces.

This is the right posture for overlap: a configuration to evaluate, not a ritual to apply to every document.

Prefer policy structure when you have it

File ingestion preserved headings and source locations. Use them. A heading boundary is more meaningful than an arbitrary word count when the source already says which rule belongs together.

LangChain's RecursiveCharacterTextSplitter is a common generic-text baseline. Its documentation describes trying separators in order, with the default sequence ["\n\n", "\n", " ", ""], so paragraphs are preferred before smaller cuts.^{[2]Reference 2RecursiveCharacterTextSplitterhttps://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter} Start with explicit heading boundaries when the source provides them. Add the smaller-cut fallback only for sections that still exceed your size limit.

Each chunk's indexed_text includes its heading, so the searchable text keeps the condition attached to the deadline. The locator separately preserves the path back to original evidence.

A policy table needs an equally deliberate rule. Splitting a row away from its column names turns exact information into ambiguous fragments.

For a long section, split inside it while carrying its heading and original locator into every child chunk. Merge a tiny final window back into the preceding child so the fallback doesn't emit an orphan fragment.

Heading and locator on the child record are provenance, not a substitute for answerable text. The first child's text still severs deploy from ID, so it fails the earlier answerability check if you embed that field alone. Before indexing children:

1. Prepend the section heading (and any required condition) into the string you embed, as in the prepend-section-context lab below, and/or
2. Treat incomplete children as search candidates only when parent expansion is mandatory on every hit.

Never ship orphan spans that fail answerability alone and rely on metadata fields the retriever never sees.

Evaluate boundaries with labeled questions

A chunking strategy isn't good because it sounds fancy. It works when labeled questions retrieve complete supporting evidence.

Start with a tiny, transparent score before involving a vector database. For the query terms {rollback, failure, deploy, minutes}, the complete rollback-runbook chunk matches all four. A fragment that contains only {rollback, failure} may rank, but it can't answer the question.

This lexical scorer is deliberately simple. It isolates the effect of boundaries; replace its score with real embeddings after the fixture and expected evidence are stable.

Now compare a broken fixed-window configuration against a section-aware configuration using the same question and the same expected evidence phrase.

One question proves little. Ship a small labeled set containing policy exceptions, tables, and boundary failures, then measure each candidate splitter on exactly that set.

When you replace this lexical baseline with embeddings, the assertions stay useful: retrieve the correct source and retain text sufficient to answer.

Search small, answer with enough context

Some questions match a narrow sentence, while a faithful answer needs its surrounding section. A parent-child design indexes small children for search and stores a pointer to the larger source section returned for generation.^{[3]Reference 3ParentDocumentRetrieverhttps://reference.langchain.com/python/langchain-classic/retrievers/parent_document_retriever/ParentDocumentRetriever}

Child windows inside a longer section also benefit from their section label. Embedding a child with a contextual header is a cheap hypothesis to test against your labeled set, not a promise of improvement.

Escalate only when the baseline exposes a gap

Structure-aware chunks handle many handbooks and policy pages. Some corpora force different choices:

Semantic chunking places candidate boundaries where neighboring sentence representations change sharply. The next small lab uses transparent topic vectors, so you can see the boundary without trusting an external embedding service.

Late chunking is a different escalation. Günther et al. encode the longer text first and apply chunk pooling after the transformer's contextual token representations have been computed.^{[4]Reference 4Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models.https://arxiv.org/abs/2409.04701} That means a child representation can carry information from surrounding document text, but it requires a compatible long-context embedding stack and must earn its additional cost in evaluation.

Ship a chunking decision, not a guess

For the incident-runbook corpus, a credible first release is straightforward:

Your release gate can be encoded as an executable manifest check.

Mastery check

Clean evidence records from ingestion can now become retrieval units that can be evaluated.

Key concepts

• A chunk is a searchable evidence unit, not an arbitrary slice of text.
• Fixed windows expose boundary and overlap costs directly.
• Structure-aware splitting is a strong baseline when headings or tables carry meaning.
• Parent-child expansion separates precise search from sufficient answer context.
• Semantic or late chunking should be escalations justified by measured failures.
• A labeled retrieval set must check both correct selection and answerable evidence.

Evaluation rubric

• Foundational: Identifies why a broken chunk can't answer the rollback-failure question
• Foundational: Implements fixed windows and explains what overlap repeats
• Intermediate: Preserves headings, tables, source IDs, and locators in chunk records
• Intermediate: Compares candidate boundaries on labeled retrieval cases
• Intermediate: Uses parent-child expansion when narrow retrieval lacks sufficient context
• Advanced: Chooses semantic or late chunking only after measuring a baseline failure

Follow-up questions

Why is a chunk containing 15 minutes with deploy ID insufficient by itself?

Answer

It omits the condition that the deadline applies to rollback failures. Without that condition, the model can't cite the paging window as the answer to a failed-rollback question.

When should overlap be set to zero?

Answer

Start with zero when a chunk already follows a complete boundary, such as a policy section or intact table row group. Add overlap only when a measured boundary failure shows that nearby text must be repeated.

Why retain the locator from the ingestion record in every chunk?

Answer

The locator lets a retrieved answer cite original evidence, compare re-ingestion changes, and debug which boundary rule created a failed result.

What problem does parent-child retrieval solve?

Answer

It lets the search index use a small, specific match while the generator receives the larger section needed for a complete, cited answer.

What evidence justifies switching from structure-aware splitting to late chunking?

Answer

A labeled evaluation set must show failures caused by missing broader document context, and a late-chunking implementation must improve retrieval or answer quality enough to justify its latency and infrastructure cost.

Common pitfalls

• A deadline loses its condition: A window retrieves 15 minutes without Rollback failure. Fix: split at section boundaries and assert answerability.
• A pronoun loses its antecedent: A chunk says This process reduces wear without naming the process. Fix: keep the referred entity in the same unit, prepend section context, or expand to the parent when coreference would leave the span ambiguous.
• A fallback emits an orphan tail: A tiny final window retrieves 14 days. without Routine deploy notes. Fix: merge undersized tails into the preceding child or block them for review.
• A table row loses its header: A paging or archive window becomes ambiguous. Fix: keep column names with rows and test exact-value questions.
• Overlap creates duplicates without fixing answers: Repeated chunks dominate top results. Fix: measure overlap against both complete evidence and index cost.
• Word windows overflow the embedder: A chunk fits a word budget but exceeds max_seq_length, so the model silently drops the end of the rule. Fix: size and gate with the embedding tokenizer.
• Chunks can't be cited: Search looks plausible but support can't defend the answer. Fix: retain source, locator, heading, and checksum metadata.
• An advanced splitter is adopted by reputation: Complexity rises without better results. Fix: preserve a simple baseline and evaluate every escalation on the same labeled set.