Chunking Strategies

In File Ingestion for AI, you already turned incident-runbook-v3.pdf into a located, checksummed record:

incident-runbook-v3.pdf, page 7: Rollback failure: page on-call within 15 minutes with deploy ID.

To answer from that sentence, the generator needs its condition, deadline, and required ID together in the returned context. Split it so one chunk has "Rollback failure" and another has "15 minutes with deploy ID," and a single search hit can retrieve only part of the rule.

Chunking is the cut that turns those records into searchable evidence units. In a retrieval-augmented generation (RAG) system, a retriever selects passages from an external index and the generator answers from those passages.^{[1]Reference 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401} Your chunk boundaries decide what a retrieved passage can prove.

What a useful chunk must preserve

Before comparing splitters, define success in answer terms. Suppose an engineer asks, "A rollback failed after deploy pay-742. When do I page, and what ID do I include?"

For this question, a complete retrieval unit must connect the rollback condition to the deadline and required identifier.

The final context can contain one complete chunk or several complementary passages. Smaller fragments can be useful search candidates when retrieval reliably supplies their missing context through parent expansion or another explicit assembly step. Check the combined, permitted source passages under the generation budget; neither a parent pointer nor a hope that another hit will appear establishes sufficiency. The first lab deliberately evaluates one chunk at a time.

from dataclasses import dataclass

@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    text: str
    source_id: str
    locator: str

def is_answerable(chunk: Chunk) -> bool:
    required = ["rollback failure", "15 minutes", "deploy id"]
    lowered = chunk.text.lower()
    return all(phrase in lowered for phrase in required)

chunks = [
    Chunk("broken-condition", "Rollback failure: page on-call within", "incident-runbook-v3.pdf", "page=7"),
    Chunk("broken-window", "15 minutes with deploy ID. Routine deploy notes archive within 14 days.", "incident-runbook-v3.pdf", "page=7"),
    Chunk("complete", "Rollback failure: page on-call within 15 minutes with deploy ID.", "incident-runbook-v3.pdf", "page=7"),
]

results = [is_answerable(chunk) for chunk in chunks]
for chunk, ok in zip(chunks, results):
    print(f"{chunk.chunk_id}: answerable={ok}")
assert results == [False, False, True]

broken-condition: answerable=False
broken-window: answerable=False
complete: answerable=True

A substring check is only a fixture test, not a general answerability detector. Rollback failure: do not page within 15 minutes with deploy ID passes it while reversing the instruction. Review negation, exceptions, and which condition governs each value. If this chunk is the only passage the generator sees, can it cite the paging rule without guessing?

So what happens if you ignore the heading ingestion already preserved and cut the page by word count instead?

Fixed windows show the boundary failure

Start with a mechanical baseline: windows of tokens. If a window has size $N$ and overlap $O$, the next window starts after $N - O$ tokens. Overlap repeats boundary text, which can help continuity, but it can't guarantee that a complete policy rule survives.

The labs below use whitespace-separated words as visible token stand-ins. A production pipeline should measure the final input with the tokenizer used by its embedding model, including added headings, prompts, and special tokens. Word counts aren't embedding-token counts. Oversized inputs may be rejected or truncated, depending on the model and API. Check before encoding; don't rely on either behavior to preserve a rule.

Predict the first window before running the splitter. The two-rule fixture has 17 words. With $N = 6$, that first window reads Rollback failure page on-call within 15. The cut falls between the number 15 and its unit minutes, already breaking the paging rule.

def fixed_windows(text: str, size: int, overlap: int) -> list[str]:
    if size <= 0 or overlap < 0 or overlap >= size:
        raise ValueError("require size > overlap >= 0")
    words = text.split()
    step = size - overlap
    chunks = []
    for start in range(0, len(words), step):
        end = min(start + size, len(words))
        chunks.append(" ".join(words[start:end]))
        if end == len(words):
            break  # Don't emit a tail already covered by this window.
    return chunks

policy = (
    "Rollback failure page on-call within 15 minutes with deploy ID. "
    "Routine deploy notes archive within 14 days."
)

for index, chunk in enumerate(fixed_windows(policy, size=6, overlap=2)):
    print(f"{index}: {chunk}")

0: Rollback failure page on-call within 15
1: within 15 minutes with deploy ID.
2: deploy ID. Routine deploy notes archive
3: notes archive within 14 days.

Window 3 (notes archive within 14 days.) exposes a second failure: a tail can lose the condition that gives a value meaning. The loop stops once a window reaches the end, so it doesn't emit an extra, fully duplicated days. chunk. Removing that redundancy still doesn't make the tail sufficient evidence. Rebalance the boundary, expand to its parent before generation, or send it for review. Merging without checking length can break the size cap.

Now measure what overlap bought you. The rollback rule is ten words long; no seven-word window can contain it, however far adjacent windows overlap. Repetition may help shorter facts cross a boundary, but it can't enlarge a window.

def fixed_windows(text: str, size: int, overlap: int) -> list[str]:
    if size <= 0 or overlap < 0 or overlap >= size:
        raise ValueError("require size > overlap >= 0")
    words = text.split()
    step = size - overlap
    chunks = []
    for start in range(0, len(words), step):
        end = min(start + size, len(words))
        chunks.append(" ".join(words[start:end]))
        if end == len(words):
            break
    return chunks

def carries_rollback_rule(text: str) -> bool:
    lowered = text.lower()
    return all(term in lowered for term in ["rollback failure", "15 minutes", "deploy id"])

policy = "Rollback failure page on-call within 15 minutes with deploy ID. Routine deploy notes archive within 14 days."
for overlap in [0, 2, 4]:
    chunks = fixed_windows(policy, size=7, overlap=overlap)
    complete = sum(carries_rollback_rule(chunk) for chunk in chunks)
    indexed_words = sum(len(chunk.split()) for chunk in chunks)
    print(f"overlap={overlap}: chunks={len(chunks)} indexed_words={indexed_words} complete={complete}")
    assert complete == 0

overlap=0: chunks=3 indexed_words=17 complete=0
overlap=2: chunks=3 indexed_words=21 complete=0
overlap=4: chunks=5 indexed_words=33 complete=0

For a long document, ignoring end effects, overlap fraction $r = O/N$ multiplies window count and processed text by approximately $1/(1-r)$. A 20% overlap therefore costs about $1.25$ times as many windows. The short fixture has large end effects, so use its exact counts rather than that approximation.

🔬 Research insight: Bennani and Moslonka report no measurable answer-quality gain from overlap in their Natural Questions experiments using SPLADE retrieval and Ministral-8B generation.^{[2]Reference 2A Systematic Analysis of Chunking Strategies for Reliable Question Answering.https://arxiv.org/abs/2601.14123} That Wikipedia setup doesn't establish the best overlap for a headed runbook. Compare complete evidence and duplication on your own labeled questions.

Ingestion already kept headings and source locations. If word windows can't reconstruct the rule, those headings give us a better boundary to test.

Prefer policy structure when you have it

Overlap made more copies but never restored the rule. Use the source's own structure when it tells you which text belongs together.

LangChain's RecursiveCharacterTextSplitter is a generic-text baseline. It tries separators in order, defaulting to ["\n\n", "\n", " ", ""]: paragraph breaks, line breaks, spaces, then individual characters.^{[3]Reference 3RecursiveCharacterTextSplitterhttps://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter} There's no sentence-boundary detector in that default list. Languages without space-separated words need different separators or a language-aware splitter.

That splitter measures size with length_function, which defaults to Python len (characters), not embedding tokens. Start with explicit heading boundaries when the source provides them. Use the smaller-cut fallback only for sections that still exceed your size limit, and measure that limit with the embedding tokenizer.

from dataclasses import dataclass

@dataclass(frozen=True)
class PolicyChunk:
    heading: str
    body: str
    indexed_text: str
    locator: str

def headed_chunks(markdown: str, source_locator: str) -> list[PolicyChunk]:
    chunks: list[PolicyChunk] = []
    heading = "Document"
    body: list[str] = []

    def flush() -> None:
        if any(line.strip() for line in body):
            body_text = "\n".join(body).strip()
            chunks.append(
                PolicyChunk(
                    heading,
                    body_text,
                    f"{heading}\n{body_text}",
                    f"{source_locator};section={len(chunks)}",
                )
            )
        body.clear()

    for line in markdown.strip().splitlines():
        if line.startswith("## "):
            flush()
            heading = line[3:]
        else:
            body.append(line)
    flush()
    return chunks

policy = """## Rollback failure
Page on-call within 15 minutes with deploy ID.
## Routine deploy notes
Archive within 14 days after release."""

for chunk in headed_chunks(policy, "page=7"):
    print(chunk.indexed_text.replace("\n", " | "), "|", chunk.locator)

Rollback failure | Page on-call within 15 minutes with deploy ID. | page=7;section=0
Routine deploy notes | Archive within 14 days after release. | page=7;section=1

Each chunk's indexed_text includes its heading, so searchable text keeps the condition attached to the deadline. This small parser handles flat ## sections only, not arbitrary Markdown with nested headings or fenced code. Section ordinals distinguish repeated heading names within this version; they're application metadata, not native PDF anchors. In production, retain the ingestion record's raw-to-normalized offset map and page coordinates rather than rebuilding provenance from heading text.

Tables create the next boundary test. Splitting a row away from its column names turns exact information into ambiguous fragments.

table = [
    "| Condition | Window | Required evidence |",
    "| --- | --- | --- |",
    "| Rollback failure | 15 minutes | Deploy ID |",
    "| Routine deploy notes | 14 days | Release summary |",
]

def table_chunk(lines: list[str], heading: str) -> dict[str, str]:
    if len(lines) < 3:
        raise ValueError("need column names, separator, and at least one row")
    return {
        "heading": heading,
        "text": heading + "\n" + "\n".join(lines),
        "quality_check": "header_present" if lines[0].startswith("| Condition |") else "review",
    }

chunk = table_chunk(table, "Runbook windows")
print(chunk["heading"], chunk["quality_check"], f"rows={len(table) - 2}")
print("15 minutes" in chunk["text"] and "Deploy ID" in chunk["text"])
assert chunk["quality_check"] == "header_present"

Runbook windows header_present rows=2
True

This check recognizes our fixture's header; it isn't a Markdown table validator. If a table exceeds the budget, repeat its column names with each row group. Keep row spans and the table locator so a value can be traced to its original cell.

For a long section, a hard size cap must remain hard. Fifteen words cut into windows of seven give lengths 7, 7, 1. Merging the last two produces 7, 8, which breaks the cap. The fallback below balances those fifteen words across three children of five words each. It preserves exact substrings and half-open character offsets into the normalized section, but it doesn't promise semantic boundaries.

import math
import re

def section_windows(
    text: str, heading: str, source: str, size: int, min_tail_words: int
) -> list[dict]:
    if not 0 < min_tail_words <= size:
        raise ValueError("require 0 < min_tail_words <= size")
    words = list(re.finditer(r"\S+", text))
    if not words:
        return []
    count = math.ceil(len(words) / size)
    base, extra = divmod(len(words), count)
    children = []
    cursor = 0
    for index in range(count):
        length = base + int(index < extra)
        start = words[cursor].start()
        end = words[cursor + length - 1].end()
        children.append({
            "text": text[start:end],
            "heading": heading,
            "source": source,
            "span": (start, end),
            "needs_review": length < min_tail_words,
            "chunk_id": f"{source}:chars={start}-{end}",
        })
        cursor += length
    return children

children = section_windows(
    "Page on-call within 15 minutes with deploy ID. Include the service name and rollback attempt.",
    heading="Rollback failure",
    source="incident-runbook-v3.pdf:page=7;section=0",
    size=7,
    min_tail_words=4,
)

for child in children:
    print(child["chunk_id"], "|", child["heading"], "|", child["text"])

incident-runbook-v3.pdf:page=7;section=0:chars=0-30 | Rollback failure | Page on-call within 15 minutes
incident-runbook-v3.pdf:page=7;section=0:chars=31-58 | Rollback failure | with deploy ID. Include the
incident-runbook-v3.pdf:page=7;section=0:chars=59-93 | Rollback failure | service name and rollback attempt.

The heading and locator on a child record are provenance, not a substitute for answerable text. The first child now contains Page on-call within 15 minutes, but not deploy ID. Balancing repaired length, not evidence completeness. If the section is too short to meet min_tail_words, it stays under the hard cap and is flagged for review. The cap here counts body words only; the final embedding input needs a separate token-budget check. Before indexing children:

1. Prepend the section heading (and any required condition) into the string you embed, as in the prepend-section-context lab below, and/or
2. Treat incomplete children as search candidates only when parent expansion is mandatory on every hit.

Don't return an incomplete child as sufficient evidence merely because a heading or parent pointer exists elsewhere in storage.

The headed chunks look right on the page. That still doesn't prove a retriever will select them or that selected text can answer.

Evaluate boundaries with labeled questions

Once a chunk looks coherent, test whether retrieval can find it for a labeled question. A complete rule that's never selected won't help the answer.

Start with a tiny, transparent score before involving a vector database. For the query terms {rollback, failure, deploy, minutes}, the complete rollback-runbook chunk matches all four. A fragment that contains only {rollback, failure} may rank, but it can't answer the question.

This lexical scorer is deliberately simple. It isolates the effect of boundaries. Replace its score with real embeddings after the fixture and expected evidence are stable.

import re

def terms(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def retrieve(query: str, chunks: list[dict[str, str]]) -> dict[str, str]:
    query_terms = terms(query)
    return max(chunks, key=lambda chunk: len(query_terms & terms(chunk["text"])))

query = "rollback failure deploy minutes"
chunks = [
    {"id": "rollback-rule", "text": "Rollback failure: page on-call within 15 minutes with deploy ID."},
    {"id": "routine-rule", "text": "Routine deploy notes: archive within 14 days."},
]

hit = retrieve(query, chunks)
print(hit["id"], hit["text"])
assert hit["id"] == "rollback-rule"

rollback-rule Rollback failure: page on-call within 15 minutes with deploy ID.

Now compare two hand-cut fixtures using the same question and expected evidence phrase. This isolates one boundary failure; it isn't a benchmark of a library splitter. Both broken fragments tie on this scorer, which selects the first. Neither contains the complete rule, so the conclusion doesn't depend on that tie.

import re

def terms(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def top_chunk(query: str, chunks: list[str]) -> str:
    query_terms = terms(query)
    return max(chunks, key=lambda text: len(query_terms & terms(text)))

query = "rollback failure deploy id"
expected = "Rollback failure: page on-call within 15 minutes with deploy ID."
configs = {
    "broken-fixed": [
        "Rollback failure: page on-call within",
        "15 minutes with deploy ID. Routine deploy notes: archive within 14 days.",
    ],
    "section-aware": [
        "Rollback failure: page on-call within 15 minutes with deploy ID.",
        "Routine deploy notes: archive within 14 days.",
    ],
}

for name, chunks in configs.items():
    hit = top_chunk(query, chunks)
    print(f"{name}: complete={expected in hit}")

broken-fixed: complete=False
section-aware: complete=True

One question proves little. This three-case smoke test checks the paging rule, archive deadline, and commander instruction. Extend it with policy exceptions, tables, and boundary failures before comparing real splitters.

import re

def terms(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def retrieve(query: str, chunks: list[dict[str, str]]) -> dict[str, str]:
    query_terms = terms(query)
    return max(chunks, key=lambda chunk: len(query_terms & terms(chunk["text"])))

chunks = [
    {"id": "rollback", "text": "Rollback failure: page on-call within 15 minutes with deploy ID."},
    {"id": "routine", "text": "Routine deploy notes: archive within 14 days."},
    {"id": "commander", "text": "Commander handoff requires the active incident channel."},
]
cases = [
    ("rollback failure deploy id", "rollback", "15 minutes"),
    ("routine deploy archive", "routine", "14 days"),
    ("commander incident channel", "commander", "active incident"),
]

passed = 0
for query, expected_id, evidence in cases:
    hit = retrieve(query, chunks)
    ok = hit["id"] == expected_id and evidence in hit["text"]
    passed += int(ok)
    print(query, "PASS" if ok else "FAIL")
print(f"summary={passed}/{len(cases)}")
assert passed == len(cases)

rollback failure deploy id PASS
routine deploy archive PASS
commander incident channel PASS
summary=3/3

When you replace this lexical baseline with embeddings, retain the two checks: correct evidence source and sufficient returned text. These fixed fixtures can assert chunk IDs. Across splitter configurations, label stable source versions and answer-bearing spans instead, since chunk IDs and boundaries change. Score the final context after any parent expansion, under the same generation-token budget. Equal top-k isn't equal context when chunks have different lengths. Also measure duplicate source spans, retrieval latency, and index size.

Sometimes the right search unit is smaller than the unit a faithful answer needs. That's a different failure from a broken boundary, and it needs a different design.

Search small, answer with enough context

Some questions match a narrow sentence, while a faithful answer needs its surrounding section. A parent-child design indexes small children for search and stores a pointer to the larger source section returned for generation.^{[4]Reference 4ParentDocumentRetrieverhttps://reference.langchain.com/python/langchain-classic/retrievers/parent_document_retriever/ParentDocumentRetriever}

The parent can be a section or the whole document. For this runbook, expand to the section, not the entire handbook.

parents = {
    "rollback": {
        "text": "Rollback failure: page on-call within 15 minutes with deploy ID. Include the service name.",
        "source": "incident-runbook-v3.pdf", "locator": "page=7", "version": 3,
    },
}
children = [
    {"text": "15 minutes with deploy ID", "parent_id": "rollback", "version": 3},
    {"text": "Include the service name", "parent_id": "rollback", "version": 3},
]

def expand(matches: list[dict], allowed_parent_ids: set[str], word_budget: int) -> list[dict]:
    selected = {}
    for match in matches:
        parent_id = match["parent_id"]
        if parent_id not in allowed_parent_ids:
            raise PermissionError("parent access denied")
        parent = parents[parent_id]
        if match["version"] != parent["version"]:
            raise ValueError("stale child-to-parent link")
        selected[parent_id] = parent  # Two child hits need one parent copy.
    if sum(len(parent["text"].split()) for parent in selected.values()) > word_budget:
        raise ValueError("context budget exceeded; select smaller complete evidence")
    return list(selected.values())

context = expand(children, allowed_parent_ids={"rollback"}, word_budget=20)
print(f"child_hits={len(children)} parent_copies={len(context)}")
print(context[0]["text"])
assert "Rollback failure" in context[0]["text"]

failure_cases = [
    ("stale", [{**children[0], "version": 2}], {"rollback"}, 20,
     ValueError, "stale child-to-parent link"),
    ("denied", children, set(), 20, PermissionError, "parent access denied"),
    ("too small", children, {"rollback"}, 10,
     ValueError, "context budget exceeded; select smaller complete evidence"),
]
for name, matches, allowed, budget, expected_error, expected_message in failure_cases:
    try:
        expand(matches, allowed, budget)
    except expected_error as error:
        assert str(error) == expected_message
        print(f"{name}: {error}")
    else:
        raise AssertionError(f"{name} returned context instead of rejecting")

child_hits=2 parent_copies=1
Rollback failure: page on-call within 15 minutes with deploy ID. Include the service name.
stale: stale child-to-parent link
denied: parent access denied
too small: context budget exceeded; select smaller complete evidence

The allowed IDs stand in for a serving-time authorization decision, not a client-supplied list. Production links need immutable document and chunking versions, and access checks must cover the expanded parent, not just the child. This lab deduplicates parents and rejects oversized context using words for visibility. Use the generator's tokenizer for the real context budget, reserving space for instructions, question, and output. Never silently truncate the deadline to make an expansion fit. The three rejection traces exercise separate invariants: the child points to the current parent, that parent is permitted, and its complete text fits. More child hits don't satisfy any of those checks.

Child windows inside a longer section also benefit from their section label. Embedding a child with a contextual header is a cheap hypothesis to test against your labeled set, not a promise of improvement.

def indexed_text(heading: str, text: str, source: str) -> str:
    return f"Source: {source}\nSection: {heading}\n{text}"

child = indexed_text(
    heading="Rollback failure",
    text="Page on-call within 15 minutes with deploy ID.",
    source="Incident Runbook",
)
print(child)
assert "Section: Rollback failure" in child

Source: Incident Runbook
Section: Rollback failure
Page on-call within 15 minutes with deploy ID.

Measure the string printed above, not just the body. The heading and source label consume budget too. Keep this retrieval representation separate from the original evidence span so an added label doesn't become a fabricated source quotation.

Some leftover failures aren't a size problem. A paragraph switches topic, or a local span is unclear without a name that appeared pages earlier. Those are candidates for a more expensive splitter after measuring the baseline.

Semantic boundaries and late chunking

Structure-aware chunks handle many handbooks and policy pages. Some corpora force different choices:

Bennani and Moslonka report that sentence-preserving splits matched semantic chunking up to roughly 5k retrieved-context tokens in their Wikipedia/Natural Questions setup.^{[2]Reference 2A Systematic Analysis of Chunking Strategies for Reliable Question Answering.https://arxiv.org/abs/2601.14123} That is a context budget, not a 5k-token chunk size. A headed runbook is a different document. Use the headings first, and treat semantic splits as an experiment for leftover topic shifts.

Semantic chunking proposes boundaries where neighboring sentence representations change sharply. The next lab uses transparent topic vectors, so you can see the boundary without trusting an external embedding service.

Count whole words, not substrings. Otherwise id matches incident, and the cosine stops meaning what you think.

import math
import re

def vector(sentence: str) -> list[float]:
    words = set(re.findall(r"[a-z0-9]+", sentence.lower()))
    return [
        float(sum(word in words for word in ["rollback", "deploy", "minutes", "id"])),
        float(sum(word in words for word in ["commander", "handoff", "channel"])),
    ]

def cosine(left: list[float], right: list[float]) -> float | None:
    if len(left) != len(right):
        raise ValueError("vector dimensions differ")
    dot = sum(a * b for a, b in zip(left, right))
    left_norm = math.sqrt(sum(a * a for a in left))
    right_norm = math.sqrt(sum(b * b for b in right))
    return dot / (left_norm * right_norm) if left_norm and right_norm else None

sentences = [
    "Rollback failures require deploy IDs.",
    "Page on-call within 15 minutes.",
    "Commander handoff uses the active incident channel.",
]

for left, right in zip(sentences, sentences[1:]):
    similarity = cosine(vector(left), vector(right))
    if similarity is None:
        print("unknown: review boundary")
    else:
        print(f"{similarity:.2f}", "boundary" if similarity < 0.50 else "keep together")

1.00 keep together
0.00 boundary

The first two sentences activate only the hand-written rollback/deploy axis, so cosine is 1.00. The commander sentence activates the other axis, giving cosine 0.00. This illustrates a threshold, not the quality of learned embeddings. An unfamiliar sentence can map to [0, 0]; its cosine is undefined, not proof of a topic change. With real embeddings, tune the threshold on labeled cases and inspect boundaries that detach exceptions or antecedents.

Late chunking is a different escalation. Günther et al. still run a chunker to get boundary cues. They encode the longer text through the transformer first, then mean-pool the token embeddings that fall inside each span.^{[5]Reference 5Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models.https://arxiv.org/abs/2409.04701}

The pooled ranges can stay the same while each child vector gains surrounding context. In the paper's example, the representation of "the city" gains information from the earlier name "Berlin." That doesn't insert Berlin into the returned chunk text. A generator receiving only the ambiguous sentence still needs the antecedent through explicit context or parent expansion.

The mean-pooling method needs an embedder that exposes contextual token states. Map source spans to tokenizer offsets and pool the corresponding content tokens, handling special tokens and model-specific normalization deliberately. Whole-document context is available only when the input fits the encoder and its attention pattern allows it. The paper's long-document variant uses overlapping larger windows, so context is still bounded.^{[5]Reference 5Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models.https://arxiv.org/abs/2409.04701} Compare its retrieval quality, encoding cost, and returned evidence against the baseline.

Averaging independently encoded chunk vectors isn't late chunking: those vectors were already formed without the other chunks' context. Late chunking pools token states after the shared encoder pass. It also differs from ColPali/ColBERT late interaction in the ingestion lesson: this mean-pooling method stores one vector per chunk, while late interaction retains multiple vectors for token-level matching.^{[5]Reference 5Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models.https://arxiv.org/abs/2409.04701}

The shared input introduces an access boundary too. By construction, a public span encoded alongside restricted text can have a vector influenced by that restricted context. Filtering only the returned chunk text doesn't undo that dependence. Keep encoder contexts within the appropriate permission and trust scope, and retain the source/version of the context used to produce each vector. This follows from the encoding mechanism; it isn't a claim that the paper measured a particular data-extraction attack.

A release manifest for the runbook

For this runbook, a first release can look like this:

The manifest check below enforces required metadata and a prior review decision. It doesn't calculate answerability: quality="ready" must come from the evidence checks, not from the splitter declaring itself successful.

from hashlib import sha256

chunks = [
    {
        "id": "rollback",
        "text": "Rollback failure: page on-call within 15 minutes with deploy ID.",
        "source": "incident-runbook-v3.pdf",
        "locator": "page=7;section=0",
        "heading": "Rollback failure",
        "quality": "ready",
    },
    {
        "id": "broken",
        "text": "15 minutes with deploy ID.",
        "source": "incident-runbook-v3.pdf",
        "locator": "page=7;section=1",
        "heading": "Fragment",
        "quality": "review",
    },
]

for chunk in chunks:
    chunk["chunk_checksum"] = sha256(chunk["text"].encode()).hexdigest()

required_metadata = ("source", "locator", "heading", "chunk_checksum")
indexable = [
    chunk for chunk in chunks
    if chunk.get("quality") == "ready"
    and bool(chunk.get("text", "").strip())
    and all(chunk.get(field) for field in required_metadata)
]
print(f"indexable={[chunk['id'] for chunk in indexable]}")
print(f"blocked={len(chunks) - len(indexable)}")
assert [chunk["id"] for chunk in indexable] == ["rollback"]

indexable=['rollback']
blocked=1

The chunk checksum detects changes to this text; it isn't the original document checksum and doesn't certify meaning. Retain the ingestion checksum and immutable source version separately. Include the splitter configuration and source span in an index manifest so re-ingestion can replace old chunks rather than mixing versions.

The section ordinals are application locators within this fixture, not clickable anchors embedded in a PDF. This gate checks metadata presence, nonempty text, and the prior quality decision. Production records must also carry the ingestion trust labels and authorization scope, with quarantined evidence blocked and permissions enforced on the final expanded context.

Practice: try to break the boundaries

1. Change the fixed-window size to ten and overlap to zero. Predict chunk lengths and complete rollback rules before running it.
2. Run the balanced fallback on fifteen words with size seven and minimum tail six. Can every child satisfy both length rules? Which rule must win?
3. Add No action is required. to the semantic lab. What happens to its topic vector, and why shouldn't that alone trigger a split?
4. In the parent-expansion lab, change one child's version to two, then remove permission, then reduce the budget to ten. Each run should fail before returning context. Restore valid inputs and confirm two hits still produce one parent.