Code Completion System

Content moderation put a safety decision on a low-latency path. Code completion adds a moving editor state: a developer types is_allowed =, continues with graph., and then receives a suggestion built for the older buffer. If accepting that suggestion replaces the newer line, the system has corrupted the edit. Even a correct suggestion feels broken when it arrives after the developer has moved on.

A code completion system predicts useful edits while a developer is still typing. GitHub Copilot's product family splits that job into dimmed ghost text at the cursor and next-edit suggestions that also predict where the next change should land.^{[1]Reference 1GitHub Copilot code suggestions in your IDEhttps://docs.github.com/en/copilot/concepts/completions/code-suggestions} Remote suggestions can use fill-in-the-middle (FIM), which conditions the missing code on both the text before the cursor and the text after it.

Coding agents that plan multi-file patches sit a layer above and get a different latency budget. In the design scenario below, the inline path has a 200 ms p95 service-level objective (SLO). A real product has to set and validate its own objective from user research and telemetry.

Before tracing the path, predict what should happen to an exact member lookup after graph. and to an open-ended assignment after is_allowed =. One should stay local; the other may justify remote generation. The figure makes that split, and its freshness gate, visible:

You've already seen how Transformer-based large language models (LLMs) predict the next token and how the key-value (KV) cache avoids recomputing shared work. This chapter puts those ideas on a keystroke path: the client may notice every edit, but it only sends qualified requests and only displays fresh results. The Python examples below are local contract tests with invented inputs, not model-quality or hardware benchmarks.

One cursor, five lanes

The running example is a half-typed validate_token_scope function. Ask which lane should complete PolicyGraph, which should answer graph., and which can suggest the expression after is_allowed =:

Lexical and semantic first

A prefix index can finish PolicyGraph from the import line. After graph., prefer the language server when it knows the receiver type and valid members. Dynamic code, missing dependencies, and incomplete indexing can leave it uncertain. A lexical fallback remains useful without a model call.

Inline generation for an open-ended span

The empty assignment is_allowed = is the generative case. No symbol table can invent the policy check that has to produce is_allowed for the return below. That's the fill-in-the-middle path the 200 ms budget is for.

Agents stay off the keystroke clock

A coding agent can research the repo, run tests, and edit several files.^{[2]Reference 2GitHub Copilot cloud agenthttps://docs.github.com/en/copilot/concepts/agents/cloud-agent/about-cloud-agent} That's a different product loop. Ghost text still has to gather a small prompt, generate a candidate, and stay under the 200 ms p95 clock.

The keystroke budget

An inline suggestion on is_allowed = competes with the next keystroke. If it arrives too late, the developer has already typed past it. If it's irrelevant, it becomes visual noise.

Those two failure modes set the constraints:

• Latency: For this scenario, keep single-line inline completion under 200 ms at p95. Give multi-line suggestions a separately measured budget.
• Context: Use open files, imports, function signatures, and nearby project structure, not the current file alone.
• Quality: Track acceptance and accepted-and-retained characters. The exact target varies by language, editor UX, and how aggressively the client shows suggestions.
• Scale: Large global developer fleets with diurnal spikes, so GPU utilization and cancel rate both matter.
• Privacy: Define what code may leave the client, what is retained, whether training use is disabled by default, and how tenants are isolated.

Measure edit-to-usable-suggestion from the qualifying editor event until a fresh span is painted. That includes debounce, context construction, queueing, network transfer, generation of enough text to display, and paint. Separately measure client time-to-first-token (TTFT) from request submission to the first response token. TTFT already includes network and server waiting; don't add it to those stages again.

This serial allocation describes an immediately triggered request, not observed percentiles. A longer prompt can increase transfer and prefill; profile both.

The allocation totals 183 ms, leaving 17 ms against the 200 ms objective. Adding the later 150 ms ordinary-typing debounce makes this path 333 ms. Either shorten that debounce, route locally, or declare a separate pause-to-suggestion objective. Don't hide the delay by starting the clock at HTTP submission.

The calculation below also shows why a percentile must be computed over complete request traces, not by summing stage percentiles:

import math

STAGE_BUDGET_MS = {
    "client_parse": 12,
    "network": 55,
    "context": 28,
    "queue_and_model_through_span": 78,
    "paint": 10,
}
INLINE_P95_OBJECTIVE_MS = 200

total_ms = sum(STAGE_BUDGET_MS.values())
headroom_ms = INLINE_P95_OBJECTIVE_MS - total_ms

assert total_ms <= INLINE_P95_OBJECTIVE_MS
assert total_ms + 150 > INLINE_P95_OBJECTIVE_MS

def p95(values: list[int]) -> int:
    if not values:
        raise ValueError("no completed traces")
    return sorted(values)[math.ceil(0.95 * len(values)) - 1]

# Invented traces: two different requests have slow stages.
traces = [(100, 1), (1, 100)] + [(1, 1)] * 18
assert p95([sum(trace) for trace in traces]) == 101
assert sum(p95(list(stage)) for stage in zip(*traces)) == 2
print("allocated_path_ms:", total_ms)
print("headroom_ms:", headroom_ms)
print("synthetic_trace_p95_ms:", p95([sum(trace) for trace in traces]))

allocated_path_ms: 183
headroom_ms: 17
synthetic_trace_p95_ms: 101

Who owns the is_allowed = keystroke

Three pieces share that keystroke: the IDE (Integrated Development Environment) extension, a context engine, and an inference server.

The extension owns keystrokes, open tabs, and cursor position. The context engine, which can run on the client or the gateway, picks the snippets that actually fit the live prompt budget. The Language Server Protocol (LSP) is the usual interface between an editor and a language server, so the same semantic engine can power completion, go-to-definition, and diagnostics across editors.^{[3]Reference 3Language Server Protocolhttps://microsoft.github.io/language-server-protocol/}

Now make the parser answer a smaller question: after the unfinished is_allowed =, which function and imports surround the cursor? Incremental parsers such as Tree-sitter are designed to produce useful syntax trees despite errors.^{[4]Reference 4Tree-sitterhttps://tree-sitter.github.io/tree-sitter/} Recovery can leave error or missing nodes; it doesn't guarantee a correct enclosing scope for every malformed buffer. Ask for the nearest reliable scope, fall back conservatively, and let the language server supply available type information.

That hybrid is the product, not an optimization. After a dot, semantic candidates from the language server are usually faster and more accurate than free-form generation. The model earns GPU time on longer spans, comment-to-code, and cases the type system doesn't capture.

The inference server hosts the model and uses continuous batching so many short, cancellable requests can share resident weights.

Local fallback lane

Not every keystroke should take the remote generation path. Keep a deterministic lane for the cheap, high-confidence cases:

• Local symbols and imported APIs: Exact member completion from the parser or language server.
• Prefix indexes or tries: Keyword and snippet lookup with predictable latency.
• Fuzzy matching: Edit-distance recovery for small typos like pritn instead of print.

That lane avoids network latency for exact matches and provides a fallback when remote inference is unavailable. Its latency still depends on index readiness and client workload.

A backward scan illustrates the difference between a cheap context hint and an enclosing-scope parser. This helper returns a preceding function header, including on incomplete input. It doesn't establish whether the cursor is inside that function:

def nearest_def_header(source: str, cursor_line: int) -> str | None:
    if cursor_line < 0:
        raise ValueError("cursor_line must be nonnegative")
    lines = source.splitlines()
    start = min(cursor_line, len(lines) - 1)
    for index in range(start, -1, -1):
        stripped = lines[index].lstrip()
        if stripped.startswith(("def ", "async def ")):
            return stripped
    return None

source = """from authz.policies import PolicyGraph

def validate_token_scope(request):
    is_allowed =
    return is_allowed
"""

assert nearest_def_header(source, 3) == "def validate_token_scope(request):"
assert nearest_def_header(source, 0) is None
assert nearest_def_header("", 0) is None
# A preceding header is not an enclosing scope after a dedent.
outside = "def finished():\n    pass\nx = 1\n"
assert nearest_def_header(outside, 2) == "def finished():"
print("preceding_header:", nearest_def_header(source, 3))

preceding_header: def validate_token_scope(request):

A production client uses syntax-node ranges and the language server for scope and types. The scan can match text inside multiline strings, miss multiline definitions, and cross a dedent. Never use it to decide symbol visibility or file authorization. Convert editor positions using the editor's documented encoding; UTF-16 code units, UTF-8 byte offsets, and Python string indices are not interchangeable. VS Code's Position.character counts UTF-16 code units; its positionAt and offsetAt methods convert between document positions and offsets in that same unit.^{[5]Reference 5VS Code API: document versions, positions, and inline completion providers (pinned source)https://github.com/microsoft/vscode/blob/1fe7285a1162756215a684ee702b16d0ce42bdb4/src/vscode-dts/vscode.d.ts#L269}

Put the caret after the emoji in a😀b. What suffix should an infill request preserve? It is b, but copying the editor's character offset directly into a Python slice loses it:

line = "a😀b"
python_index = 2  # caret after a and the emoji
prefix = line[:python_index]
utf16_offset = len(prefix.encode("utf-16-le")) // 2
utf8_offset = len(prefix.encode("utf-8"))

assert (python_index, utf16_offset, utf8_offset) == (2, 3, 5)
assert line[python_index:] == "b"
assert line[utf16_offset:] == ""  # wrong unit for this slice
print("offsets (Python, UTF-16, UTF-8):", python_index, utf16_offset, utf8_offset)
print("correct suffix:", repr(line[python_index:]))
print("wrong-unit suffix:", repr(line[utf16_offset:]))
try:
    line.encode("utf-8")[:utf16_offset].decode("utf-8")
except UnicodeDecodeError:
    print("wrong-unit byte slice: splits the emoji")
else:
    raise AssertionError("expected an incomplete UTF-8 sequence")

offsets (Python, UTF-16, UTF-8): 2 3 5
correct suffix: 'b'
wrong-unit suffix: ''
wrong-unit byte slice: splits the emoji

The unit belongs to the API contract, not the language of the file being edited. Preserve the editor snapshot, convert positions before slicing or replacing, and validate the range again against the current document version.

The client keeps lightweight parsing and exact-member lookup on-device, so only qualified prompts reach GPUs. The architecture figure above is that fork: local lane, remote lane, then a freshness gate.

Context gathering and management

The model needs enough context to make useful suggestions, but the live prompt budget is deliberately kept small because long prefills destroy latency. Even if the base model advertises a much larger context window, you can't stuff the whole repository into the keystroke path. The design question is not "how much can fit?" but "which tokens can change the next edit?"

Context priority strategy

First filter candidates by authenticated repository access and upload policy, including unsaved buffers and retrieved files. Rank only the permitted candidates. In this scenario the entire prompt is capped at 8,000 tokens, including FIM markers and other framing; reserve output separately within the model's total context limit. Keep a minimum suffix allocation before filling the remaining space with prefix and retrieved definitions. A greedy prefix-first truncation can otherwise erase the infill boundary.

Fill-in-the-middle (FIM)

Standard causal language models predict the next token based only on the past (left-to-right). In coding, you often insert code in the middle of a file. Pause before naming the format: if the suffix contains return is_allowed, what can a prefix-only model not see? It might generate valid code that conflicts with the closing braces or logic below.

FIM context: A prefix-only completion prompt exposes only code before the cursor. FIM also exposes code after the cursor before generation, so the completion can match a return type, closing delimiter, later variable use, or test expectation that appears below the edit.

FIM reorders the prompt so the model sees the suffix before generating the middle. Marker strings differ by tokenizer and model; the notation below names their roles rather than defining a universal API:

$P_{\text{FIM}} = \texttt{<PRE>} + x_{\text{prefix}} + \texttt{<SUF>} + x_{\text{suffix}} + \texttt{<MID>}$

Building the FIM prompt

Build the input in FIM order: prefix marker + code before cursor + suffix marker + code after cursor + middle marker. Then the model generates the missing middle section.

Use a concrete example: you're editing an authorization helper and your cursor sits inside an empty function body.

Prefix (code before cursor)

from authz.policies import PolicyGraph

def validate_token_scope(request):
    """Check whether the requested API scope is allowed."""
    is_allowed =

Suffix (code after cursor)

    return is_allowed

FIM prompt sent to the model

<PRE>from authz.policies import PolicyGraph

def validate_token_scope(request):
    """Check whether the requested API scope is allowed."""
    is_allowed = <SUF>
    return is_allowed
<MID>

The suffix gives the model a constraint, not an authorization specification. PolicyGraph is a fictional API here; real context must include its actual signature and policy semantics. Generated permission checks require security review and tests. No completion should grant access or run tools merely because the model emitted it.

Before wiring up a real tokenizer, test the transformation itself. This formatter is notation only. A serving adapter must insert the model's sentinel IDs and encode code as ordinary content, rather than accidentally interpreting marker-like strings inside a file as control tokens.

def format_fim(prefix: str, suffix: str) -> str:
    return f"<PRE>{prefix}<SUF>{suffix}<MID>"

prefix = "def validate_token_scope(request):\n    is_allowed = "
suffix = "\n    return is_allowed\n"
prompt = format_fim(prefix, suffix)

assert prompt.endswith("<MID>")
assert prompt.index("<SUF>") < prompt.index("return is_allowed")
print(prompt.replace("\n", "\\n"))

<PRE>def validate_token_scope(request):\n    is_allowed = <SUF>\n    return is_allowed\n<MID>

FIM gives a decoder-only model suffix information without changing left-to-right decoding. A native FIM adapter must use the checkpoint's documented sentinel tokens and input format. An instruction-following causal model can also receive prefix and suffix in an ordinary task prompt and attempt an insertion; left-to-right attention doesn't prohibit that. It is a different interface, whose output boundaries, quality, and latency need their own evaluation. Arbitrary <PRE> strings don't establish native FIM support.

During pre-training, a subset of examples is transformed:

1. Split a sequence at two boundaries into (prefix, middle, suffix).
2. Reorder to (prefix, suffix, middle) with sentinel tokens between segments.
3. Apply causal next-token training to the transformed sequence; at its final segment, the model predicts the middle conditioned on the preceding prefix and suffix.

The two common inference formats are:

• PSM (Prefix-Suffix-Middle): <PRE>prefix<SUF>suffix<MID>middle. This is the format used in the prompt above, and the one Qwen2.5-Coder documents for file-level completion.^{[6]Reference 6Qwen2.5-Coder Technical Reporthttps://arxiv.org/abs/2409.12186}
• SPM (Suffix-Prefix-Middle), in Bavarian et al.'s joint-training encoding: <PRE><SUF>suffix<MID>prefixmiddle. Prefix and generated middle are contiguous. At inference, stop the prompt after prefix and generate middle. Other SPM encodings exist, so follow the checkpoint's training format.^{[7]Reference 7Efficient Training of Language Models to Fill in the Middle.https://arxiv.org/abs/2207.14255}

Bavarian et al. found that FIM pretraining could preserve left-to-right capability on their evaluations. That result doesn't guarantee every checkpoint or data mixture behaves identically. Character-level span selection also exposes mid-token edits during training.^{[7]Reference 7Efficient Training of Language Models to Fill in the Middle.https://arxiv.org/abs/2207.14255}

Repository-level context

Large repos need more than simple file buffering. Retrieve definitions that can change the next few tokens, using an index updated outside the request path. Imagine scope_validator.py imports PolicyGraph and ScopeRule, while metrics.py contains only counters. Two candidate ranking methods are:

1. Jaccard similarity: Divide the intersection of unique symbols by their union. In the fixture below, three active symbols and three candidate symbols share two names: the score is 2 / (3 + 3 - 2) = 0.5. That is a ranking hint, not proof of relevance or permission.
2. BM25 / Sparse Retrieval: A lightweight keyword search over the local repo index to find defining files for classes used in the current buffer.^{[8]Reference 8The Probabilistic Relevance Framework: BM25 and Beyond.https://doi.org/10.1561/1500000019}

Dense vector retrieval can also fit if embeddings and indexes are ready. Measure lookup, optional reranking, and prompt assembly together. This design leaves iterative retrieve-generate-retrieve loops on the slower route because each serial model call consumes another latency budget.

This three-file fixture keeps scope_rules.py and policy_graph.py. It tests ranking arithmetic, not repository-scale lookup speed. All candidates are assumed already authorized:

active_symbols = {"PolicyGraph", "ScopeRule", "TokenClaims"}
candidate_files = {
    "scope_rules.py": {"ScopeRule", "TokenClaims", "AuditEvent"},
    "metrics.py": {"Counter", "Histogram"},
    "policy_graph.py": {"PolicyGraph", "Node"},
}

def jaccard(left: set[str], right: set[str]) -> float:
    union = left | right
    return len(left & right) / len(union) if union else 0.0

ranked = sorted(
    ((jaccard(active_symbols, symbols), path) for path, symbols in candidate_files.items()),
    reverse=True,
)
selected = [path for score, path in ranked if score >= 0.25]

assert selected == ["scope_rules.py", "policy_graph.py"]
assert jaccard(set(), set()) == 0.0
assert jaccard({"A"}, {"A"}) == 1.0
print("selected_context:", selected)

selected_context: ['scope_rules.py', 'policy_graph.py']

The score is a retrieval signal, not proof that a file belongs in the prompt. If a known definition repeatedly misses the threshold, inspect symbol extraction and index freshness before increasing the context window.

Model serving for low latency

Profile queueing, prefill, and decode separately. Long context can make prefill dominant, but short output doesn't make decode free. Speculation targets serial decoding, not prompt prefill, and can worsen first-token latency through added draft work. If it proposes five tokens and rejects token five, how many tokens does a correction step emit?

Speculative decoding

In the classical two-model version of speculative decoding, a smaller, faster draft model proposes the next $K$ tokens, then a larger target model verifies them in one parallel pass.^{[9]Reference 9Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192} Other proposal methods include prompt n-grams and checkpoint-native multi-token prediction, so a separate learned draft model isn't universal. Serving stacks expose several methods with compatibility and workload-specific caveats; measure acceptance and latency before rollout.^{[10]Reference 10Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}

If the target agrees, you emit several tokens after one target pass. If the draft misses, the target supplies the correction and you continue from there.

Treat graph, ., allows, (, and user as five toy token IDs; an actual tokenizer may segment them differently. With four accepted drafts, the correction request makes five emitted tokens. With one accepted draft, a correction makes two. With an immediate rejection, only the correction is emitted, but draft overhead was still paid. If all five are accepted, the standard algorithm can emit a sixth, bonus target token, subject to stopping rules.

For stochastic decoding, "agrees" means accepting draft token $x$ with probability $\min(1,p(x)/q(x))$, where $p$ and $q$ are target and draft distributions over the same token vocabulary. On rejection, sample from normalized $\max(p-q,0)$. A greedy implementation instead verifies against the target's greedy choices. A confidence threshold isn't a substitute for this distribution-preserving rule.^{[9]Reference 9Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

Under an intentionally simplified cost model, five draft steps costing one target step plus verification costing one more yield 5 / 2 = 2.5 times the baseline output per unit cost when four drafts survive. Verification is not generally one single-token step in cost; this is arithmetic, not a performance forecast.

Code with repetitive imports or boilerplate can raise acceptance. Treat that as a rollout hypothesis, not a property of all languages: measure accepted draft length and end-to-end latency by language and suggestion type before enabling it broadly.^{[9]Reference 9Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

The sketch treats target verification as roughly one target step for the whole draft. Under continuous batching, verifying many draft tokens can steal decode slots from neighbors, raise their TTFT, and erase the headline speedup. Re-measure with production batch occupancy; see speculative decoding for that interaction.

The following invented measurements test a candidate-selection rule. They are not observed Python or SQL performance. A release also needs target-distribution or greedy-equivalence checks, representative load, enough samples for tail estimates, and unchanged quality and failure-rate guardrails:

measurements = {
    "python_imports": {"mean_accepted_tokens": 3.8, "baseline_p95_ms": 178, "spec_p95_ms": 136},
    "sql_queries": {"mean_accepted_tokens": 1.1, "baseline_p95_ms": 169, "spec_p95_ms": 176},
}

def enable_speculation(sample: dict[str, float]) -> bool:
    latency_gain_ms = sample["baseline_p95_ms"] - sample["spec_p95_ms"]
    return sample["mean_accepted_tokens"] >= 2.0 and latency_gain_ms >= 10

enabled = [name for name, sample in measurements.items() if enable_speculation(sample)]
assert enabled == ["python_imports"]
print("speculative_decode_enabled_for:", enabled)

speculative_decode_enabled_for: ['python_imports']

KV cache reuse (prefix caching)

Developers often type, pause, and type again in the same file. The file's header (imports, class definitions, and previously written functions) remains constant across these rapid sequential interactions. Ask what a cache hit can remove before reading the timing example: prefill work, decode work, or both?

Instead of prefilling the same stable file header on every request, the server can cache Key-Value (KV) blocks for a shared prefix. When the next prompt begins with the same cacheable blocks under the same tenant and model policy, it reuses those blocks and prefills only the uncached delta. Automatic prefix caching does not make generation of new output tokens cheaper; it removes duplicate prefill work for reused input context.^{[11]Reference 11Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

For a synthetic arithmetic example, suppose the complete 1,280-token prompt contains a 1,200-token reusable prefix and an 80-token delta. Assign a miss 72 ms of prefill plus 24 ms through the output span; assign a hit 11 ms plus 24 ms. The resulting model-service totals are 96 ms and 35 ms. They exclude network, queueing, and paint and are not TTFT measurements.

Reuse requires an exact token prefix and compatible KV computation, not just matching source text. Include framing and retrieved context before the file. Appending characters can change the last token, so retokenize and match full cache blocks. In PSM, editing the prefix also invalidates the later suffix's cached states. SPM can preserve a suffix-first prefix when supported by the trained model.

Those savings depend on stable prefixes, block matching, routing affinity, and isolation. A long header with a small cursor-adjacent delta is a good candidate. An early edit or a cross-tenant request is a miss by design.

Use a cache key that enforces isolation as well as computation compatibility. The tuple below is a contract fixture, not vLLM's block-hash implementation or a real KV tensor. The gateway supplies the security scope from authenticated identity and current policy; it must not trust a tenant or salt chosen by an arbitrary client. vLLM's cache_salt can isolate reuse between trust groups.^{[11]Reference 11Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

from dataclasses import dataclass, replace

@dataclass(frozen=True)
class CacheKey:
    security_scope: str  # server-derived tenant/repo/access-policy epoch
    model_revision: str
    tokenizer_revision: str
    adapter_revision: str
    kv_layout: str       # dtype, position policy, and engine compatibility
    prefix_ids: tuple[int, ...]

same_scope = CacheKey("acme/repo-a/policy-7", "weights-3", "tok-3",
                      "no-adapter", "fp16-layout-1", (10, 20, 30))
cached = {same_scope: "illustrative-kv-handle"}
other_tenant = replace(same_scope, security_scope="contoso/repo-a/policy-7")

assert cached.get(same_scope) == "illustrative-kv-handle"
assert cached.get(other_tenant) is None
assert replace(same_scope, prefix_ids=(10, 20, 31)) not in cached
assert replace(same_scope, adapter_revision="adapter-2") not in cached
assert replace(same_scope, security_scope="acme/repo-a/policy-8") not in cached
print("same_scope_hit:", same_scope in cached)
print("cross_tenant_hit:", other_tenant in cached)

same_scope_hit: True
cross_tenant_hit: False

KV reuse is not response caching: sampling settings can change without changing prefix KV, whereas reusing a completed suggestion requires its full input, model, generation settings, and policy contract. Neither kind of cache bypasses current authorization or the editor's freshness gate.

Quantization

Code models may tolerate post-training quantization, but re-check completion quality after compression. Relative to FP16, 8-bit and 4-bit weights use half and one quarter of the raw weight payload. Scales, metadata, unquantized layers, KV cache, and temporary buffers reduce whole-model savings.^{[12]Reference 12GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformershttps://arxiv.org/abs/2210.17323[13]Reference 13GPTQhttps://huggingface.co/docs/transformers/main/en/quantization/gptq}

Weight compression can help bandwidth-bound decode when the hardware has efficient kernels for that format. It doesn't guarantee proportional traffic reduction or speedup. Profile prefill and decode separately, and size memory for the admitted batch's KV states as well as weights.

GPTQ (a post-training method for generative pretrained transformers) compresses off-the-shelf weights without a full fine-tune. That's useful when you need a FIM model to fit a specific GPU, not when you want a new capability.

Debounce r and o; fire on .

Sending remote work for every edit of is_allowed floods the GPU with requests that will be stale before they return. The client has to decide when to request, when to wait, and when to cancel. Imagine requests for r, o, and . crossing in flight. Predict which response the editor may paint before following the request IDs through the diagram:

Debouncing and cancellation

For the cancellation demonstration, use a dynamic debounce:

• 0 ms delay on trigger characters such as ., (, and \n.
• 150 ms delay on ordinary typing.

The 150 ms setting doesn't satisfy the earlier 200 ms edit-to-paint allocation. It's a debounce fixture, not a recommended production value. Invalidate the epoch immediately on every document, cursor, selection, or policy change, even when no replacement request will be sent. Clear visible ghost text too. VS Code's provider receives a document, position, completion context, and cancellation token; the client still needs to validate its application state when asynchronous work finishes.^{[14]Reference 14Visual Studio Code API: InlineCompletionItemProviderhttps://code.visualstudio.com/api/references/vscode-api#InlineCompletionItemProvider}

This local test covers cancelled debounce tasks, an explicitly delivered late response, a cursor-only move, and a change between paint and acceptance. It uses no HTTP service:

import asyncio
from dataclasses import dataclass

@dataclass(frozen=True)
class Snapshot:
    document: str
    version: int
    selection: tuple[int, int]  # Python offsets in this fixture only
    policy_epoch: int

class RequestManager:
    def __init__(self):
        self.pending_task: asyncio.Task[None] | None = None
        self.epoch = 0
        self.snapshot: Snapshot | None = None
        self.ghost: tuple[int, Snapshot, str] | None = None
        self.trigger_chars = {'.', '(', '\n'}
        self.sent_requests: list[int] = []
        self.shown_suggestions: list[str] = []

    def on_change(self, snapshot: Snapshot, char: str | None = None) -> None:
        self.epoch += 1
        self.snapshot = snapshot
        self.ghost = None
        if self.pending_task and not self.pending_task.done():
            self.pending_task.cancel()
        if char is None:  # cursor/policy change: invalidate without requesting
            return
        delay_sec = 0.0 if char in self.trigger_chars else 0.15
        self.pending_task = asyncio.create_task(
            self.debounce_fetch(delay_sec, self.epoch, snapshot)
        )

    async def debounce_fetch(self, delay: float, epoch: int, snapshot: Snapshot) -> None:
        try:
            await asyncio.sleep(delay)
            self.sent_requests.append(epoch)
            self.handle_response(epoch, snapshot, f"completion-{epoch}")
        except asyncio.CancelledError:
            return

    def handle_response(self, epoch: int, snapshot: Snapshot, suggestion: str) -> None:
        if epoch == self.epoch and snapshot == self.snapshot:
            self.ghost = epoch, snapshot, suggestion
            self.shown_suggestions.append(suggestion)

    def accept(self) -> str | None:
        if self.ghost is None:
            return None
        epoch, snapshot, suggestion = self.ghost
        self.ghost = None
        if epoch != self.epoch or snapshot != self.snapshot:
            return None
        return suggestion  # real editor applies a version-checked edit atomically

async def demo() -> None:
    manager = RequestManager()
    tasks = []
    for version, char in enumerate('ro.', 1):
        snapshot = Snapshot("handler.py", version, (version, version), 7)
        manager.on_change(snapshot, char)
        tasks.append(manager.pending_task)
    await asyncio.gather(*tasks, return_exceptions=True)
    manager.handle_response(2, snapshot, "late-completion-2")
    assert manager.ghost is not None
    manager.on_change(Snapshot("handler.py", 3, (0, 0), 7))
    manager.handle_response(3, snapshot, "late-after-cursor-move")
    assert manager.accept() is None
    current = manager.snapshot
    assert current is not None
    manager.handle_response(manager.epoch, current, "fresh")
    manager.on_change(Snapshot("handler.py", 3, (0, 0), 8))
    assert manager.accept() is None  # policy changed after paint
    assert manager.sent_requests == [3]
    assert manager.shown_suggestions == ["completion-3", "fresh"]
    print("sent_requests:", manager.sent_requests)
    print("shown_suggestions:", manager.shown_suggestions)

asyncio.run(demo())

sent_requests: [3]
shown_suggestions: ['completion-3', 'fresh']

Propagate an abort to the server's queued or running request, but don't assume closing a connection instantly stops GPU work. Bound deadlines and output length, account for work already executed, and discard late chunks. Check freshness again when Tab applies the edit, using the editor's atomic version/range contract. A render-time check alone leaves a race before insertion.

Ghost text, not a second product

The inline surface is ghost text: a suggestion at the cursor.^{[15]Reference 15Programmatic Language Features: Show Inline Completionshttps://code.visualstudio.com/api/language-extensions/programmatic-language-features} A client may keep a remainder when typed characters match, but only by explicitly rebasing it to the new document version and range. Otherwise invalidate it. A comment such as // validate token scopes against the policy graph can still trigger an inline completion, with a natural-language prefix.

Repo-wide questions and test generation belong on the slower agent path, not on is_allowed =. Mixing them into the keystroke SLO is how teams accidentally build a chat product with ghost-text cosmetics.

Did is_allowed = actually help?

A developer can accept a whole suggestion, take just a word or line, or ignore it. Copilot supports partial acceptance, so a binary accept event hides how much text was actually inserted.^{[16]Reference 16Getting code suggestions in your IDE with GitHub Copilot: accepting partial suggestionshttps://docs.github.com/en/copilot/how-tos/get-code-suggestions/get-ide-code-suggestions#accepting-partial-suggestions} A four-character accept that gets deleted still counts as an accept. Suppose a new model raises acceptance while developers delete more generated code. Which release improved the product? Hold that question while the figure separates inserted text from retained text.

A useful scoreboard measures how often suggestions appear, how often they're inserted, how much of that inserted text survives, and whether the path still meets its latency objective. Split those metrics by language, suggestion class, and traffic period or a model regression hides inside an aggregate.

Offline evaluation on benchmarks like HumanEval^{[18]Reference 18Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} is useful for catching model regressions, but it doesn't measure editor timing, shown-rate policy, or accepted edits that users later undo. An online experiment is needed to determine whether a model or context heuristic improves the product experience.

Optimizing solely for acceptance rate can lead to a model that only suggests short, obvious tokens like closing parens because they are safe. You need value metrics such as retained characters, not click-through alone.

The metric computation should retain that distinction. This synthetic cohort has one row per distinct shown suggestion, three with accepted text and two with surviving characters. accepted_chars counts only text actually inserted, including a partial accept. Repeated word-by-word accepts of the same shown suggestion must not become several accepted suggestions. Size alone can't tell whether either edit was useful or correct. The fixture counts Python characters; production telemetry must choose a consistent unit and observation window:

suggestions = [
    {"shown": True, "accepted_chars": 18, "retained_chars": 0},
    {"shown": True, "accepted_chars": 4, "retained_chars": 4},
    {"shown": True, "accepted_chars": 42, "retained_chars": 35},
    {"shown": True, "accepted_chars": 0, "retained_chars": 0},
]

def metrics(events):
    for item in events:
        assert 0 <= item["retained_chars"] <= item["accepted_chars"]
        assert item["shown"] or item["accepted_chars"] == 0
    shown = sum(item["shown"] for item in events)
    accepted = sum(item["accepted_chars"] > 0 for item in events)
    inserted = sum(item["accepted_chars"] for item in events)
    retained = sum(item["retained_chars"] for item in events)
    return (accepted / shown if shown else None,
            retained / inserted if inserted else None)

acceptance_rate, retained_char_rate = metrics(suggestions)
assert metrics([]) == (None, None)
assert metrics([suggestions[-1]]) == (0.0, None)
assert acceptance_rate == 0.75
assert round(retained_char_rate, 3) == 0.609
print("acceptance_rate:", f"{acceptance_rate:.0%}")
print("retained_char_rate:", f"{retained_char_rate:.1%}")

acceptance_rate: 75%
retained_char_rate: 60.9%

For an offline completion set, freeze the buffer at a real edit point, hide the target insertion, and retrieve only files available at that point. Split by repository and time where possible to avoid near-duplicate leakage. Test whether the reconstructed file parses and whether relevant tests pass inside an isolated, no-secret execution environment. Exact-string match penalizes valid alternatives; passing tests still doesn't prove a safe policy check.

For an online experiment, randomize at a stable user or workspace boundary, hold trigger and display policy fixed, and compare eligible, requested, shown, accepted, and retained counts. Count cancellation, timeout, stale-drop, and error rates alongside latency; a low p95 among the few responses shown can hide a failing service. Treat unobserved retention windows as censored, not automatically deleted. Track retained characters locally where possible rather than uploading source to compute them.

Tie release metrics to language and suggestion-class slices. Investigate queueing and cache behavior when latency regresses, and model/context changes when quality falls. Use predefined rollback guardrails with enough observations, not one noisy percentile. Keep a last known-good route for each slice.

Routing by suggestion size

A more capable model can write a better function body and still lose if it arrives after the developer has moved on. A fast model that emits the wrong API also fails.

Start by evaluating a smaller FIM-trained checkpoint for the keystroke path; measure its speed and accuracy. The Qwen2.5-Coder report documents FIM training, file-level formatting, and repository markers, making it an example of the required interface rather than a current model recommendation.^{[6]Reference 6Qwen2.5-Coder Technical Reporthttps://arxiv.org/abs/2409.12186} A larger model or distilled model is another candidate, subject to the same measured constraints.

Don't assume parameter count predicts editor usefulness. Measure TTFT, acceptance, and retained characters on the actual trigger mix.

How much to generate is a separate routing decision. A single extra token after . and an empty function body have different requirements. For this design, assign separate scenario targets:

A lightweight client classifier can pick the request class before sending remote work. The fixture makes policy denial override remote routing even for a multiline request. Only the actual boolean True permits the remote lane; malformed or missing policy values keep the buffer local. The gateway must independently enforce that policy; client classification is not authentication.

def route(trigger: str, after_signature: bool, *,
          semantic_available: bool = True, remote_permitted: bool = True) -> tuple[str, int]:
    if semantic_available and trigger in {".", "("}:
        return "LOCAL_SEMANTIC", 60
    if remote_permitted is not True:
        return "LOCAL_FALLBACK", 60
    if after_signature:
        return "REMOTE_MULTILINE", 500
    return "REMOTE_INLINE", 200

cases = [
    route(".", False),
    route("\n", True),
    route("r", False),
]
assert cases == [
    ("LOCAL_SEMANTIC", 60),
    ("REMOTE_MULTILINE", 500),
    ("REMOTE_INLINE", 200),
]
assert route(".", False, semantic_available=False) == ("REMOTE_INLINE", 200)
assert route("\n", True, remote_permitted=False) == ("LOCAL_FALLBACK", 60)
for unknown_policy in ("False", None, 1):
    assert route("\n", True, remote_permitted=unknown_policy) == ("LOCAL_FALLBACK", 60)
print("string False policy:", route("\n", True, remote_permitted="False"))
print("routes:", cases)

string False policy: ('LOCAL_FALLBACK', 60)
routes: [('LOCAL_SEMANTIC', 60), ('REMOTE_MULTILINE', 500), ('REMOTE_INLINE', 200)]

A Python string "False" is truthy. A gate using if not remote_permitted would choose the remote multiline lane for that value and could upload a buffer the configuration intended to keep local. The explicit is not True check treats it as unknown permission. Validate policy at its source too; this local choice still needs independent gateway enforcement.

The fleet behind one keystroke

Code completion is a high-churn GPU workload: many requests die while the developer is still typing is_allowed. Provisioning has to protect the 200 ms objective during spikes, not just average tokens per second.

GPU fleet management and load balancing

Route for latency, cache affinity, and isolation at the same time:

• Prefix-aware routing: Prefer an authorized instance with matching warm KV blocks when its queue permits. An overloaded warm instance can lose more time in queueing than a cooler instance spends on prefill; cache affinity isn't an unconditional pin.
• Geographic routing: Prefer an approved nearby region when residency and capacity allow. A 50 ms network round trip already consumes one quarter of the 200 ms objective.
• Model tiering: Smaller models for single-line completions, larger models for multi-line bodies.

Fleet sizing sketch

Count tokens, not "editors per GPU." Before calculating, estimate the surviving prefill load for 5,000 active editors. The cancellation rate and prefix-hit rate both change the answer, so one labeled peak-hour scenario is:

1. 5,000 editors actively typing in a region, not 50,000 idle sessions.
2. After debounce, about 0.4 remote requests per typing editor per second → 2,000 raw req/s.
3. 60% cancelled before GPU admission → 800 requests that still need serving work. A cancel after admission still burns partial GPU time and belongs in utilization.
4. Every surviving request has 2,048 prompt tokens in this worksheet. 50% are prefix-cache hits that skip 75% of their prefill. Equal mean lengths in the hit and miss groups would give the same arithmetic.

Effective prefill rate is then 800 × 2048 × (0.50 × 0.25 + 0.50 × 1.0) = 1,024,000 tokens/s. With an assumed capacity of 20,000 effective prefill tokens/s per GPU, the quotient rounds up to 52. This is a prefill-only arithmetic envelope, not a fleet recommendation. Tokens are not uniform work across sequence lengths. Decode, partial work on late cancellations, KV-memory capacity, load imbalance, failover, and headroom remain unmodeled. Measure the mixed workload at the required latency before choosing a GPU count.

Now change only the length distribution. Keep 800 req/s, a 2,048-token overall mean, and a 50% request hit rate. Can you still multiply those averages? No: which requests hit matters.

The aggregate mean and request hit rate are identical in all three rows. Sum each cohort's rate times its remaining prompt tokens, or measure uncached token work directly. A request-level hit rate alone doesn't give a token-weighted saving.

import math

TYPING_EDITORS = 5_000
REQ_PER_EDITOR_PER_S = 0.4
CANCEL_BEFORE_GPU = 0.60
TOKENS_PER_REQUEST = 2_048
PREFIX_HIT_RATE = 0.50
PREFIX_SKIP_FRACTION = 0.75
GPU_PREFILL_TOK_PER_S = 20_000

raw_qps = TYPING_EDITORS * REQ_PER_EDITOR_PER_S
gpu_qps = raw_qps * (1 - CANCEL_BEFORE_GPU)
miss_or_partial = (1 - PREFIX_HIT_RATE) + PREFIX_HIT_RATE * (1 - PREFIX_SKIP_FRACTION)
prefill_tok_s = gpu_qps * TOKENS_PER_REQUEST * miss_or_partial
gpus = math.ceil(prefill_tok_s / GPU_PREFILL_TOK_PER_S)

assert raw_qps == 2_000
assert gpu_qps == 800
assert prefill_tok_s == 1_024_000
assert gpus == 52
print("gpu_qps:", gpu_qps)
print("prefill_tok_s:", int(prefill_tok_s))
print("busy_gpus:", gpus)

def prefill_load(cohorts):
    return sum(qps * tokens * (1 - skip) for qps, tokens, skip in cohorts)

long_hits = [(400, 3_072, 0.75), (400, 1_024, 0.0)]
long_misses = [(400, 1_024, 0.75), (400, 3_072, 0.0)]
for cohorts in (long_hits, long_misses):
    assert sum(qps for qps, _, _ in cohorts) == 800
    assert sum(qps * tokens for qps, tokens, _ in cohorts) / 800 == 2_048
assert prefill_load(long_hits) == 716_800
assert prefill_load(long_misses) == 1_331_200
print("long-hit prefill/GPU envelope:", int(prefill_load(long_hits)),
      math.ceil(prefill_load(long_hits) / GPU_PREFILL_TOK_PER_S))
print("long-miss prefill/GPU envelope:", int(prefill_load(long_misses)),
      math.ceil(prefill_load(long_misses) / GPU_PREFILL_TOK_PER_S))

gpu_qps: 800.0
prefill_tok_s: 1024000
busy_gpus: 52
long-hit prefill/GPU envelope: 716800 36
long-miss prefill/GPU envelope: 1331200 67

Continuous batching

Waiting to fill a fixed batch can add queue delay before first-token work begins. Continuous batching allows requests to join and leave an active serving schedule. vLLM^{[19]Reference 19Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} supports optimized scheduling and uses PagedAttention to manage KV-cache storage in blocks. TGI also documents continuous batching, although Hugging Face now labels TGI maintenance mode and recommends newer engines such as vLLM or SGLang for future work.^{[20]Reference 20Text Generation Inference.https://huggingface.co/docs/text-generation-inference/index} More batching isn't automatically faster. Ask whether its saved GPU work outweighs added queueing at the inline p95. Tune batching against TTFT and throughput because higher utilization can still harm inline latency when queues grow.

Cost economics

At scale, code completion is expensive because the IDE emits a steady stream of short-lived requests while the user is typing. The key design question isn't just cost per token. It's cost per useful suggestion.

• High churn: Many requests are cancelled before the user ever sees the output.
• Prefill-heavy workload: Much of the cost sits in reading context, not in generating long responses.
• Useful north-star metric: Track cost per accepted-and-retained suggestion, not cost per request alone.

Those controls can reduce wasted work, but aggressive abstention can also reduce usefulness, and cache-affinity routing can overload a warm shard. Compare cost and user outcomes together.

What of this buffer may leave the laptop

validate_token_scope may sit next to proprietary source, configuration, and credentials accidentally present in the buffer. The product has to say what may be uploaded, logged, or retained before the first remote call. Before trusting TLS or a cache setting, ask whether the policy still holds in prompt logs, GPU batches, and recovery paths.

A useful design begins with explicit contracts: which buffers may be uploaded, whether prompts can be logged or used for training, how long operational data is retained, and which controls prevent cross-tenant reuse.

Data isolation

For an enterprise configuration that forbids code reuse across organizations, enforce tenant isolation across request ingestion, telemetry, and caches:

• No training on user code is this design's contractual default, not a claim about every vendor. Disable downstream training exports and raw-prompt collection too. Frozen inference weights alone don't enforce this policy.
• Prompt data retention: Define explicit, minimal retention policies. Enterprises often require no raw-code retention or short, auditable windows for operational logs.
• Tenant isolation: Prevent one organization's code context from leaking through another organization's cache, logs, or outputs. Shared GPU batches can be acceptable only when the runtime preserves per-request state boundaries and observability keeps tenant scopes intact. Partition batches where policy or implementation can't prove that boundary.

An allowlist can demonstrate minimal telemetry, but it's not an end-to-end privacy test. Logs in proxies, exceptions, tracing, crash dumps, and third-party SDKs need separate checks. The fixture below accepts already validated identifiers and a bounded latency bucket; arbitrary client-provided strings could themselves contain source or secrets:

def audit_record(request: dict[str, str]) -> dict[str, str]:
    record = {
        "tenant": request["tenant"],
        "model": request["model"],
        "latency_bucket": request["latency_bucket"],
    }
    return record

request = {
    "tenant": "acme-devtools",
    "model": "code-fim-v3",
    "latency_bucket": "100_to_200ms",
    "prompt": "API_TOKEN='secret-value'",
}
record = audit_record(request)

assert "prompt" not in record
print("audit_fields:", sorted(record))

audit_fields: ['latency_bucket', 'model', 'tenant']

Secret redaction and upload controls

For enterprise usage, client-side redaction can reduce the chance that recognized secrets leave the developer's machine. Regular-expression and entropy-based scanners can catch some API keys, tokens, and passwords, while policy-based filters can mask selected identifiers before prompt construction. Scanners are incomplete, so redaction complements upload controls, restricted logging, access control, and incident response.

If a policy promises that matching secret patterns won't be uploaded, replacement must run before upload. The system replaces matched strings with placeholders such as <API_KEY> before constructing the remote prompt.

Don't automatically restore secrets into model-chosen output positions, even for a known placeholder: generated code could put the value into a log or an outbound request. Keep placeholders visible and require explicit local review, preferably replacing them with a secret-store lookup. A repository comment or retrieved file is untrusted model input, not authority to change upload policy, read another repository, or execute a tool.

import re

TOKEN = re.compile(r"demo_api_token_[A-Za-z0-9]+")

def redact(text: str) -> tuple[str, dict[str, str]]:
    if "<API_KEY_" in text:
        raise ValueError("placeholder collision: do not upload this fixture")
    mapping: dict[str, str] = {}
    def replace(match: re.Match[str]) -> str:
        placeholder = f"<API_KEY_{len(mapping) + 1}>"
        mapping[placeholder] = match.group(0)
        return placeholder
    return TOKEN.sub(replace, text), mapping

prompt, local_mapping = redact("client = API('demo_api_token_abc123')")
assert "demo_api_token_" not in prompt
assert local_mapping["<API_KEY_1>"] == "demo_api_token_abc123"
assert redact("ordinary code")[0] == "ordinary code"
print("upload_prompt:", prompt)

upload_prompt: client = API('<API_KEY_1>')

On-premises deployment

Some enterprise policies prohibit sending source code to a shared external service. A product serving those customers may need private Virtual Private Cloud (VPC), self-hosted, or offline deployment options.

• Self-hosted models: Deploy suitable code models inside customer-controlled infrastructure. This changes trust boundaries, but still requires identity, network, logging, and supply-chain controls.
• Air-gapped environments: Support fully offline operation for classified environments. The context engine, model, and inference server all run locally.

Public-code matching and the mastery gap

A completion can be locally useful and still be the wrong artifact to insert. Copilot-style products can check suggestions against publicly available code and then discard a match or attach a reference, depending on policy.^{[1]Reference 1GitHub Copilot code suggestions in your IDEhttps://docs.github.com/en/copilot/concepts/completions/code-suggestions} That's one control for training-data regurgitation, not a license to skip review.

A separate risk is accepting code without understanding it. Suggestions can invent APIs, use deprecated calls, or introduce insecure behavior. Retention is a usage signal, not evidence that a developer understood the code. Pair it with review and correctness checks, especially for the authorization helper in this example.

Common pitfalls

Treat these as diagnoses, not a release checklist. Start with the first user-visible symptom, isolate the stage or state that can cause it, change one lever, and compare the next measured slice. That sequence keeps a model-quality complaint from turning into blind serving work.

• Symptom: Suggestions appear after the user already typed past them. Cause: No real cancellation path, or UI trusts arrival order instead of request IDs. Fix: Abort when possible, invalidate on every relevant state change, and check the snapshot at paint and acceptance.

• Symptom: Completions are often exact but still feel unhelpful. Cause: System optimizes for raw acceptance with tiny safe suggestions. Fix: Track accepted-and-retained characters and shown rate, not acceptance alone.

• Symptom: Member completion is slower and less accurate after . than IDE autocomplete used to be. Cause: Every keystroke is routed to the LLM instead of keeping semantic lane for deterministic cases. Fix: Let parser or language server own exact symbol completion and reserve GPU work for open-ended spans.

• Symptom: Prefix caching hit rate stays low even though users edit the same file repeatedly. Cause: Routing breaks shard affinity, so matching prefixes miss the cached KV blocks. Fix: Inspect token-prefix stability, block eviction, compatible model/adapter state, and permitted cache scope before adding shard affinity.

• Symptom: Inserted code fights the code below the cursor. Cause: System ignores suffix context or uses left-to-right continuation where infill is required. Fix: Preserve suffix context and use the checkpoint's supported infill interface; verify that the generated span fits both boundaries.

• Symptom: Local scope is empty while the developer is mid-token, so every request ships the whole file. Cause: The client waits for a valid AST and gives up on incomplete buffers. Fix: Recover reliable syntax-node ranges from incomplete input. A backward scan is only a context hint and can't establish enclosing scope.

• Symptom: Model quality looks strong in offline code benchmarks but users still dislike the product. Cause: Benchmarks miss stale-response behavior, latency tails, abstention policy, and IDE interaction friction. Fix: Pair offline evals with online product metrics and real editor A/B tests.

What to ship with

Code completion moves part of the work from typing to reviewing suggestions. A useful system gathers bounded context, reuses permitted computation, drops stale output, and measures whether retained edits justify the latency and the data you touched.

Those same isolation and accounting checks become tenant-scoped in the next capstone, where many products share one GPU platform.