Code Completion System

Content moderation designed a low-latency safety pipeline that decides whether content can enter the product. Code completion reuses those same production muscles, but the failure is visible at the cursor. A developer types is_allowed =, continues with graph., and then receives a suggestion built for the older buffer. If that stale text replaces the newer line, the system has corrupted the edit. Even a correct suggestion feels broken when it arrives after the developer has moved on.

A code completion system predicts useful edits while a developer is still typing. GitHub Copilot's product family splits that job into dimmed ghost text at the cursor and next-edit suggestions that also predict where the next change should land.^{[1]Reference 1GitHub Copilot code suggestions in your IDEhttps://docs.github.com/en/copilot/concepts/completions/code-suggestions} Remote suggestions can use fill-in-the-middle (FIM), which conditions the missing code on both the text before the cursor and the text after it.

Coding agents that plan multi-file patches sit a layer above and get a different latency budget. In the design scenario below, the inline path has a 200 ms p95 service-level objective (SLO). A real product has to set and validate its own objective from user research and telemetry.

Before tracing the path, predict what should happen to an exact member lookup after graph. and to an open-ended assignment after is_allowed =. One should stay local; the other may justify remote generation. The figure makes that split, and its freshness gate, visible:

You've already seen how Transformers predict the next token and how the KV cache avoids recomputing shared work. This chapter puts those ideas on a keystroke path: the client may notice every edit, but it only sends qualified requests and only displays fresh results.

What makes code completion harder than a normal chat product?

Answer

The user keeps editing while inference is in flight. The product must choose when to request a suggestion, use code before and after the cursor for infill, meet a measured latency objective, and suppress stale responses.

One cursor, five lanes

The running file is validate_token_scope. The cursor sits in a half-typed authorization helper. Treat it as one live buffer, not a museum of IDE history. Before naming the lanes, ask which one should answer PolicyGraph, which one should answer graph., and which one can invent the body of is_allowed =:

Lexical and semantic first

A prefix index can finish PolicyGraph from the import line. After graph., the language server should win. It knows the receiver type and the legal members. That's parsing, a symbol table, and type inference, not a model call.

Inline generation only when the type system is stuck

The empty assignment is_allowed = is the generative case. No symbol table can invent the policy check that has to produce is_allowed for the return below. That's the fill-in-the-middle path the 200 ms budget is for.

Agents stay off the keystroke clock

A coding agent can research the repo, run tests, and edit several files.^{[2]Reference 2GitHub Copilot cloud agenthttps://docs.github.com/en/copilot/concepts/agents/cloud-agent/about-cloud-agent} That's a different product loop. Ghost text still has to gather a small prompt, generate a candidate, and stay under the 200 ms p95 clock.

Why should a design separate inline completion from coding agents?

Answer

Inline completion optimizes keystroke latency and local context. Coding agents optimize multi-step repository changes, tool use, tests, and review. They share model and context foundations, but their latency budgets and product loops differ.

The keystroke budget

An inline suggestion on is_allowed = competes with the next keystroke. If it arrives too late, the developer has already typed past it. If it's irrelevant, it becomes visual noise.

Those two failure modes set the constraints:

• Latency: For this scenario, keep single-line inline completion under 200 ms at p95. Give multi-line suggestions a separately measured budget.
• Context: Use open files, imports, function signatures, and nearby project structure, not the current file alone.
• Quality: Track acceptance and accepted-and-retained characters. The exact target varies by language, editor UX, and how aggressively the client shows suggestions.
• Scale: Large global developer fleets with diurnal spikes, so GPU utilization and cancel rate both matter.
• Privacy: Define what code may leave the client, what is retained, whether training use is disabled by default, and how tenants are isolated.

This is more than a chat endpoint with a code prompt. The path from the edit listener to the inference engine needs a measured time-to-first-token (TTFT): the delay from request submission until the first response token arrives.

One practical way to reason about the budget is to split it across stages: editor + network overhead, context assembly, first-token inference, and UI rendering. The exact numbers vary by region and model size, but every stage is on the clock. If the prompt contains a long prefix, which row would you expect to grow first: the network or first-token inference? Keep that prediction in mind while reading the scenario allocation below.

For the scenario below, those stage budgets fit under a 200 ms p95 objective. They are allocation guardrails, not a measured fleet percentile. Multi-line suggestions can have a different objective, but they still need cancellation and freshness checks.

Use an executable budget check instead of treating a latency target as a promise. This small calculation fails the candidate path when any stage pushes total latency over the scenario objective:

If a completion arrives after the user typed three more characters, what should the client do?

Answer

Treat it as stale. Abort the request if possible and gate UI rendering by request ID so old suggestions never overwrite the user's newer editor state.

---

Who owns the is_allowed = keystroke

Three pieces share that keystroke: the IDE (Integrated Development Environment) extension, a context engine, and an inference server.

The extension owns keystrokes, open tabs, and cursor position. The context engine, which can run on the client or the gateway, picks the snippets that actually fit the live prompt budget. The Language Server Protocol (LSP) is the usual interface between an editor and a language server, so the same semantic engine can power completion, go-to-definition, and diagnostics across editors.^{[3]Reference 3Language Server Protocolhttps://microsoft.github.io/language-server-protocol/}

Now make the parser answer a smaller question: after the unfinished is_allowed =, which function and imports surround the cursor? It should return useful scope even when a batch compiler would reject the buffer. Incremental parsers such as Tree-sitter still return a concrete syntax tree with ERROR and MISSING nodes, which is enough to recover the enclosing function, locals, and imports on every keystroke. Language servers then supply typed members. The LLM supplies longer infill. The client merges or gates those lanes by cursor position.

That hybrid is the product, not an optimization. After a dot, semantic candidates from the language server are usually faster and more accurate than free-form generation. The model earns GPU time on longer spans, comment-to-code, and cases the type system doesn't capture.

The inference server hosts the model and uses continuous batching so many short, cancellable requests can share resident weights.

When should deterministic semantic completion beat the LLM?

Answer

After exact trigger characters like ., when the language server knows the receiver type and valid members. It's faster, more precise, works offline, and avoids spending GPU cycles on a deterministic lookup.

Local fallback lane

Not every keystroke should take the remote generation path. Keep a deterministic lane for the cheap, high-confidence cases:

• Local symbols and imported APIs: Exact member completion from the parser or language server.
• Prefix indexes or tries: Keyword and snippet lookup with predictable latency.
• Fuzzy matching: Edit-distance recovery for small typos like pritn instead of print.

That lane does two jobs: sub-50 ms suggestions for exact matches, and a fallback when the network is slow, the model abstains, or the workspace is offline.

The smallest useful parse is "which function am I in?" You don't need Tree-sitter to see why a backward scan of the buffer beats sending the whole file. This helper walks up from the cursor line and returns the nearest def header:

A production client replaces that scan with an incremental concrete syntax tree, then asks the language server for types. The idea is the same: extract local scope from a broken buffer before you spend tokens on remote infill.

Why is a local fallback lane a product requirement rather than an optimization?

Answer

It preserves flow when remote inference is slow, blocked by policy, unavailable offline, or unnecessary for exact symbols. A completion product that goes blank during every network hiccup feels unreliable.

The client keeps lightweight parsing and exact-member lookup on-device, so only qualified prompts reach GPUs. The architecture figure above is that fork: local lane, remote lane, then a freshness gate.

---

Context gathering and management

The model needs enough context to make useful suggestions, but the live prompt budget is deliberately kept small because long prefills destroy latency. Even if the base model advertises a much larger context window, you can't stuff the whole repository into the keystroke path. The design question is not "how much can fit?" but "which tokens can change the next edit?"

Context priority strategy

Rank context by how likely it is to change the next few tokens, then fill a small budget (8k tokens in this scenario) from the top of the table down until the window is full. Before looking at the ranking, predict what you would keep when only 8k tokens remain: the code around the cursor, a distant file with a matching filename, or an unrelated open tab. The order below turns that intuition into a policy:

Fill-in-the-middle (FIM)

Standard causal language models predict the next token based only on the past (left-to-right). In coding, you often insert code in the middle of a file. Pause before naming the format: if the suffix contains return is_allowed, what can a prefix-only model not see? It might generate valid code that conflicts with the closing braces or logic below.

FIM context: A next-token prompt exposes only code before the cursor. FIM also exposes code after the cursor before generation, so the completion can match a return type, closing delimiter, later variable use, or test expectation that appears below the edit.

FIM reorders the prompt so the model sees the suffix before generating the middle. Marker strings differ by tokenizer and model; the notation below names their roles rather than defining a universal API:

$P_{\text{FIM}} = \texttt{<PRE>} + x_{\text{prefix}} + \texttt{<SUF>} + x_{\text{suffix}} + \texttt{<MID>}$

Building the FIM prompt

Build the input in FIM order: prefix marker + code before cursor + suffix marker + code after cursor + middle marker. Then the model generates the missing middle section.

Use a concrete example: you're editing an authorization helper and your cursor sits inside an empty function body.

Prefix (code before cursor)

Suffix (code after cursor)

FIM prompt sent to the model

The model now generates the middle section, using both the docstring above and the return is_allowed below to infer that it should write a policy-scope check, not an unrelated parser. Without the suffix, it could generate code that never produces is_allowed, leaving the following line broken.

Before wiring up a real tokenizer, test the transformation itself. A serving adapter would replace these readable markers with the exact sentinel tokens required by its selected FIM-capable model.

FIM gives a decoder-only model suffix information without changing left-to-right decoding. The model only understands that prompt if it was trained for it.

During pre-training, a subset of examples is transformed:

1. Split a sequence at two boundaries into (prefix, middle, suffix).
2. Reorder to (prefix, suffix, middle) with sentinel tokens between segments.
3. Train the model to predict middle given (prefix, suffix).

The two common inference formats are:

• PSM (Prefix-Suffix-Middle): <PRE>prefix<SUF>suffix<MID>middle. This is the format used in the prompt above, and the one Qwen2.5-Coder documents for file-level completion.^{[4]Reference 4Qwen2.5-Coder Technical Reporthttps://arxiv.org/abs/2409.12186}
• SPM (Suffix-Prefix-Middle): <SUF>suffix<PRE>prefix<MID>middle. Prefix and generated middle form one contiguous span.

Bavarian et al. call the surprising result "FIM-for-free": jointly training both formats, with FIM rates up to 90% in their tests, added infilling without costing ordinary left-to-right generation on their benchmarks.^{[5]Reference 5Efficient Training of Language Models to Fill in the Middle.https://arxiv.org/abs/2207.14255} They recommend character-level splits so the cursor can land mid-token. For mid-token robustness they found PSM inference can beat the particular SPM format they tested; the training mix is the bigger lever. Teams still tune the mix, because too much infill-only data can shift plain continuation.

Why does fill-in-the-middle matter more for IDE completion than for chat?

Answer

Developers usually edit inside existing files. For an insertion that must fit existing code, the model needs suffix context such as closing braces, return statements, or following tests.

Repository-level context

Large repos need more than simple file buffering. Retrieval still has one job: find definitions that can change the next few tokens without turning context assembly into a second latency problem. Imagine scope_validator.py imports PolicyGraph and ScopeRule, while metrics.py contains only counters. Which file should enter first? Two lightweight retrieval methods run fast enough to stay inside the keystroke budget.

1. Jaccard Similarity: Calculate the intersection of unique tokens (variable names, imports) between the current file and other open files. High overlap means high relevance. For example, if the active file scope_validator.py imports PolicyGraph, ScopeRule, and TokenClaims, and scope_rules.py shares two of those three names, its Jaccard score is 2/4 = 0.5 (high enough to pull in a few symbol definitions from it).
2. BM25 / Sparse Retrieval: A lightweight keyword search over the local repo index to find defining files for classes used in the current buffer.^{[6]Reference 6The Probabilistic Relevance Framework: BM25 and Beyond.https://doi.org/10.1561/1500000019}

Dense vector retrieval usually stays off the hottest keystroke path unless it's cached or precomputed. Lookup, reranking, and prompt assembly together are hard to keep inside a 10-40 ms context budget. Iterative retrieve-then-generate loops that use a first draft to fetch better snippets belong on slower multi-line or chat routes, not on every keystroke. Pre-build a small symbol graph at editor startup rather than scanning the repository on every edit.

This overlap check is small enough to run on every keystroke. It keeps scope_rules.py and policy_graph.py because they share PolicyGraph, ScopeRule, or TokenClaims. metrics.py stays out of the live prompt:

The score is a retrieval signal, not proof that a file belongs in the prompt. If a known definition repeatedly misses the threshold, inspect symbol extraction and index freshness before increasing the context window.

Why is "send the whole repository" the wrong context strategy?

Answer

Long prefills break latency and irrelevant files can distract the model. Rank context by cursor proximity, suffix, imports, recent edits, and lightweight symbol overlap before spending tokens.

---

Model serving for low latency

The budget fight is TTFT. Decode of a short suggestion is cheap compared with prefilling thousands of context tokens. Before choosing an optimization, predict what a draft model can safely save: if it proposes five tokens and the target rejects token five, should tokens one through four still count?

Speculative decoding

Speculative decoding uses a smaller, faster draft model to propose the next $K$ tokens, then a larger target model verifies them in one parallel pass.^{[7]Reference 7Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192} Serving stacks expose this as an operational feature with workload-specific caveats, so it still needs acceptance-rate measurement before rollout.^{[8]Reference 8Speculative Decodinghttps://docs.vllm.ai/en/latest/features/speculative_decoding/}

If the target agrees, you emit several tokens after one target pass. If the draft misses, the target supplies the correction and you continue from there.

Take the running helper. After is_allowed =, a draft might emit graph . allows ( user. The target accepts the first four tokens and samples request from the residual. That's four useful tokens for one target verification pass, plus cheap draft work.

Suppose the draft is about 5x faster than the target and proposes 5 tokens:

• 4 of 5 accepted: 5 draft steps cost about one target step, plus one verify pass, so about 4 tokens for 2 target-step equivalents, near a 2x speedup. Exact gain depends on draft cost and accepted length.
• 1 of 5 accepted: you keep that token and throw the rest away. Net speedup is small or slightly negative once draft overhead is counted.
• First token rejected: you fall back to ordinary single-token generation for that step.

Code with repetitive imports or boilerplate can raise acceptance. Treat that as a rollout hypothesis, not a property of all languages: measure accepted draft length and end-to-end latency by language and suggestion type before enabling it broadly.^{[7]Reference 7Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

The sketch treats target verification as roughly one target step for the whole draft. Under continuous batching, verifying many draft tokens can steal decode slots from neighbors, raise their TTFT, and erase the headline speedup. Re-measure with production batch occupancy; see speculative decoding for that interaction.

The rollout gate can be expressed as a small policy. Here, Python boilerplate improves latency enough to enable speculation, while a low-acceptance configuration remains on ordinary decoding:

Your draft model proposes 5 tokens and the target accepts 3. Did speculative decoding fail?

Answer

No. You accepted three tokens with roughly one target-model verification step plus cheap draft work. It only fails to help when acceptance is consistently low enough that draft overhead outweighs saved target steps.

KV cache reuse (prefix caching)

Developers often type, pause, and type again in the same file. The file's header (imports, class definitions, and previously written functions) remains constant across these rapid sequential interactions. Ask what a cache hit can remove before reading the timing example: prefill work, decode work, or both?

Instead of prefilling the same stable file header on every request, the server can cache Key-Value (KV) blocks for a shared prefix. When the next prompt begins with the same cacheable blocks under the same tenant and model policy, it reuses those blocks and prefills only the uncached delta. Automatic prefix caching does not make generation of new output tokens cheaper; it removes duplicate prefill work for reused input context.^{[9]Reference 9Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

In this scenario a 1,200-token file header is stable and the cursor-adjacent delta is 80 tokens. A miss still spends 72 ms on prefill plus 24 ms of decode. A hit on the same tenant, tokenizer, and shard drops prefill to 11 ms and leaves decode at 24 ms.

Those savings depend on stable prefixes, block matching, routing affinity, and isolation. A long header with a small cursor-adjacent delta is a good candidate. An early edit or a cross-tenant request is a miss by design.

Use a cache key that enforces isolation as well as affinity. The example reuses a stable header for the same tenant and model, but never treats another tenant's identical text as a hit:

This can eliminate repeated prefill work on sequential edits when the prefix stays stable and the reuse boundary is valid.

Why is prefix caching especially useful for a developer typing in one file?

Answer

Most consecutive requests share imports, class definitions, and earlier functions. Reusing cached KV blocks avoids recomputing the same prefix on every keystroke.

Quantization

Code models may tolerate post-training quantization, but you still have to re-check acceptance and retained-edit metrics after compression. Relative to FP16 weights, INT8 and INT4 weight storage can cut model-memory traffic by about 2x and 4x before format overhead.^{[10]Reference 10GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformershttps://arxiv.org/abs/2210.17323[11]Reference 11GPTQhttps://huggingface.co/docs/transformers/main/en/quantization/gptq}

Token-by-token decode is often memory-bandwidth bound, so weight compression can raise throughput and sometimes help latency. TTFT can still be dominated by queueing, context length, and prefill. Profile those phases separately before you claim a win. Compression can be the difference between serving a mid-sized coder model on one 24 GB to 48 GB GPU and needing model parallelism, depending on the KV-cache budget.

GPTQ (a post-training method for generative pretrained transformers) compresses off-the-shelf weights without a full fine-tune. That's useful when you need a FIM model to fit a specific GPU, not when you want a new capability.

What metric should you re-check after quantizing a code model?

Answer

Don't check throughput alone. Re-check acceptance rate, accepted-and-retained characters, syntax/error regressions, and latency percentiles by language and suggestion type.

---

Debounce r and o; fire on .

Sending remote work for every edit of is_allowed floods the GPU with requests that will be stale before they return. The client has to decide when to request, when to wait, and when to cancel. Imagine requests for r, o, and . crossing in flight. Predict which response the editor may paint before following the request IDs through the diagram:

Debouncing and cancellation

Use a dynamic debounce:

• 0 ms delay on trigger characters such as ., (, and \n.
• 150 ms delay on ordinary typing.

The RequestManager below is that policy in code. Trigger characters fire immediately. Ordinary characters cancel any pending request and reschedule with a 150 ms delay. Each request gets a monotonically increasing ID so a late response can't paint over newer editor state:

In a production editor, you usually combine both layers: abort the HTTP request when possible, and still guard UI updates with request IDs in case the server races or ignores the cancellation.

Why do you need both HTTP cancellation and request-ID gating?

Answer

Cancellation saves server and network work when it succeeds. Request-ID gating protects the editor even when the server races, ignores cancellation, or returns an old response after a newer keystroke.

Ghost text, not a second product

The inline surface is ghost text: a gray suggestion at the cursor. If the next character matches the suggestion prefix, the client can keep the remainder visible. Tab inserts what's left.^{[12]Reference 12Programmatic Language Features: Show Inline Completionshttps://code.visualstudio.com/api/language-extensions/programmatic-language-features} A comment such as // validate token scopes against the policy graph is still an inline completion, just with a natural-language prefix.

Repo-wide questions and test generation belong on the slower agent path, not on is_allowed =. Mixing them into the keystroke SLO is how teams accidentally build a chat product with ghost-text cosmetics.

---

Did is_allowed = actually help?

Unlike chat, completion gives a click: the developer accepts the suggestion or they don't. That click isn't enough. A four-character accept that gets deleted still counts as a click. Suppose a new model raises acceptance while developers delete more generated code. Which release improved the product? Hold that question while the figure separates inserted text from retained text.

A useful scoreboard measures how often suggestions appear, how often they're inserted, how much of that inserted text survives, and whether the path still meets its latency objective. Split those metrics by language, suggestion class, and traffic period or a model regression hides inside an aggregate.

Offline evaluation on benchmarks like HumanEval^{[14]Reference 14Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} is useful for catching model regressions, but it doesn't measure editor timing, shown-rate policy, or accepted edits that users later undo. An online experiment is needed to determine whether a model or context heuristic improves the product experience.

Optimizing solely for acceptance rate can lead to a model that only suggests short, obvious tokens like closing parens because they are safe. You need value metrics such as retained characters, not click-through alone.

The metric computation should retain that distinction. In this example, three accepted suggestions become only one meaningfully retained edit:

Why is acceptance rate easy to game?

Answer

A model can suggest tiny safe completions that users accept often but that save little work. Accepted-and-retained characters and shown rate better capture useful assistance.

Metrics matter operationally when they are tied to a release slice. If p95 rises while retention is flat, inspect queueing, context length, and cache hits, then roll back the serving or routing change. If p95 holds while retention or syntax checks fall, roll back the model or context change. Keep the last known-good route per language and suggestion class so one bad slice doesn't force a fleet-wide guess.

---

Routing by suggestion size

A more capable model can write a better function body and still lose if it arrives after the developer has moved on. A fast model that emits the wrong API also fails.

Keep the keystroke path on a smaller FIM-tuned model. Reserve slower, more capable models for multi-line bodies, next-edit suggestions, and chat. Qwen2.5-Coder is a useful existence proof: code-focused pre-training plus an explicit FIM objective, including repo-level markers, lets smaller coder models compete on completion benchmarks without putting an agent-scale model on every keystroke.^{[4]Reference 4Qwen2.5-Coder Technical Reporthttps://arxiv.org/abs/2409.12186} Distillation is the other common way to move that capability into a cheaper draft or inline model.

Don't assume parameter count predicts editor usefulness. Measure TTFT, acceptance, and retained characters on the actual trigger mix.

How much to generate is a separate routing decision. Before assigning a model, predict what the developer is asking for after . versus after a function signature. A single extra token after . wants exactness in 60 ms. An empty function body can wait 500 ms for a structured block:

A lightweight client classifier can pick the request class before sending remote work. Multi-line completions can use a more capable model; exact member completion stays local.

Why route single-token and multi-line completions differently?

Answer

They have different budgets and quality needs. Single-token completions need ultra-low latency and exactness; multi-line blocks can spend more time for structure, tests, and style consistency.

---

The fleet behind one keystroke

Code completion is a high-churn GPU workload: many requests die while the developer is still typing is_allowed. Provisioning has to protect the 200 ms objective during spikes, not just average tokens per second.

GPU fleet management and load balancing

Route for latency, cache affinity, and isolation at the same time:

• Prefix-aware routing: Requests that share a long prompt prefix should hit the same GPU instance. If someone is editing src/auth/handler.py, the next keystroke in that file should see the warm KV blocks.
• Geographic routing: Prefer an approved nearby region when residency and capacity allow. A 50 ms network round trip already consumes one quarter of the 200 ms objective.
• Model tiering: Smaller models for single-line completions, larger models for multi-line bodies.

Fleet sizing sketch

Count tokens, not "editors per GPU." Before calculating, estimate the surviving prefill load for 5,000 active editors. The cancellation rate and prefix-hit rate both change the answer, so one labeled peak-hour scenario is:

1. 5,000 editors actively typing in a region, not 50,000 idle sessions.
2. After debounce, about 0.4 remote requests per typing editor per second → 2,000 raw req/s.
3. 60% cancelled before GPU admission → 800 requests that still need serving work. A cancel after admission still burns partial GPU time and belongs in utilization.
4. Mean prompt 2,048 tokens. 50% prefix-cache hits that skip 75% of prefill.

Effective prefill rate is then 800 × 2048 × (0.50 × 0.25 + 0.50 × 1.0) = 1,024,000 tokens/s. If one GPU sustains about 20,000 prefill tokens/s at this mix, you need ceil(1,024,000 / 20,000) = 52 busy GPUs before headroom, plus shards for residency and prefix affinity. This is an arithmetic envelope, not a hardware promise. Measure tok/s, cancel rate, hit rate, queueing, and p95 on your traffic before sizing a fleet.

Continuous batching

Waiting to fill a fixed batch can add queue delay before first-token work begins. Continuous batching allows requests to join and leave an active serving schedule. vLLM^{[15]Reference 15Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} supports optimized scheduling and uses PagedAttention to manage KV-cache storage in blocks. TGI also documents continuous batching, although Hugging Face now labels TGI maintenance mode and recommends newer engines such as vLLM or SGLang for future work.^{[16]Reference 16Text Generation Inference.https://huggingface.co/docs/text-generation-inference/index} More batching isn't automatically faster. Ask whether its saved GPU work outweighs added queueing at the inline p95. Tune batching against TTFT and throughput because higher utilization can still harm inline latency when queues grow.

Cost economics

At scale, code completion is expensive because the IDE emits a steady stream of short-lived requests while the user is typing. The key design question isn't just cost per token. It's cost per useful suggestion.

• High churn: Many requests are cancelled before the user ever sees the output.
• Prefill-heavy workload: Much of the cost sits in reading context, not in generating long responses.
• Useful north-star metric: Track cost per accepted-and-retained suggestion, not cost per request alone.

Prefix caching, request cancellation, and better abstention policies improve both cost and user experience at the same time.

What is a better cost metric than cost per request?

Answer

Cost per accepted-and-retained suggestion. Completion systems generate many cancelled or ignored requests, so request volume alone overstates useful value.

---

What of this buffer may leave the laptop

validate_token_scope may sit next to proprietary source, configuration, and credentials accidentally present in the buffer. The product has to say what may be uploaded, logged, or retained before the first remote call. Before trusting TLS or a cache setting, ask whether the policy still holds in prompt logs, GPU batches, and recovery paths.

A useful design begins with explicit contracts: which buffers may be uploaded, whether prompts can be logged or used for training, how long operational data is retained, and which controls prevent cross-tenant reuse.

Data isolation

For an enterprise configuration that forbids code reuse across organizations, enforce tenant isolation across request ingestion, telemetry, and caches:

• No training on user code by default. Model weights are frozen at deployment. User code is processed transiently for inference only.
• Prompt data retention: Define explicit, minimal retention policies. Enterprises often require no raw-code retention or short, auditable windows for operational logs.
• Tenant isolation: Prevent one organization's code context from leaking through another organization's cache, logs, or outputs. Shared GPU batches can be acceptable only when the runtime preserves per-request state boundaries and observability keeps tenant scopes intact. Partition batches where policy or implementation can't prove that boundary.

The isolation contract should be executable. A logging policy can retain safe metadata for latency debugging while dropping raw source by default:

PII redaction

For enterprise usage, client-side redaction can reduce the chance that recognized secrets leave the developer's machine. Regular-expression and entropy-based scanners can catch some API keys, tokens, and passwords, while policy-based filters can mask selected identifiers before prompt construction. Scanners are incomplete, so redaction complements upload controls, restricted logging, access control, and incident response.

If a policy promises that matching secret patterns won't be uploaded, replacement must run before upload. The system replaces matched strings with placeholders such as <API_KEY> before constructing the remote prompt.

When the server returns a generated completion, the client should only reinsert masked values if the placeholder maps to a known local value. Otherwise it should keep the placeholder visible and require an explicit user edit. That prevents the model from inventing a secret-looking string and having the client silently treat it as real.

Why must secret redaction happen before prompt upload?

Answer

When policy forbids uploading known secret patterns, scanning before upload prevents matched values from entering prompt logs, caches, batches, or model context. It's one control, not proof that every secret is caught.

On-premises deployment

Some enterprise policies prohibit sending source code to a shared external service. A product serving those customers may need private Virtual Private Cloud (VPC), self-hosted, or offline deployment options.

• Self-hosted models: Deploy suitable code models inside customer-controlled infrastructure. This changes trust boundaries, but still requires identity, network, logging, and supply-chain controls.
• Air-gapped environments: Support fully offline operation for classified environments. The context engine, model, and inference server all run locally.

Public-code matching and the mastery gap

A completion can be locally useful and still be the wrong artifact to insert. Copilot-style products can check suggestions against publicly available code and then discard a match or attach a reference, depending on policy.^{[1]Reference 1GitHub Copilot code suggestions in your IDEhttps://docs.github.com/en/copilot/concepts/completions/code-suggestions} That's one control for training-data regurgitation, not a license to skip review.

The other failure is quieter. If developers accept ghost text they don't understand, debugging and design instincts weaken. The product still helps with boilerplate, unfamiliar APIs, and lowering the syntax tax. It also hallucinates libraries, emits deprecated calls, and can suggest insecure snippets. Reward retained useful edits, not blind accepts.

What ethical failure can happen even when completions are technically correct?

Answer

Developers may accept code they don't understand, miss security problems, or stop building debugging instincts. Product telemetry should reward retained useful edits, not blind acceptance.

---

Common pitfalls

Treat these as diagnoses, not a release checklist. Start with the first user-visible symptom, isolate the stage or state that can cause it, change one lever, and compare the next measured slice. That sequence keeps a model-quality complaint from turning into blind serving work.

• Symptom: Suggestions appear after the user already typed past them. Cause: No real cancellation path, or UI trusts arrival order instead of request IDs. Fix: Abort in-flight work when possible and gate rendering on newest request ID.

• Symptom: Completions are often exact but still feel unhelpful. Cause: System optimizes for raw acceptance with tiny safe suggestions. Fix: Track accepted-and-retained characters and shown rate, not acceptance alone.

• Symptom: Member completion is slower and less accurate after . than IDE autocomplete used to be. Cause: Every keystroke is routed to the LLM instead of keeping semantic lane for deterministic cases. Fix: Let parser or language server own exact symbol completion and reserve GPU work for open-ended spans.

• Symptom: Prefix caching hit rate stays low even though users edit the same file repeatedly. Cause: Routing breaks shard affinity, so matching prefixes miss the cached KV blocks. Fix: Add prefix-aware routing keyed by tenant, model, and stable prompt prefix.

• Symptom: Inserted code fights the code below the cursor. Cause: System ignores suffix context or uses left-to-right continuation where infill is required. Fix: Use FIM prompt formatting and FIM-trained models for in-file edits.

• Symptom: Local scope is empty while the developer is mid-token, so every request ships the whole file. Cause: The client waits for a valid AST and gives up on incomplete buffers. Fix: Use an incremental concrete syntax tree (or a cheaper backward scan) that still returns the enclosing function on broken input.

• Symptom: Model quality looks strong in offline code benchmarks but users still dislike the product. Cause: Benchmarks miss stale-response behavior, latency tails, abstention policy, and IDE interaction friction. Fix: Pair offline evals with online product metrics and real editor A/B tests.

---

What to ship with

Code completion moves part of the work from typing to reviewing suggestions. A useful system gathers bounded context, reuses permitted computation, drops stale output, and measures whether retained edits justify the latency and the data you touched.

Those same isolation and accounting checks become tenant-scoped in the next capstone, where many products share one GPU platform.