Multi-Tenant LLM Platform

Code completion gave you a single high-frequency product surface: one developer, one editor context, one low-latency serving path. A multi-tenant large language model (LLM) platform generalizes that serving path into shared infrastructure where many tenants, adapters, quotas, and privacy boundaries coexist on the same GPU fleet.

A multi-tenant LLM platform shares expensive serving infrastructure while enforcing tenant-scoped state, scheduler policy, and measurable latency objectives. This design chapter covers routing, quotas, batching, data boundaries, and cost control.

A shared AI platform for developer workspaces serves one hundred teams. They use it to answer internal-doc questions, summarize incidents, and draft migration notes. Each workspace requires authorization boundaries around private docs, prompts, adapters, and usage records. Your job is to share GPU capacity while keeping every stateful path scoped to the authorized tenant.

This is the multi-tenant LLM serving problem. One concrete request moves through a shared platform below, layer by layer, so you can see how the platform enforces tenant scopes, schedules shared work, and measures latency.

Start with three ideas from earlier in the curriculum. First, an LLM generates text one token at a time. Second, to avoid recomputing the entire prompt on every single token, the model stores intermediate results in a structure called the KV cache. Third, batching lets the GPU apply resident model weights across multiple requests in shared serving steps, amortizing work across users. The design below builds on all three.

What is the central tension in a multi-tenant LLM platform?

Answer

The platform shares selected capacity to reduce duplicated cost, while tenant-scoped state, scheduler policy, measured latency objectives, and metering keep shared execution accountable.

Why shared capacity needs hard boundaries

Consider a design scenario with a dense 72-billion-parameter model stored in FP16 (16-bit floating point). Weight storage alone is about 144 GB in decimal units. One NVIDIA H100 SXM configuration has 80 GB of HBM3 memory.^{[1]Reference 1H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/} In this scenario, one copy of the model weights exceeds one such GPU's memory before adding KV cache or serving overhead.

If each of one hundred tenants had a separate copy of those weights, weight memory alone would be 14.4 TB. Shared base weights can avoid that duplication, but they don't automatically isolate prompts, retrieved documents, adapters, caches, or billing state.

Make the scenario calculation runnable before discussing schedulers:

Sharing introduces three concrete engineering tensions:

1. Compute contention. All tenants want the GPU's CUDA cores during deploy windows, incidents, and review bursts.
2. Memory contention. Every active conversation consumes KV cache memory. A tenant with a long design-review transcript can evict another tenant's chat if limits aren't enforced.
3. Weight customization. Tenants want different behaviors. One needs terse API-reference answers; another needs incident-triage summaries. Loading a full model copy per customization wastes capacity, so we need scoped lightweight adapters or separate pools where required.

The rest of the article solves these three tensions in order.

Why is "one model copy per tenant" usually impractical for a large serving fleet?

Answer

Large dense models can require more memory than one GPU just for weights. Sharing base weights reduces duplication, but each shared-state mechanism still needs an explicit authorization and accounting boundary.

How we pack requests together: continuous batching

When a GPU processes a batch, one serving step applies the resident model weights across multiple requests. The simplest approach is static batching: collect eight requests, run them together, and wait until every single one finishes before starting a new batch. This is easy to implement but wasteful. If Tenant A's API-reference answer generates only 10 output tokens while Tenant B's incident retrospective generates 500 tokens, the GPU slot sits idle for Tenant A while Tenant B finishes the remaining 490 tokens.

Continuous batching (also called in-flight batching, described in the Orca paper^{[2]Reference 2Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu}) replaces completed requests with queued work at iteration boundaries. When Tenant A reaches its EOS (End of Sequence) token, the scheduler can admit another request without waiting for Tenant B to finish.

The throughput gain depends on prompt lengths, decode lengths, admission policy, and scheduler overhead. The useful principle is that a finished request need not occupy a decode slot.

In a multi-tenant environment, the scheduler also has to respect priority and fairness. A high-tier tenant may have tighter latency SLOs. A tenant-aware continuous batcher therefore balances throughput (packing as many tokens as possible) against measured latency objectives for prioritized tenants. The rate-limiting and preemption sections below show how that works.

Static batching is a charter bus that waits until every passenger reaches their destination before returning to the depot. Continuous batching lets an empty seat be offered to the queue at the next scheduled stop.

This miniature schedule keeps a long request active while replacing a completed short request:

Why does continuous batching need tenant awareness?

Answer

Pure throughput scheduling can let one tenant dominate decode slots or KV memory. Tenant-aware batching keeps the GPU full while honoring priority tiers, quotas, and latency SLOs.

How we customize behavior without duplicating weights: LoRA adapters

Tenants may need different behaviors without separate full model copies. LoRA (Low-Rank Adaptation^{[3]Reference 3LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}) learns low-rank matrices next to selected original weight layers. During inference, the base model remains fixed and the selected adapter contributes an additional projection.

Adapter size isn't one universal number: it depends on rank, target modules, model dimensions, and dtype. It's typically much smaller than a full base copy, but a platform must measure its chosen adapter footprint and decide how many adapters can remain resident alongside KV-cache budgets.

The S-LoRA (Serving Thousands of Concurrent LoRA Adapters) system^{[4]Reference 4S-LoRA: Serving Thousands of Concurrent LoRA Adapters.https://arxiv.org/abs/2311.03285} studies serving many concurrent adapters while keeping base weights shared. Adapter caching is only part of the problem; the runtime also has to execute requests with different adapters in shared serving steps. Multi-LoRA engines need runtime support that maps each request to its adapter while preserving base-model sharing.^{[5]Reference 5LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/}

Punica introduces Segmented Gather Matrix-Vector multiplication (SGMV) for batched LoRA serving and evaluates mixed-adapter overhead.^{[6]Reference 6Punica: Multi-Tenant LoRA Servinghttps://arxiv.org/abs/2310.18547} Production engines such as vLLM expose LoRA serving configuration including enabled adapters, resident adapter limits, and maximum supported rank.^{[5]Reference 5LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/} Benchmark your actual adapter mix and hardware before treating the paper result as a fleet capacity plan.

Analogy: a shared compiler backend Imagine one compiler backend that loads a small approved plugin for each workspace. The backend (the base model) stays shared. The plugin (the adapter) is small, but the runtime must still verify which tenant is authorized to use it.

The platform stores adapters in an object-storage registry such as S3 (Simple Storage Service) and loads them into GPU memory on demand. Frequently used adapters stay in an LRU cache on the GPU; rarely used ones are evicted to host RAM or disk.

What does LoRA share and what does it customize?

Answer

The frozen base model weights are shared across tenants. Small adapter matrices customize tone, domain behavior, or tenant policy without loading a separate full model per tenant.

The conceptual lifecycle below shows how the adapter manager takes a request, resolves the correct adapter, and runs inference. Multi-LoRA runtimes use adapter-aware kernels and memory management so requests with different adapters can share base-model serving steps. Adapter lookup, residency, and mixed-adapter execution still have measurable cost, so benchmark the actual mix:

Adapters are much smaller than full base weights, but loading one from object storage into GPU memory is still measurable. If Tenant A and Tenant B alternate on every request, the adapter cache thrashes and latency spikes. Profile your actual adapter reuse patterns before assuming LRU is sufficient. Some platforms pin high-tier adapters permanently and only evict best-effort ones.

Why can an adapter cache become a latency problem?

Answer

If hot tenants alternate and the GPU adapter cache is too small, every request triggers an adapter load or eviction. Pin high-service-level-agreement (SLA) adapters or schedule with adapter locality to avoid cache thrash.

How we keep conversations separate: KV cache isolation

Every token the model generates relies on the KV cache, which stores intermediate key and value vectors from earlier tokens. Without it, autoregressive decoding repeats work. In a multi-tenant system, KV allocation is also sensitive state: the runtime must not attach Tenant A's live blocks or cache entries to Tenant B's request.

The memory cost in concrete numbers

KV-cache memory per request grows with context length. For a decoder using fixed-width KV heads, a useful estimate is:

$$\begin{aligned} \text{KV memory per request} = &2 \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \\ &\times \text{seq\_len} \times \text{dtype\_bytes} \end{aligned}$$

The factor of 2 counts keys and values. The other terms are model depth, KV heads, head dimension, request context length, and bytes per element. Grouped-query attention (GQA) stores fewer KV heads than attention heads, which is why the n_kv_heads term matters.

Before you memorize the symbols, calculate a concrete scenario. Suppose a model has 80 layers, 8 KV heads, head dimension 128, a 4,000-token context, and FP16 (2 bytes per element):

$2 \times 80 \times 8 \times 128 \times 4{,}000 \times 2 = 1{,}310{,}720{,}000 \text{ bytes} \approx 1.22 \text{ GiB per request}$

If the same model family used 16 KV heads instead of 8, that doubles to about 2.44 GiB. At high concurrency, this memory can become the admission constraint before compute throughput does.

Why is KV cache both a capacity problem and a privacy boundary?

Answer

It grows with active context and can dominate VRAM. It also stores conversation-derived state, so allocation or cache-namespace bugs can violate isolation.

PagedAttention: paging for GPUs

PagedAttention (introduced in the vLLM paper^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttentionhttps://arxiv.org/abs/2309.06180}) treats KV storage in blocks rather than reserving one long contiguous chunk for each request. A block table maps each request's logical sequence to physical blocks. The example figure uses 16-token blocks to make the mapping visible; production block sizes are runtime configuration and performance choices.

In that illustration, Tenant A's 47-token conversation occupies three physical blocks and Tenant B's 31-token conversation occupies two. PagedAttention improves memory allocation efficiency; it doesn't by itself enforce tenant authorization. The serving layer must associate request ownership with block tables, invalidate released references, and implement any required clearing policy before reallocation.

HBM is the hottest KV tier, not the only possible tier

Local GPU HBM gives active decode steps their lowest transfer latency, but keeping every reusable or paused KV block there limits concurrency. A tiered design can keep active blocks in HBM, spill colder blocks to CPU DRAM, and place reusable KV in a remote cache service. Mooncake's disaggregated architecture uses CPU, DRAM, SSD, and RDMA-capable network resources to move and reuse KV cache independently of one inference worker.^{[8]Reference 8Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079}

Disaggregation changes the bottleneck rather than deleting it:

The scheduler should estimate whether the transfer time is cheaper than recomputing the prefix. A remote hit can be a loss for a short prompt or congested link. A token count alone is insufficient; admission needs byte-level KV budgets, measured transfer bandwidth, and a deadline check.

Tenant isolation follows the blocks off GPU. Remote keys must include the trust scope plus model, adapter, tokenizer, and KV layout version. The cache service must authorize reads before returning block metadata, encrypt traffic as required by the threat model, and invalidate ownership when a request or tenant is deleted. RDMA reduces CPU involvement in data movement; it doesn't provide application-level authorization by itself.

When should the scheduler restore remote KV instead of recomputing a prefix?

Answer

The scheduler should restore it only when the predicted lookup and transfer time fits the request deadline and is lower than the recomputation cost. Include network congestion, prefix size, compatible model state, and tenant authorization in the decision.

Prefix caching and the cross-tenant leak risk

Multi-tenant traffic often repeats the same system prompt, tool schema, or long retrieved prefix. Runtimes can cache those KV blocks and skip recomputing the shared prefix on later requests. SGLang introduced RadixAttention for this pattern, while vLLM's automatic prefix caching uses hashed KV blocks rather than a radix tree.^{[9]Reference 9SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104[10]Reference 10Automatic Prefix Cachinghttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/}

Prefix caching mainly lowers TTFT (Time to First Token) because it eliminates repeated prefill work. It doesn't make decode itself cheaper.

The critical rule is isolation. Reuse prefixes only inside an authorized cache namespace, such as one tenant or an explicitly public shared prompt. vLLM's prefix-caching design docs describe an optional cache salt intended to isolate cache reuse across trust groups and mitigate timing-based probing; platform routing must also keep adapter and model compatibility consistent.^{[10]Reference 10Automatic Prefix Cachinghttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/}

Enabling private-prefix reuse without a tenant or trust-group namespace is an isolation bug first, even if the original change was meant to improve performance. Scope private cache reuse by tenant or authorized trust group and compatible model, tokenizer, and adapter configuration.

What must be included in a safe prefix-cache namespace?

Answer

Tenant or explicit shared trust-group identity plus compatible base model, adapter, and tokenizer configuration. Public shared prompts need a deliberate shared namespace, not accidental reuse.

Chunked prefill for multi-tenant fairness

The prefill phase processes the input prompt to build the initial KV cache. It's compute-intensive but doesn't generate tokens. The decode phase generates output tokens one at a time. It's memory-intensive but uses less compute.

Without scheduling controls, a long prefill from one tenant can delay decode operations from others. Chunked prefill (studied in Sarathi-Serve^{[11]Reference 11Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.https://arxiv.org/abs/2308.16369}) breaks long prompts into bounded chunks so decode work from other requests can be scheduled between prefill chunks.

The chunk size is a tuning knob, not a fixed constant. It's controlled by the runtime's per-step token budget (max_num_batched_tokens in vLLM). Smaller budgets lower inter-token latency for in-flight decodes; larger budgets improve TTFT and prefill throughput. Current vLLM docs show chunked prefill enabled whenever possible in V1, example low-latency settings like 2,048 tokens, and throughput-oriented settings above 8,192 tokens, so tune it to your latency target instead of copying one number blindly.^{[11]Reference 11Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.https://arxiv.org/abs/2308.16369[12]Reference 12Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

In multi-tenant settings, chunked prefill is one useful noisy neighbor control. It still needs admission limits and a tenant-aware scheduler; chunking alone doesn't guarantee fair service.

Why does chunked prefill protect other tenants from a long prompt?

Answer

It breaks a long compute-heavy prefill into smaller slices that can interleave with decode work from other tenants, preventing one 32K prompt from blocking the whole batch.

Tenant-aware preemption

When admitted work approaches its KV budget, the scheduler may reject, queue, or preempt requests according to the published service policy. A priority tier can permit an enterprise request to displace best-effort work, but the choice must be metered and observable rather than hidden.

Current vLLM V1 docs default to RECOMPUTE rather than SWAP because recomputation has lower overhead in that architecture. A platform should still choose a recovery mode based on runtime support and measured cost.^{[12]Reference 12Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

The conceptual scheduler below demonstrates the decision logic. It sorts running requests by priority, then by KV footprint, then by tokens already generated. If a new request has higher priority than the lowest-priority running request, the scheduler preempts the victim and schedules the newcomer:

The miniature loop accounts for released KV memory before admitting the newcomer. A production scheduler must make release and allocation atomic so concurrent scheduling decisions can't over-admit work.

What should a tenant-aware preemption policy optimize for?

Answer

Free enough KV memory while minimizing service-objective impact. Prefer lower-priority tenants, large KV footprints, and requests with less generated work, then recompute or swap depending on runtime support.

Hard per-tenant limits

Scheduling alone isn't enough. The platform also enforces hard quotas on context sizes and concurrent requests based on the tenant's tier. The illustration below summarizes the isolation spectrum from shared pools to stronger runtime boundaries:

Example admission policy (numbers are scenario inputs, not universal tiers):

Derive each budget from max_concurrent × max_context × bytes_per_token, then add headroom for fragmentation, multi-node pools, and peak admission. Using the earlier ~1.22 GiB @ 4K-token estimate (~0.000305 GiB per token for that model shape):

• Enterprise peak: $50 \times 8000 \times 0.000305 \approx 122$ GiB raw KV; the 256 GB row is roughly 2× that for HBM packing waste, multi-worker share, and burst.
• Business peak: $20 \times 4000 \times 0.000305 \approx 24$ GiB raw; 80 GB is a multi-replica pool budget, not a single-request cap.
• Starter peak: $5 \times 2000 \times 0.000305 \approx 3$ GiB raw; 20 GB keeps small tenants from thrashing under noisy neighbors while still hard-capping spend.

Publish the formula and the measured bytes/token for your model; don't treat the table numbers as industry standards.

Apply those limits before scheduling GPU work:

Why are hard KV limits necessary even with fair scheduling?

Answer

Schedulers arbitrate admitted work, but a single oversized prompt can allocate too much memory before fairness helps. Hard limits reject or downsize requests before they consume shared KV blocks.

How we prevent one tenant from overwhelming the rest: rate limiting and fair queues

Rate limiting sits at the gateway, before a request ever reaches the GPU. It enforces two distinct budgets:

• Requests per minute (RPM): Controls burst traffic to protect the API gateway from connection exhaustion.
• Tokens per minute (TPM): Controls sustained throughput to protect GPU compute capacity.

The difference matters. A tenant sending one request with a 64K prompt consumes far more GPU time than a tenant sending one hundred requests with 100-token prompts, even though the first tenant uses fewer requests. RPM alone would let the 64K prompt through and monopolize the KV cache.

Distributed sliding-window enforcement

For RPM, a distributed sliding-window limiter using Redis with a Lua script gives consistent enforcement across all gateway nodes. A local in-memory limiter isn't enough because requests are load-balanced across many gateway instances.

The Lua script below removes entries older than the window, counts the remaining requests, and either allows the new request or rejects it. The key detail is using a unique sorted-set member (a request ID with a timestamp) instead of the raw timestamp alone. If two requests land in the same clock tick and you use the timestamp as both score and member, Redis collapses them into one entry and undercounts traffic:

TPM is trickier because you don't know the final output length at admission time. In practice, reserve a budget based on prompt tokens plus max_output_tokens, then reconcile the counter with actual usage when the stream finishes.

For high-throughput services, strictly synchronized Redis limits can become a bottleneck. A platform may choose bounded burst allowance or approximate local counters, but that weakens strict limit semantics and must be documented and measured.

Token admission needs reservation and reconciliation. Reserve prompt plus maximum allowed output before execution, then release unused output capacity after the stream completes:

Fairness inside the scheduler

RPM and TPM protect the gateway edge, but they don't fully solve scheduler fairness inside the serving engine. A tenant with one 64K prompt can consume far more GPU time than dozens of tenants sending short chat turns.

Inside the runtime, keep per-tenant queues and charge a virtual token budget for every admitted prefill chunk and every decode step. Then schedule by priority tier plus virtual finish time, not raw request count. That gives each tenant forward progress while still letting higher-SLA traffic buy more share.

Don't collapse rate limiting and quota management into one counter. Rate limits prevent burst and protect the system, while quotas cap total usage and protect the budget. A tenant can stay under their RPM limit and still burn through their monthly token quota in one afternoon.

Why is RPM not enough for LLM rate limiting?

Answer

One request can contain a huge prompt and request many output tokens. TPM and KV budgets protect GPU work and memory, while RPM mostly protects the gateway from bursty connection pressure.

How we keep data private: the isolation stack

Every state-bearing layer needs an authorization boundary and a testable release policy. A cross-tenant retrieval, adapter, cache, or KV access is a security incident even if the other layers behaved correctly.

The RAG relevance vs. authorization gap

Many multi-tenant platforms augment LLMs with retrieval-augmented generation (RAG). A vector database finds the most relevant documents for a query. In multi-tenancy, "relevant" doesn't mean "allowed."

Tenant X searches for "incident-retention exception policy." The vector DB might find a highly relevant private postmortem that belongs to Tenant Y, because both teams use the same reliability vocabulary. Without a hard filter, the LLM could summarize Tenant Y's confidential incident details and return them to Tenant X.

Use authorization filtering in the retrieval operation itself. An application should pass authorized scope into the database query, and tests should fail if another tenant's result can cross that boundary:

The authorization predicate is applied before candidate results leave the data layer. Post-filtering after a broad top-K query can return foreign identifiers, scores, or content to application memory and may also erase every authorized candidate.

Why must tenant filtering happen inside the vector database query?

Answer

Post-filtering can retrieve foreign document IDs, scores, or embeddings before dropping them. Authorization must be part of the retrieval predicate so disallowed evidence is never returned to the application.

Model and prompt isolation

• LoRA adapter isolation. Authorize adapter IDs against the tenant before loading encrypted artifacts. If adapters contain sensitive tenant tuning, include residency and release rules in the data contract.
• Prompt isolation. Keep raw prompts out of plaintext logs and authenticate/encrypt gateway-to-worker transport. The exact mTLS (mutual Transport Layer Security) and memory-boundary policy depends on the deployment threat model.
• Harder runtime boundaries. For workloads whose threat model rules out shared workers, use dedicated node pools, MIG (Multi-Instance GPU) partitions, or VM boundaries as evaluated controls.^{[13]Reference 13Supported GPUshttps://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-gpus.html} Kubernetes placement selects hardware; it doesn't create isolation by itself.

State sanitization

When a shared worker serves more than one tenant, state lifecycle rules matter:

• KV reference lifecycle. Remove request access to released KV blocks when a sequence completes. If the threat model requires cleared memory before cross-tenant reuse, implement and verify that clearing policy rather than assuming the allocator provides it.
• Batch construction policy. Multi-tenant batching is fine on a shared base model, but every row in the batch must keep its own tenant ID, adapter handle, KV block table, and metering context. For regulated workloads, the simpler answer is dedicated pools or MIG / VM boundaries instead of trying to harden every shared-kernel path.

PII masking as a tiered control

For regulated workloads that can tolerate redaction, requests can pass through a lightweight PII masking service (e.g., Presidio^{[14]Reference 14Presidio: Data Protection and De-identification SDK.https://github.com/microsoft/presidio}) before they hit the model router. This reduces the chance that the LLM ever sees raw credit card numbers or Social Security numbers, while still letting downstream systems map placeholders back to original values when needed.

When is PII masking helpful, and when is it insufficient?

Answer

It reduces exposure for data that can be safely replaced with placeholders. It's insufficient for regulated tenants that need hard hardware or VM isolation, strict audit controls, or full-fidelity private context.

How we attribute cost: per-tenant metering and chargeback

A shared fleet only stays profitable if you can answer one question per tenant: what did this workspace cost us, and what should we charge? Token counts alone are a weak proxy because two requests with the same token counts can consume very different GPU time depending on prompt-vs-output split, batch occupancy, preemptions, and cache hits.

A defensible metering record attaches to every request and carries: tenant_id, model and adapter version, prompt tokens, output tokens, cache-hit tokens, queue wait, prefill time, decode time, KV blocks held, preemption count, and GPU worker type. Cache-hit tokens matter because reused prefix blocks skip prefill compute. A public billing policy may choose a distinct cached-input rate, as current provider pricing documents illustrate.^{[15]Reference 15Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching[16]Reference 16Prompt caching.https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching}

For internal cost allocation rather than customer billing, the honest unit is GPU-time, not tokens. A reasonable per-request cost estimate looks like:

$\text{cost}_{\text{req}} \approx \text{gpu\_seconds} \times \text{node\_hourly\_rate} / 3600 + \text{adapter\_residency} + \text{storage}$

where gpu_seconds is the request's share of busy GPU time (prefill plus its decode steps, divided by batch occupancy so shared steps are split across co-batched tenants). Charge customers on the simpler token-and-tier dimensions, but reconcile against measured GPU-time so you can spot tenants whose traffic shape (long prompts, low batchability, adapter thrash) costs far more than their token bill suggests.

Why is per-tenant token count not enough for cost attribution?

Answer

Equal token counts can map to very different GPU time depending on prompt/output split, batch occupancy, preemptions, and cache hits. Attribute internal cost in GPU-seconds and discount cache-hit tokens, then map that to a token-and-tier price.

How the system grows: scaling, canary, and fault tolerance

A production platform should handle spikes, deploy new models with release gates, and degrade predictably during failures.

Auto-scaling on queue depth

CPU utilization alone doesn't describe LLM serving pressure. Queue depth, admitted-token backlog, inference latency, and KV-cache utilization provide signals for capacity and admission decisions. When new workers can't become ready in time, shed or downgrade eligible best-effort work instead of silently violating every tenant's objective.

Cold-starting GPU workers includes making model weights available before the pool can serve requests. Depending on cost and latency objectives, a platform may keep warm capacity, forecast predictable demand, or reject low-priority excess load while workers start.

Worked example: steady demand needs 12 replicas at 4K decode tok/s each to hold target TTFT. Measured cold-start (pull weights + warm CUDA graphs) is 8 minutes, and traffic can ramp 30% in under 10 minutes at the top of a business day. Keep a warm buffer of about $0.3 \times 12 \approx 4$ ready replicas (or scale on forecast before the ramp), and autoscale when p95 queue age exceeds, say, 200 ms for two consecutive windows instead of relying on average CPU alone. If the buffer is exhausted, shed starter-tier traffic before enterprise SLOs fail.

Model versioning and canary rollouts

Models and LoRA adapters require versioned rollout because a new artifact can degrade generation quality or latency.

In one canary rollout policy, when a tenant deploys a new adapter version (for example, v2), the router sends a controlled slice of eligible traffic to it while the rest continues on v1. Monitor quality evaluation, latency, error rates, and safety signals over a predefined observation window.

If gates pass, the router can increase exposure. If a gate fails, route new eligible requests back to v1; active streams and adapter residency still need explicit handling, so a routing change isn't a blanket zero-downtime promise.

For base model updates, the process is more complex. Unlike lightweight adapters, base models require spinning up entirely new GPU worker pools. The platform routes shadow traffic (duplicate asynchronous requests) to the new base model cluster to validate correctness and measure throughput before exposing it to real tenant traffic. Once validated, the gateway shifts live traffic to the new cluster and gracefully drains the old one.

Why are adapter canaries easier than base-model canaries?

Answer

Adapters are small and can usually be routed per tenant on an existing base model. Base-model changes require new worker pools, separate KV caches, shadow traffic, throughput validation, and drain logic.

Fault tolerance

System reliability relies on handling GPU failures and model crashes gracefully:

These mechanisms need a control plane or equivalent coordination layer to track worker readiness, orchestrate model deployments, and update routing. Readiness checks reduce routing to unavailable workers; they don't prove model quality or prevent every runtime failure.

Why does a poison request need a dead-letter queue?

Answer

Repeated CUDA OOMs or malformed inputs can trigger retry storms and consume shared capacity. A DLQ removes the repeatedly failing request after bounded retries so it no longer consumes retries.