Multi-Tenant LLM Platform

At 09:00, one hundred workspaces send requests to an internal-doc assistant. By 09:04, Tenant A's long design review has filled the KV cache, Tenant B's incident summary is waiting, and the GPU dashboard says 60% utilization. A trace then shows a worse symptom: Tenant B's retrieved context includes a private postmortem from Tenant A.

Pause on the diagnosis. Should you add GPUs because users are waiting? Not yet. Low utilization beside a growing queue points first to admission, scheduling, or memory shape. The cross-tenant context is a separate authorization failure. More capacity won't repair either boundary.

Code completion gave you one product surface: one developer, one editor context, one low-latency serving path. A multi-tenant large language model (LLM) platform puts that path on shared GPU capacity while keeping workspace identity attached to prompts, adapters, caches, retrieval, and bills.

We'll follow Tenant A's request through that platform. Three earlier ideas supply the machinery: an LLM generates one token at a time, the KV cache stores intermediate attention state, and batching lets one resident model serve several requests in shared steps.

What is the central tension in a multi-tenant LLM platform?

Answer

The platform shares selected capacity to reduce duplicated cost, while tenant-scoped state, scheduler policy, measured latency objectives, and metering keep shared execution accountable.

Before choosing an allocator or queue, define what “healthy” means for each tenant. For request $r$, measure first-token delay from arrival and each gap after the first token:

$$\mathrm{TTFT}_r = t_{\mathrm{first\ token},r} - t_{\mathrm{arrival},r}, \qquad \mathrm{ITL}_{r,i} = t_{r,i} - t_{r,i-1}$$

An illustrative service contract might require p95(TTFT) <= 800 ms and p95(ITL) <= 120 ms for an interactive tier. Those are scenario inputs, not universal targets. A request counts as goodput only when it completes successfully and meets the tier's latency gates. Queue wait, prefill, decode, adapter loading, and network transfer all belong in the trace, or a passing GPU metric can hide a failing user experience.

Now we have a decision rule for the incident: fix the term that moved the percentile, and don't call a hardware specification a measured result. The sections below attach that rule to each shared state boundary.

Why shared capacity needs hard boundaries

Start with the first admission question: can the base model fit? In this scenario, a dense 72-billion-parameter model stored in FP16 needs about 144 GB in decimal units for weights alone. NVIDIA lists 80 GB of HBM3 for one H100 SXM configuration.^{[1]Reference 1H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/} One copy therefore needs more than one such GPU before KV cache, activations, or runtime work are counted.

If one hundred tenants each received a full copy, weights alone would consume 14.4 TB. Predict what sharing changes: it removes duplicate weight storage, but it says nothing about who may read a prompt, adapter, KV block, retrieval result, or usage record.

Run the arithmetic before we choose a scheduler:

Sharing introduces three concrete engineering tensions:

1. Compute contention. All tenants want the GPU's CUDA cores during deploy windows, incidents, and review bursts.
2. Memory contention. Every active conversation consumes KV cache memory. A tenant with a long design-review transcript can evict another tenant's chat if limits aren't enforced.
3. Weight customization. Tenants want different behaviors. One needs terse API-reference answers; another needs incident-triage summaries. Loading a full model copy per customization wastes capacity, so we need scoped lightweight adapters or separate pools where required.

Why is "one model copy per tenant" usually impractical for a large serving fleet?

Answer

Large dense models can require more memory than one GPU just for weights. Sharing base weights reduces duplication, but each shared-state mechanism still needs an explicit authorization and accounting boundary.

Weight sharing solves one cost problem. It doesn't make an inference engine a tenant boundary. Serving engines such as vLLM and SGLang load weights, page KV, and run kernels. The surrounding control plane authenticates the tenant, enforces quotas, chooses an adapter, and meters GPU-time around that engine.

Predict what happens if tenant_id disappears after the gateway. A Tenant A docs request still reaches a valid worker, but its cache, adapter, or bill can now be attributed to the wrong workspace. Keep identity on every hop:

If any hop drops tenant_id, you don't have a platform. You have a shared GPU with a leak surface.

How we pack requests together: continuous batching

When a GPU processes a batch, one serving step applies resident weights across several requests. Suppose Tenant A's answer needs 10 output tokens and Tenant B's needs 500. A static batch holds both until B finishes, so A's slot has nothing useful to do for most of the run.

Predict the next step: once A emits its end-of-sequence token, should the scheduler wait for B or admit a queued request? Continuous batching, also called in-flight batching and described in the Orca paper,^{[2]Reference 2Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu} replaces completed work at iteration boundaries. The empty slot can take Tenant C while B keeps decoding.

That improves utilization only when admission and scheduling keep the mix healthy. Prompt length, output length, queue policy, and scheduler overhead still determine throughput. A finished request need not occupy a decode slot, but a tenant can still monopolize slots with long prefills or large KV allocations.

The scheduler also has to meet tiered latency objectives. If an enterprise tenant has a tighter p95 TTFT or ITL target, packing more tokens is useful only while that percentile stays inside its contract. Rate limits and preemption provide the next controls.

The toy schedule below makes the slot transition visible. Tenant A has one token left, Tenant B has three, and Tenant C is waiting. What should remain active after one iteration?

Why does continuous batching need tenant awareness?

Answer

Pure throughput scheduling can let one tenant dominate decode slots or KV memory. Tenant-aware batching keeps the GPU full while honoring priority tiers, quotas, and latency SLOs.

If you add speculative decoding to a shared pool, the draft path has to be authorized for that tenant and compatible with the target model, tokenizer, and adapter. A faster wrong token is still a wrong token. A draft that reused another tenant's prefix is an isolation bug, not a cache win.

The slot is now doing useful work for C. The next question is what C should customize without forcing a second 144 GB base copy.

How we customize behavior without duplicating weights: LoRA adapters

Tenant A wants terse API answers and Tenant B wants incident summaries. Predict the cheapest change: duplicate the 144 GB base or attach a smaller request-specific update? LoRA (Low-Rank Adaptation^{[3]Reference 3LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}) learns low-rank matrices beside selected original weight layers. The base stays fixed while the selected adapter changes the projection for that request.

An adapter isn't one universal size. Rank, target modules, model dimensions, and dtype determine its footprint. It's usually far smaller than a base copy, but the worker still has to budget adapter residency beside KV blocks. Measure that footprint for the model and ranks you actually serve.

The S-LoRA (Serving Thousands of Concurrent LoRA Adapters) system^{[4]Reference 4S-LoRA: Serving Thousands of Concurrent LoRA Adapters.https://arxiv.org/abs/2311.03285} studies how to keep many adapters available while sharing base weights. Caching alone doesn't solve the problem. A multi-LoRA runtime also needs to map each row in a batch to the right adapter while preserving the common base path.^{[5]Reference 5LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/}

Punica's Segmented Gather Matrix-Vector multiplication (SGMV) targets batched LoRA requests with mixed adapters.^{[6]Reference 6Punica: Multi-Tenant LoRA Servinghttps://arxiv.org/abs/2310.18547} vLLM exposes related controls: max_loras limits resident adapters and --max-lora-rank sizes the largest supported rank.^{[5]Reference 5LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/} Setting a very high rank cap for small adapters can reserve workspace you never use. Benchmark the actual adapter mix and hardware before turning a paper's concurrency result into a fleet plan. Runtime adapter loading (VLLM_ALLOW_RUNTIME_LORA_UPDATING) is documented as a fully trusted, isolated-environment feature, not a public tenant API.

Same 144 GB base, different adapters Tenant A loads docs-v2 for terse API answers. Tenant B loads incident-v7 for triage summaries. The frozen 72B weights stay shared. The adapter is small, but residency on the GPU still isn't a permission grant.

The platform can keep versioned adapters in an object-storage registry such as S3 (Simple Storage Service), then load an authorized one into GPU memory on demand. Frequently used adapters stay in an LRU cache; cold entries can move to host RAM or disk. Residency is a performance decision, never an authorization decision.

What does LoRA share and what does it customize?

Answer

The frozen base model weights are shared across tenants. Small adapter matrices customize tone, domain behavior, or tenant policy without loading a separate full model per tenant.

The small lifecycle below checks authorization before it looks in the cache. Predict the last three lines: Tenant C should evict the least-recently-used entry, and Tenant A's request for C's adapter should fail even when that adapter is hot. Multi-LoRA runtimes can share base-model steps across adapters, but lookup, residency, and mixed-adapter execution still have measurable cost, so benchmark the actual mix:

The output shows both decisions: LRU controls residency, while the authorization map controls use. If Tenant A and Tenant B alternate on every request, object-storage loads and evictions can thrash the adapter cache and raise TTFT. Profile reuse before assuming LRU is enough; pin high-tier adapters only when the measured latency benefit pays for their reserved memory.

Adapters customize behavior. They don't separate conversation state. That boundary is the KV budget.

Why can an adapter cache become a latency problem?

Answer

If hot tenants alternate and the GPU adapter cache is too small, every request triggers an adapter load or eviction. Pin high-service-level-agreement (SLA) adapters or schedule with adapter locality to avoid cache thrash.

How we keep conversations separate: KV cache isolation

Every generated token reads key and value vectors from earlier tokens. The KV cache keeps that work so autoregressive decoding doesn't recompute the whole history. That cache is tenant-sensitive state: a block table or cache entry owned by Tenant A must never become visible to Tenant B.

The memory cost in concrete numbers

KV-cache memory grows with context length, so use it as an admission input rather than an after-the-fact alert. For a decoder with fixed-width KV heads, a useful first-order estimate is:

$$\begin{aligned} \text{KV memory per request} = &2 \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \\ &\times \text{seq\_len} \times \text{dtype\_bytes} \end{aligned}$$

The leading 2 counts keys and values. The remaining terms are model depth, KV heads, head dimension, context length, and bytes per element. Grouped-query attention (GQA) stores fewer KV heads than attention heads, which is why n_kv_heads sets the memory slope.

Now make the estimate concrete. Suppose a model has 80 layers, 8 KV heads, head dimension 128, a 4,000-token context, and FP16 (2 bytes per element). Before calculating, predict what happens if the model uses 16 KV heads: every other term stays fixed, so the KV footprint doubles.

$2 \times 80 \times 8 \times 128 \times 4{,}000 \times 2 = 1{,}310{,}720{,}000 \text{ bytes} \approx 1.22 \text{ GiB per request}$

Sixteen KV heads doubles the result to about 2.44 GiB. At high concurrency, that per-request state can become the admission constraint before compute throughput does.

Why is KV cache both a capacity problem and a privacy boundary?

Answer

It grows with active context and can dominate VRAM. It also stores conversation-derived state, so allocation or cache-namespace bugs can violate isolation.

PagedAttention: paging for GPUs

Suppose Tenant A's 47-token conversation and Tenant B's 31-token conversation arrive together. If each request reserves one contiguous maximum-length region, holes appear as soon as their lengths diverge. PagedAttention, introduced in the vLLM paper,^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttentionhttps://arxiv.org/abs/2309.06180} instead stores KV in blocks and maps each logical sequence to physical blocks. The figure uses 16-token blocks to make that mapping visible; production block size remains a runtime and performance choice.

Tenant A occupies three physical blocks and Tenant B occupies two. Paging makes those blocks packable, but it doesn't grant access. The serving layer still owns request-to-table authorization, released-reference invalidation, and any memory-clearing policy required by its threat model.

HBM is the hottest KV tier, not the only possible tier

Keep active decode blocks in local GPU high-bandwidth memory (HBM) when latency matters. That pool is scarce, though. Paused or reusable blocks can move to CPU DRAM or a remote cache, and Mooncake studies a disaggregated design using CPU, DRAM, SSD, and remote direct memory access (RDMA) resources independently of one inference worker.^{[8]Reference 8Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079}

Predict the tradeoff before reading the tiers: a remote hit saves local recomputation only if transfer and lookup fit the deadline. Disaggregation moves the bottleneck rather than deleting it:

The scheduler should compare measured transfer time with local recomputation. A remote hit can lose for a short prompt or a congested link. Token count alone isn't enough; admission needs byte-level KV budgets, measured bandwidth, and a deadline check.

Tenant isolation follows the blocks off GPU. Remote keys must include the trust scope plus model, adapter, tokenizer, and KV layout version. The cache service must authorize reads before returning block metadata, encrypt traffic as required by the threat model, and invalidate ownership when a request or tenant is deleted. RDMA reduces CPU involvement in data movement; it doesn't provide application-level authorization by itself.

When should the scheduler restore remote KV instead of recomputing a prefix?

Answer

The scheduler should restore it only when the predicted lookup and transfer time fits the request deadline and is lower than the recomputation cost. Include network congestion, prefix size, compatible model state, and tenant authorization in the decision.

Prefix caching and the cross-tenant leak risk

Multi-tenant traffic often repeats a system prompt, tool schema, or retrieved prefix. Prefix caching can reuse those KV blocks and skip repeated prefill work. SGLang's RadixAttention uses a prefix tree, while vLLM's automatic prefix caching hashes blocks rather than using a radix tree.^{[9]Reference 9SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104[10]Reference 10Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Ask what a hit can improve. It mainly lowers TTFT (Time to First Token) by removing repeated prefill. It doesn't make decode itself cheaper.

Now ask whether identical prompt text proves identical state. It doesn't. vLLM's block hash includes extra values such as LoRA IDs, and its optional cache_salt on the first block limits reuse to requests in the same trust group, reducing timing-based probing.^{[10]Reference 10Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/} Routing still has to enforce compatible model, tokenizer, adapter, and policy context.

Private-prefix reuse without a tenant or approved trust-group namespace is an isolation bug, even when the intended change was a TTFT optimization. Scope cache reuse first, then measure its latency benefit.

What must be included in a safe prefix-cache namespace?

Answer

Tenant or explicit shared trust-group identity plus compatible base model, adapter, and tokenizer configuration. Public shared prompts need a deliberate shared namespace, not accidental reuse.

Chunked prefill for multi-tenant fairness

The prefill phase processes an input prompt and writes initial KV state. The decode phase emits output tokens one at a time. Prefill usually asks for compute; decode repeatedly reads memory. A long prefill can therefore delay another tenant's live stream even when the first request hasn't generated anything.

Predict the fairer schedule: let the long prefill occupy the GPU until it finishes, or split it so decodes get turns? Chunked prefill, studied in Sarathi-Serve,^{[11]Reference 11Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.https://arxiv.org/abs/2308.16369} breaks the prompt into bounded chunks and interleaves those chunks with other work.

The chunk budget is a tuning knob, not a magic constant. In vLLM, max_num_batched_tokens controls it: smaller values tend to protect inter-token latency, while larger values tend to improve TTFT and prefill throughput. Current V1 docs say chunked prefill is enabled whenever possible, with example low-latency values around 2,048 and throughput-oriented values above 8,192. Treat those as starting points, then measure against your SLO.^{[11]Reference 11Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.https://arxiv.org/abs/2308.16369[12]Reference 12Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

Chunking bounds one noisy neighbor's turn. It doesn't enforce tenant quotas or guarantee fairness without admission and scheduler policy.

Why does chunked prefill protect other tenants from a long prompt?

Answer

It breaks a long compute-heavy prefill into smaller slices that can interleave with decode work from other tenants, preventing one 32K prompt from blocking the whole batch.

Tenant-aware preemption

When admitted work approaches its KV budget, the scheduler has three honest choices: reject it, leave it queued, or preempt lower-priority work. An enterprise tier may displace best-effort work, but that decision belongs in the service policy and its metrics.

Current vLLM V1 docs default to RECOMPUTE rather than SWAP because recomputation has lower overhead in that architecture. Your platform still needs a measured choice: recomputation spends extra compute and latency, while swapping spends transfer bandwidth and host memory.^{[12]Reference 12Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

In the conceptual scheduler below, predict the victim before reading the loop. The incoming enterprise request has higher priority than both running requests and needs 80 MB. The starter request has the lowest priority and the largest KV footprint, so evicting it frees enough space without displacing business traffic:

The miniature loop accounts for released KV memory before admitting the newcomer. In production, release and allocation must be atomic. Otherwise two concurrent decisions can both observe the same free blocks and over-admit work.

What should a tenant-aware preemption policy optimize for?

Answer

Free enough KV memory while minimizing service-objective impact. Prefer lower-priority tenants, large KV footprints, and requests with less generated work, then recompute or swap depending on runtime support.

Hard per-tenant limits

Scheduling alone can't protect a pool from an oversized request that arrives before a fair-share decision. The platform also enforces hard context, concurrency, and KV quotas by tenant tier. The illustration below compares shared namespaces with stronger runtime boundaries:

Example admission policy. These numbers are scenario inputs, not universal tiers:

Derive a budget from max_concurrent × max_context × bytes_per_token, then add headroom for fragmentation, replica placement, and bursts. Using the earlier ~1.22 GiB at 4K tokens, or ~0.000305 GiB per token for that model shape:

• Enterprise peak: $50 \times 8000 \times 0.000305 \approx 122$ GiB raw KV; the 256 GB row is roughly 2× that for HBM packing waste, multi-worker share, and burst.
• Business peak: $20 \times 4000 \times 0.000305 \approx 24$ GiB raw; 80 GB is a multi-replica pool budget, not a single-request cap.
• Starter peak: $5 \times 2000 \times 0.000305 \approx 3$ GiB raw; 20 GB keeps small tenants from thrashing under noisy neighbors while still hard-capping spend.

Publish the formula and measured bytes/token for your model. The table is a policy example, not an industry standard. Apply these checks before scheduling GPU work:

Why are hard KV limits necessary even with fair scheduling?

Answer

Schedulers arbitrate admitted work, but a single oversized prompt can allocate too much memory before fairness helps. Hard limits reject or downsize requests before they consume shared KV blocks.

Hard limits stop oversized prompts. They don't stop a tenant from sending a legal-sized flood, so the next boundary sits at the gateway.

How we prevent one tenant from overwhelming the rest: rate limiting and fair queues

Rate limiting sits at the gateway, before a request reaches the GPU. Predict which tenant is more expensive: one request with a 64K prompt or one hundred requests with 100-token prompts. Request count alone can't answer, so enforce two distinct budgets:

• Requests per minute (RPM): Controls burst traffic to protect the API gateway from connection exhaustion.
• Tokens per minute (TPM): Controls sustained throughput to protect GPU compute capacity.

The 64K request can consume more prefill work and KV memory than the one hundred short turns, even though it uses fewer requests. RPM alone would admit it and could let one tenant monopolize the cache.

Distributed sliding-window enforcement

For RPM, a distributed sliding-window limiter using Redis with a Lua script gives one decision across gateway nodes. A local in-memory counter sees only one instance, so load balancing lets a tenant exceed its apparent limit.

The Lua script below removes entries older than the window, counts the remaining requests, and either allows the new request or rejects it. The key detail is using a unique sorted-set member (a request ID with a timestamp) instead of the raw timestamp alone. If two requests land in the same clock tick and you use the timestamp as both score and member, Redis collapses them into one entry and undercounts traffic:

TPM is trickier because final output length is unknown at admission. Reserve prompt tokens plus max_output_tokens, then reconcile the reservation with actual usage when the stream finishes.

Strictly synchronized Redis decisions can become a bottleneck at high throughput. Bounded bursts or approximate local counters are possible alternatives, but they weaken strict-limit semantics. Document and measure that tradeoff.

Token admission needs reservation and reconciliation. Reserve prompt plus maximum allowed output before execution, then release unused output capacity after the stream completes:

Fairness inside the scheduler

RPM and TPM protect the edge, but they don't decide which admitted request gets the next GPU step. A tenant with one 64K prompt can still consume more service time than dozens of short turns.

Inside the runtime, keep per-tenant queues and charge a virtual token budget for every prefill chunk and decode step. Schedule by priority plus virtual finish time, not raw request count. Each tenant then makes forward progress, while a higher-SLO tier can receive a defined larger share.

Keep rate limiting and quota management separate. Rate limits control bursts and protect capacity; quotas cap total usage and protect the bill. A tenant can stay under its RPM limit and still exhaust a monthly token quota in one afternoon.

Why is RPM not enough for LLM rate limiting?

Answer

One request can contain a huge prompt and request many output tokens. TPM and KV budgets protect GPU work and memory, while RPM mostly protects the gateway from bursty connection pressure.

Budgets protect capacity. They don't keep Tenant Y's postmortem out of Tenant X's context. That requires an authorization boundary at every state-bearing hop.

How we keep data private: the isolation stack

Every state-bearing layer needs an authorization boundary and a testable release policy. A request can be correctly authenticated at the gateway and still leak through retrieval, adapter lookup, prefix reuse, or KV allocation. Treat any cross-tenant read as an incident, even when the other layers behaved correctly.

The RAG relevance vs. authorization gap

Many platforms augment LLMs with retrieval-augmented generation (RAG). A vector database ranks documents by relevance. Relevance isn't permission.

Tenant X searches for “incident-retention exception policy.” Tenant Y's private postmortem may rank higher because both teams use the same reliability vocabulary. If the filter is missing, the model can summarize Tenant Y's incident for Tenant X.

Predict where to apply the check: after top-K results return, or inside the vector query? Authorization belongs in the query, before foreign candidates enter application memory:

Use authorization filtering in the retrieval operation itself. Pass authorized scope into the database query, then test that a foreign result can't cross the boundary:

The predicate now runs before candidates leave the data layer. A post-filter after broad top-K can expose foreign IDs, scores, or embeddings to application memory, and it can leave no authorized candidate after the top-K slice has already been chosen.

Why must tenant filtering happen inside the vector database query?

Answer

Post-filtering can retrieve foreign document IDs, scores, or embeddings before dropping them. Authorization must be part of the retrieval predicate so disallowed evidence is never returned to the application.

Model and prompt isolation

The same check applies outside retrieval:

• LoRA adapter isolation. Authorize the adapter ID against the tenant before loading an encrypted artifact. If adapter weights contain private tuning, record residency and release rules in the data contract.
• Prompt isolation. Keep raw prompts out of plaintext logs and authenticate or encrypt gateway-to-worker transport. The mTLS (mutual Transport Layer Security) and memory-boundary policy depends on the threat model.
• Harder runtime boundaries. If that threat model rules out shared workers, evaluate dedicated node pools, MIG (Multi-Instance GPU) partitions, or VM boundaries.^{[13]Reference 13Supported GPUshttps://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-gpus.html} Kubernetes placement selects hardware; it doesn't create isolation by itself.

State sanitization

When a shared worker serves more than one tenant, state lifecycle becomes part of the security contract:

• KV reference lifecycle. Remove request access to released KV blocks when a sequence completes. If the threat model requires cleared memory before reuse, implement and verify that policy rather than assuming the allocator provides it.
• Batch construction policy. Shared-base batching is fine only when every row carries its own tenant ID, adapter handle, KV table, and metering context. For regulated workloads, dedicated pools or MIG / VM boundaries may be safer than hardening every shared-kernel path.

PII masking as a tiered control

For regulated workloads that can tolerate redaction, a request can pass through a lightweight PII masking service, such as Presidio,^{[14]Reference 14Presidio: Data Protection and De-identification SDK.https://github.com/microsoft/presidio} before it reaches the model router. Masking reduces exposure of raw credit card or Social Security numbers. It doesn't decide which tenant's documents retrieval may return, and any placeholder-to-original mapping needs its own access control.

When is PII masking helpful, and when is it insufficient?

Answer

It reduces exposure for data that can be safely replaced with placeholders. It's insufficient for regulated tenants that need hard hardware or VM isolation, strict audit controls, or full-fidelity private context.

How we attribute cost: per-tenant metering and chargeback

A shared fleet needs an answer for each tenant: what did this workspace consume, and what should it pay? Token count is only a proxy. Two requests with equal token counts can use different GPU time when their prompt/output split, batch occupancy, preemptions, or cache-hit rate differs.

Attach a defensible metering record to every request. Include tenant_id, model and adapter version, prompt and output tokens, cache-hit tokens, queue wait, prefill and decode time, KV blocks held, preemption count, and GPU worker type. Cache-hit tokens matter because reused prefixes skip prefill. A public billing policy can price cached input separately, as current provider documents show.^{[15]Reference 15Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching[16]Reference 16Prompt caching.https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching}

For internal allocation, GPU time is the honest unit. A reasonable per-request estimate is:

$\text{cost}_{\text{req}} \approx \text{gpu\_seconds} \times \text{node\_hourly\_rate} / 3600 + \text{adapter\_residency} + \text{storage}$

Here gpu_seconds is the request's share of busy GPU time: prefill plus decode steps, divided by batch occupancy so shared steps are apportioned across co-batched tenants. You can bill on simpler token-and-tier dimensions, then reconcile against measured GPU time to find traffic shapes that cost more than their token bill suggests.

Why is per-tenant token count not enough for cost attribution?

Answer

Equal token counts can map to very different GPU time depending on prompt/output split, batch occupancy, preemptions, and cache hits. Attribute internal cost in GPU-seconds and discount cache-hit tokens, then map that to a token-and-tier price.

How the system grows: scaling, canary, and fault tolerance

The opening incident is now easier to place. A queue can grow while GPU utilization stays moderate, and a canary can be healthy at the model endpoint while its adapter or KV contract is wrong. Scaling, release, and recovery need separate signals.

Auto-scaling on queue depth

CPU utilization alone doesn't describe LLM serving pressure. Queue depth, admitted-token backlog, request latency, and KV occupancy connect capacity to the SLOs we defined earlier. Current vLLM metrics expose vllm:num_requests_running plus its waiting and swapped variants, and vllm:kv_cache_usage_perc for the fraction of used KV blocks.^{[17]Reference 17Metricshttps://docs.vllm.ai/en/stable/design/metrics/} Confirm exact labels in the deployed version before writing an alert. Sustained waiting with high KV usage suggests capacity pressure; average CPU by itself doesn't identify the bottleneck.

Cold-starting a GPU worker means making weights, kernels, and runtime state ready before it can serve. A platform can keep warm replicas, scale from a forecast, or shed eligible best-effort work while new capacity starts. Each choice trades idle cost against the SLO budget.

Worked scenario, not a benchmark: assume target traffic has a measured capacity of 4,000 output tokens/s per ready replica, needs 12 replicas, and can ramp 30% in under 10 minutes. If pulling weights and warming CUDA graphs takes 8 minutes, a warm buffer of about $0.3 \times 12 \approx 4$ replicas covers the ramp. Use a p95 queue-age threshold such as 200 ms for two windows as a scenario policy, not a universal autoscaling constant. If the buffer is exhausted, shed starter-tier work before enterprise latency gates fail.

Before scaling, read the trace rather than guessing from one dashboard:

The table gives hypotheses, not proofs. Scale when the measured bottleneck is capacity; fix policy or state when it isn't.

Any GPU performance number needs a measurement receipt. Name the accelerator and count, topology, engine and version, model and dtype, prompt/output distribution, concurrency, warmup and steady-state window, exact baseline and tuning, and correctness check. Hardware specifications are ceilings, not serving benchmarks. Report goodput under the tenant's TTFT and ITL gates when comparing designs.

Model versioning and canary rollouts

Models and LoRA adapters need versioned rollout because a new artifact can change quality, latency, memory use, or compatibility.

Predict what a 1% adapter canary must prove. Routing a controlled slice of eligible traffic from v1 to v2 tests more than output text: it tests the adapter registry, residency, KV namespace, and tenant authorization. Monitor route-specific quality, p95 TTFT and ITL, error rate, safety signals, cache misses, and preemptions over a declared observation window.

If the gates pass, increase exposure gradually. If one fails, route new eligible requests to v1 and freeze the canary. Active streams may still be on v2, and old adapter residency or KV entries may still be referenced, so changing a route pointer isn't a blanket zero-downtime promise.

Base-model updates need a separate worker pool because weights, kernels, and KV state change together. Send shadow traffic to the candidate pool to check correctness and measure the target workload before exposing real tenant traffic. After the evidence passes, shift the gateway and drain the old pool without deleting state still referenced by active streams.

Rollback is a state transition, not only a router edit: restore the previous route and adapter registry, stop new traffic to the candidate, keep compatible old state available until streams drain, invalidate incompatible prefix or KV entries, and rerun health plus tenant-isolation checks. Record which gate caused the rollback so the next canary tests that failure directly.

Why are adapter canaries easier than base-model canaries?

Answer

Adapters are small and can usually be routed per tenant on an existing base model. Base-model changes require new worker pools, separate KV caches, shadow traffic, throughput validation, and drain logic.

Fault tolerance

System reliability relies on handling GPU failures and model crashes gracefully. The response should preserve tenant scope while it removes bad work:

These mechanisms need a control plane or equivalent coordination layer to track worker readiness, orchestrate model deployments, and update routing. When an incident starts, freeze the rollout, stop new routing to the suspect worker or adapter, inspect the request trace and tenant scope, then fail over or roll back. Readiness checks reduce routing to unavailable workers; they don't prove model quality or prevent every runtime failure.

Why does a poison request need a dead-letter queue?

Answer

Repeated CUDA OOMs or malformed inputs can trigger retry storms and consume shared capacity. A DLQ removes the repeatedly failing request after bounded retries so it no longer consumes retries.