GPU Serving & Autoscaling

Advanced MLOps and DevOps for AI gave you the release machinery: promotion gates, lineage, rollout policy, and rollback triggers. Now apply that machinery to the inference fleet. GPU serving and autoscaling decide whether a promoted model can survive real traffic without burning budget or missing latency goals.

A developer assistant that becomes the default way to summarize incidents, review diffs, and explain failing deploys can overload quickly. At 8 AM, ten people are using it. By 9 AM, a release freeze sends a thousand requests through the same endpoint. If you try serving the model like a normal web API, it'll quickly grind to a halt. A web server fetches pre-computed data and forgets the user immediately. An LLM has to process every prompt and retain attention state as generation proceeds. That short-term memory is a central capacity constraint alongside compute, memory bandwidth, and queueing delay.

Standard web servers don't maintain growing state for each active user. An LLM does. To generate the next token, the model must access computed attention states from previous tokens. In LLMs, this state is the KV cache, and managing its memory footprint is one of the central challenges of production serving.

Why does an LLM serving system run out of capacity differently from a normal web API?

Answer

A normal web API usually finishes a request and releases almost all state. An LLM keeps model weights resident and grows per-request KV cache state during generation, so active conversations consume scarce GPU memory until they finish.

Engineers designing serving systems optimize for a small set of latency metrics. Teams usually attach service-level objectives (SLOs), target boundaries for acceptable user experience, to each:

1. Time To First Token (TTFT): The latency before the user sees the first word. This includes queue wait, cold readiness when relevant, and the prefill phase (processing the input prompt). Prefill is often compute-bound.
2. Inter-Token Latency (ITL), also called Time Between Tokens (TBT): The gap between consecutive streamed tokens after the first token. Each gap is a sample from the decode path. Tail SLOs (p95/p99 ITL) catch jitter that mean metrics hide.
3. Time Per Output Token (TPOT): The mean time per output token after the first token, usually total generation time after first token divided by (output tokens − 1). TPOT is useful for capacity planning and average perceived speed. It isn't the same as every individual gap: a request can have healthy TPOT while a few ITL spikes still feel broken.

Decode work is usually memory-bandwidth bound for large decoder models. When dashboards or papers say "token latency," check whether they mean per-gap ITL/TBT or mean TPOT before you set an alert.

The Key-Value (KV) cache (the stored attention computations for past tokens) grows dynamically with every generated token. Poor memory management leads to fragmentation, which forces smaller batch sizes and lower throughput.

A user says "first token is slow, but streaming is fast after that." Which phase is the first suspect?

Answer

Prefill. Slow first token points at prompt processing, cold cache, queueing, or model-load effects before generation starts. After the first token, watch ITL/TBT gaps for jitter and TPOT for mean decode speed.

GPU serving mental model

To see why GPU serving is so different from normal web scaling, keep the serving path concrete:

• The GPU is the scarce execution device. It's specialized, expensive, and you can't replace it with normal web servers.
• The model weights are the resident model state. Loading them takes time, and the worker can't decode until they are in GPU memory.
• The KV cache is per-request attention state. If a conversation started ten turns ago, its cache stays allocated until generation finishes. The cache pool fills up fast during a traffic spike.
• Autoscaling adds another ready worker. Bringing a new worker online takes time, and it can't help while the node, image, weights, and warmup are still loading.

Teams often scale because the GPU is busy. Scale when queue age, queue depth, KV-cache pressure, or latency shows that requests are waiting. A busy GPU can mean you're using the hardware well. A growing queue means users are waiting, and that's when you need more capacity.

Why is "GPU utilization is high" not enough evidence to add replicas?

Answer

High GPU utilization can mean the current batch is healthy and full. Scale when user-facing pressure appears: queue depth rises, KV cache is near saturation, TTFT/TPOT SLOs degrade, or admission control starts rejecting work.

The GPU serving stack

Start with one common production GPU serving architecture before choosing a framework. It consists of several layers:

• Infrastructure layer: Kubernetes (EKS, GKE, or AKS) manages the GPU nodes. Unlike CPU nodes, GPU capacity is expensive and sometimes scarce, so a warm pool must be justified against latency targets and measured demand.

• Control layer: In a KEDA-based setup, KEDA (Kubernetes Event-Driven Autoscaling) can activate a workload from zero and configure a Horizontal Pod Autoscaler (HPA) to scale running replicas from metrics. Karpenter (AWS) or Cluster Autoscaler can then provision GPU nodes when new pods can't be scheduled on existing capacity.^{[1]Reference 1Scaling Deployments, StatefulSets & Custom Resourceshttps://keda.sh/docs/2.19/concepts/scaling-deployments/[2]Reference 2Conceptshttps://karpenter.sh/docs/concepts/}

• Serving layer: The actual inference engines (vLLM, SGLang, TensorRT-LLM) that load models into GPU memory and handle token generation. At datacenter scale, a control plane such as NVIDIA Dynamo can orchestrate disaggregated prefill/decode pools across many of these engine replicas.^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/}

• Metrics: CPU and RAM metrics alone don't describe LLM-serving capacity. NVIDIA's DCGM (Data Center GPU Manager) exposes GPU-specific telemetry, while user-facing scaling signals usually come from the serving layer: queue depth, KV cache usage, TTFT, and inter-token latency.^{[4]Reference 4Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

In this KEDA-based stack, which component decides pod count and which component adds GPU nodes?

Answer

KEDA and the generated HPA decide pod replica count from serving metrics. The Kubernetes scheduler tries to place those pods. Karpenter or Cluster Autoscaler adds GPU nodes only when pods can't be scheduled on existing capacity.

Serving frameworks

Now zoom into the serving engine: batching policy, KV-cache allocation, framework choice, and request lifecycle.

The evolution of batching

Early serving systems used static batching: waiting for $N$ requests to arrive, padding them to the same length, and processing them together. It's inefficient because requests finish at different times. If one request generates 100 tokens and another generates 1000, the GPU must continue processing the batch until the longest request finishes, leaving slots idle for the shorter requests.

Modern frameworks use continuous batching (also known as iteration-level scheduling), pioneered by Orca^{[5]Reference 5Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu}. The scheduler operates at the granularity of a single token generation step. As soon as a request finishes, a new request from the queue can be inserted into its slot in the next iteration.

What problem does continuous batching solve that static batching can't?

Answer

Static batching waits for the longest request in the batch, leaving completed slots idle. Continuous batching repacks work every token step, so finished requests free capacity immediately for queued requests.

The small simulation below uses generation lengths for four assistant replies. A static batch doesn't refill its two slots until both original replies finish; a continuous scheduler admits the next queued reply as soon as a slot opens.

With the same replies and slots, the continuous scheduler finishes sooner because it wastes fewer slot-iterations.

vLLM and PagedAttention

One widely used open-source serving engine is vLLM, which introduced PagedAttention^{[6]Reference 6Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}. Before vLLM, inference systems pre-allocated a single contiguous block of GPU memory sized to the maximum sequence length a request might ever use. Because the actual length is revealed token-by-token during decoding, this approach created two problems:

• Internal fragmentation: most of the pre-allocated chunk sat empty for short requests.
• External fragmentation: free memory became scattered and couldn't satisfy new large contiguous allocations.

Profiling across real workloads showed that prior systems typically used only 20 to 38% of the KV cache memory they had reserved, wasting 62 to 80%.^{[6]Reference 6Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} The result: the effective batch size stayed small, and throughput suffered even when the GPU had spare VRAM.

PagedAttention treats the KV cache like virtual memory in an operating system (OS). It breaks the KV cache into fixed-size blocks (pages) that can be stored in non-contiguous memory.

Logical vs. physical blocks

PagedAttention introduces two key abstractions:

• Logical KV blocks: The view of the KV cache from the model's perspective. It appears as a contiguous sequence of tokens.
• Physical KV blocks: The actual fixed-size chunks of memory allocated in HBM (High Bandwidth Memory, the GPU's ultra-fast RAM) to store these tokens.

A Block Table maintains the mapping between logical and physical blocks. When a new token is generated, vLLM checks if the current physical block has space. If not, it allocates a new physical block from a pre-allocated pool and updates the mapping. The PagedAttention paper reports much tighter memory use and higher throughput because batches can pack around real sequence lengths instead of worst-case reservations.^{[6]Reference 6Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

PagedAttention gives the runtime fine-grained control over KV blocks. In current vLLM deployments, request-to-request reuse of a shared prompt prefix usually comes from automatic prefix caching, which reuses previously computed KV blocks when a new request shares the same prefix.^{[7]Reference 7Automatic Prefix Cachinghttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/}

How are PagedAttention and prefix caching different?

Answer

PagedAttention is memory allocation: it stores each request's KV cache in fixed-size physical blocks to reduce fragmentation. Prefix caching is reuse: it avoids recomputing KV blocks when a new request shares an already-computed prefix.

The allocation arithmetic for four conversations makes the waste visible. The fixed reservation policy reserves each request for its maximum allowed length, while paging reserves blocks only for tokens currently stored:

Paging doesn't make attention state free. It stops short conversations from holding blocks for a maximum length they haven't used.

The overall request lifecycle coordinates these components. Client requests enter an API server queue, are packed by a continuous batching scheduler, and finally executed by a model engine using PagedAttention and, when enabled, prefix caching on the GPU cluster:

Framework comparison

Choosing the right serving engine depends on your specific constraints:

The TGI documentation describes TGI as maintenance mode and directs users toward newer optimized engines such as vLLM and SGLang for new deployments.^{[10]Reference 10Text Generation Inference.https://huggingface.co/docs/text-generation-inference/index} These engines are the data plane that loads weights and generates tokens. At datacenter scale, a separate control plane such as NVIDIA Dynamo can sit above them to provide disaggregated prefill/decode scheduling and KV-aware routing across engine replicas (covered later under disaggregation).^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/}

To see how this works in practice, here's an example of initializing vLLM with settings you would benchmark before deployment. The configuration below loads a large model across multiple GPUs using tensor parallelism and sets a high memory-utilization budget for weights, KV blocks, and runtime buffers. It also uses chunked prefill so long prompts can share scheduling budget with decode traffic.^{[13]Reference 13Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html} It takes the model identifier and hardware constraints as inputs, and outputs an initialized engine ready to accept inference requests:

Before choosing a serving framework, what workload facts should you map?

Answer

Map latency SLO, expected concurrency, prompt length distribution, shared-prefix rate, GPU hardware, quantization tolerance, and operational ownership. A shared-system-prompt chatbot often fits vLLM well; an ultra-low-latency NVIDIA-only service may justify TensorRT-LLM complexity.

All of these knobs are workload-dependent. Tune them against TTFT and TPOT SLOs, rather than raw tokens/sec alone.

That SLO-first framing is goodput: the maximum offered request rate that still keeps the required fraction of requests inside both a TTFT objective and a decode objective (mean TPOT and/or tail ITL). DistServe uses this dual-attainment definition so capacity plans don't celebrate raw tokens/sec while users miss latency budgets. The continuous-batching lesson develops the math; use the same word here when you size fleets, set autoscaling success criteria, and decide whether prefill-decode disaggregation is worth the transfer cost.

GPU infrastructure

Sizing and selection

Selecting the right GPU depends on model size, quantization, and expected traffic. The dominant factors are GPU memory capacity (often called VRAM; datacenter accelerators such as H100 use HBM) to fit weights plus KV cache, and memory bandwidth to serve tokens quickly.

To estimate memory requirements for capacity, use this formula:

$$\text{Memory} \approx \text{Weights} + \text{Growing KV} + \text{Hybrid State} + \text{Runtime Overhead}$$

Where:

• $\text{Weights} \approx \text{Params} \times \text{Precision (bytes)}$ (e.g., Qwen3.6-27B at BF16 is about $27B \times 2$ bytes = 54 GB)^{[14]Reference 14Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}
• $\text{Growing KV} \approx 2 \times \text{Full-Attention Layers} \times \text{KV Heads} \times \text{Head Dim} \times \text{Seq Len} \times \text{Concurrency} \times \text{Precision}$ (the factor of 2 accounts for both the key and value tensors)
• $\text{Hybrid State}$ covers fixed-size recurrent or linear-attention state per active request. It doesn't grow with sequence length, but it still consumes memory and must be added separately.

Two distinctions matter. The older shorthand using Hidden Dim assumes full multi-head attention. Many modern decoder models use grouped-query attention (GQA) or multi-query attention (MQA), so the number of KV heads is much smaller than the total attention head count. Hybrid models also mix full attention with linear or recurrent layers. Only full-attention layers append growing K/V tensors; linear layers keep a different state shape.

Here's a practical Python function that estimates a memory-fit lower bound from model size and cache expectations. It takes the model size plus the architecture terms that control KV growth (num_kv_layers, num_kv_heads, head_dim, context length, and concurrency), then reports the minimum count of several GPU types by memory alone. Interconnect topology, kernel support, and measured latency still decide whether a configuration is deployable:

Qwen3.6-27B has 64 total language layers, but its published configuration alternates 48 linear-attention layers with 16 full-attention layers.^{[14]Reference 14Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Only those 16 full-attention layers contribute sequence-growing KV in this formula. The example separately adds about 0.15 GB of fixed recurrent matrix state per active request as a teaching lower bound; the real runtime may retain additional convolution state, allocator metadata, and temporary buffers. This low-concurrency 8k budget fits one H100 by the simplified memory check, but longer context and measured headroom still decide deployment.

Why does the growing-KV formula use num_kv_heads and only full-attention layers?

Answer

GQA and MQA share key/value heads across multiple query heads, so KV cache stores K/V for the smaller head count. In a hybrid model, only full-attention layers append those tensors at every token. Linear-attention layers keep separate fixed-size recurrent state, which belongs in another budget term.

As a reference, the following table separates sequence-growing KV from other recurrent state. The values are planning estimates, not vendor benchmarks. Precision, runtime layout, context policy, and multimodal state can move them.

The MoE rows are where fit conversations often go wrong. HBM residency includes the full expert checkpoint. Per-token decode traffic is different: it includes shared dense tensors plus the experts routed for that token, not every inactive expert byte. GLM-5.2 and Flash 0731 also use specialized long-context attention, so dense-model KV formulas are not enough. Keep residency math, active-byte bandwidth, attention state, and collective traffic separate.

Why can Qwen3.6-35B-A3B look cheap from active parameters but still be a tight one-GPU plan?

Answer

The A3B suffix is about activated parameters per token, not total checkpoint residency. The full expert pool, KV cache, runtime buffers, prefix-cache headroom, and traffic spikes still count against HBM.

Multi-Instance GPU (MIG)

For cost-efficient autoscaling of smaller models, Multi-Instance GPU (MIG) can partition a single GPU into hardware-isolated slices. Ampere- and Hopper-class parts such as A100, H100, and H200 can expose up to seven instances on supported SKUs.^{[25]Reference 25Supported GPUshttps://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-gpus.html} Instead of dedicating an entire large GPU to a single 7B model, you can run multiple replicas on isolated slices with dedicated compute and memory resources.

Throughput & bandwidth constraints

For low-batch decode of large decoder models, weight reads commonly make the system memory-bandwidth bound. Batching, cache traffic, kernels, and multi-GPU communication can change which limit dominates.

The theoretical maximum throughput ($TPS_{max}$) for a batch size of 1 is roughly (a simplified model derived from memory-bandwidth analyses such as Pope et al.)^{[26]Reference 26Efficiently Scaling Transformer Inference.https://arxiv.org/abs/2211.05102}:

$$TPS_{max} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$$

Reading the formula

In this simplified batch-size-one model, the GPU streams the relevant active weight footprint from HBM for each generated token. Little weight reuse happens across sequential steps (the KV cache reuses activations, not the static weights). For a dense model, the active footprint is approximately the full model. For an MoE model, use shared weights plus routed expert bytes measured for the token; full checkpoint residency isn't the decode denominator.

$$TPS_{max} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$$

This is the reciprocal of the time required to load the relevant weight footprint from HBM once. An H100 SXM (3.35 TB/s HBM bandwidth) serving Qwen3.6-27B in BF16 (about 54 GB of dense weights) yields an upper bound of roughly 62 tokens/sec for a single sequence.^{[27]Reference 27H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/[14]Reference 14Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Real systems achieve lower throughput because:

• KV cache reads/writes, activations, and sampling also consume memory bandwidth
• Kernel launch, attention computation, and scheduler overhead
• The achieved bandwidth of fused kernels is rarely 100 % of the theoretical peak

This bound explains why quantization can improve decode throughput: halving only the weight-footprint term (FP16 to INT8, for example) doubles this simplified ceiling. Actual speedup is smaller or different when kernel support, KV-cache traffic, compute, or communication dominate. It also shows why large models may require tensor parallelism: the denominator grows while per-GPU bandwidth stays fixed.

The next calculation is deliberately a ceiling, not a benchmark. It isolates weight traffic so you can see what quantization changes before adding runtime overhead:

Why does weight quantization often improve decode speed besides memory footprint?

Answer

Decode repeatedly streams model weights from HBM. Smaller weights reduce bytes loaded per token, raising the memory-bandwidth upper bound for tokens per second.

Tensor parallelism

For models that don't fit comfortably on a single GPU, or that fit weights but not the target KV-cache and headroom budget, we use Tensor Parallelism (TP). TP splits individual weight matrices (e.g., $W_Q, W_K, W_V$) across multiple GPUs so each device computes one shard. This technique was pioneered in the Megatron-LM training system^{[28]Reference 28Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.https://arxiv.org/abs/1909.08053} and adapted for inference serving.

TP operates by slicing matrix computations, so participating GPUs exchange partial results through collective communication within each transformer layer. In the standard Megatron-style transformer block, budget two collectives per layer: one around the attention output projection and one around the feed-forward output projection. Exact collective type and placement depend on row/column partitioning and runtime, but counting only one synchronization per layer understates communication.

• Intra-node: TP works best when GPUs share a very fast fabric such as NVLink (NVIDIA high-bandwidth interconnect) or NVSwitch (NVIDIA multi-GPU switch).
• Inter-node: TP can extend across RDMA (Remote Direct Memory Access) or InfiniBand, but the communication tax rises quickly. For multi-node serving, teams often keep TP within a node when they can, and use Pipeline Parallelism (PP) or independent replicas across nodes instead. PP only communicates activations at stage boundaries, but it introduces pipeline bubbles and can worsen tail latency.

When should you prefer tensor parallelism within a node over spreading one model across nodes?

Answer

Prefer intra-node tensor parallelism when the GPUs share fast NVLink or NVSwitch and the model needs matrix shards on every layer. Crossing nodes adds communication latency; use pipeline parallelism or independent replicas when inter-node fabric would dominate tail latency.

This diagram shows where bytes move in each split:

Communication frequency isn't total cost: tensor-parallel messages may travel over faster links, while pipeline stages can create bubbles. This toy count makes the first screening question explicit:

Autoscaling strategies

CPU-based autoscaling alone is insufficient for LLM serving because CPU isn't usually the scarce inference resource. CPU can still reveal overloaded gateways or tokenizers. GPU utilization can also be misleading: a GPU might be busy with a healthy batch, or it might be memory-bound while compute duty cycle looks modest.

Key metrics

The most reliable metrics for autoscaling come from two sources: NVIDIA's DCGM (Data Center GPU Manager) for hardware-level visibility, and the serving framework itself for application-level signals. In vLLM, for example, the Prometheus endpoint exposes metrics such as vllm:num_requests_waiting, vllm:num_requests_running, vllm:kv_cache_usage_perc, vllm:time_to_first_token_seconds, and vllm:inter_token_latency_seconds.^{[4]Reference 4Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

DCGM (Data Center GPU Manager) exposes hardware-level GPU metrics like temperature, power, clock rates, engine activity, and memory traffic. While GPU utilization seems like an obvious scaling signal, it isn't a complete serving-capacity signal. High activity can reflect useful work or a bottleneck elsewhere in the memory path; low activity can reflect sparse or bursty work. Neither tells you how long requests are waiting or how many KV blocks remain schedulable.

That's why application-level metrics (queue depth, KV cache utilization, TTFT, ITL, TPOT) are more reliable scaling signals. They directly measure capacity constraints rather than hardware activity. Alert on both mean TPOT and the ITL histogram: a healthy request-mean can hide a few long gaps that still break streaming. Capacity success is goodput under those dual latency objectives, not GPU busy percent. The exact thresholds are workload-specific, so treat any numbers you see in dashboards or sample code as starting points, not universal defaults.

Which metric should page you first: average GPU utilization or sustained request queue depth?

Answer

Sustained request queue depth. It means users are waiting before admission. GPU utilization is supporting context, but it doesn't tell you whether the service has enough schedulable KV capacity for current demand.

KEDA and Karpenter integration

One control-loop design works as follows: the serving framework exposes metrics via Prometheus. KEDA can activate a scaled workload from zero and configure the generated HPA for running replica scaling. If the scheduler can't place new GPU pods, the node autoscaler layer (often Karpenter or Cluster Autoscaler) provisions nodes that satisfy pod requirements.^{[1]Reference 1Scaling Deployments, StatefulSets & Custom Resourceshttps://keda.sh/docs/2.19/concepts/scaling-deployments/[2]Reference 2Conceptshttps://karpenter.sh/docs/concepts/}

This two-layer approach separates desired pod count from hardware supply. Ready pods may scale quickly if spare GPU capacity exists; new nodes and model loading can take far longer. The useful warm buffer is a measured tradeoff between idle GPU cost and the latency damage of a cold burst.

Why is GPU autoscaling a two-loop problem?

Answer

Replica count changes quickly, but GPU node capacity appears slowly. Pod scaling decides desired serving capacity; node autoscaling supplies hardware only after unschedulable GPU pods reveal the cluster lacks the right nodes.

The autoscaling loop should separate replica pressure from hardware supply:

In a real Kubernetes deployment, KEDA and HPA compute the desired replica count for you. This Python snippet is a mental model for the control logic. It takes the current metrics and replica count as inputs, then returns the next target replica count. The thresholds are illustrative only:

The autoscaler determines a target; it doesn't erase startup time. This capacity calculation separates the eventual replica count from the traffic a warm buffer can accept immediately:

A worked example: scaling from zero

A concrete scenario exposes the scaling policy. You're hosting an 8B developer-assistant model for incident summaries and deploy reviews. Traffic is nearly zero at night, but at 9 AM it jumps to 100 concurrent users. How do you scale without blowing the budget or the user experience?

Step 1: Pick the right metric. Scaling only on CPU percentage misses engine capacity. A GPU worker can show little CPU pressure while its weights and KV blocks constrain admission. Use request queue depth, queue age, KV cache utilization, and latency SLOs; retain CPU signals for gateway or preprocessing failures.

Step 2: Set a threshold. Through benchmarking, you determine that one GPU can comfortably handle about 5 concurrent requests while keeping TPOT under 50 ms per token. That's your target capacity per replica.

Step 3: Do the math. With 100 concurrent requests and 5 requests per GPU, you need 100 / 5 = 20 GPUs. If you're starting from one warm instance at 8:59 AM, the autoscaler must add 19 more replicas quickly.

Step 4: Solve the cold-start gap. Even if a replica target changes promptly, node provisioning plus model loading can outlast the traffic's latency budget. Measure that ready time on your platform. If the morning rush is predictable, start warming nodes far enough ahead of the observed ready time, or keep a benchmarked buffer pool that absorbs its first wave.

If one replica handles 5 concurrent requests and traffic jumps to 100 concurrent requests, what must scaling plan include beyond "set max replicas to 20"?

Answer

It must cover timing. The fleet needs about 20 replicas, but cold nodes may take minutes to provision and load weights, so use predictive warmup or a warm buffer before the spike reaches the queue.

Scaling only when latency spikes is late. By the time TTFT breaches your SLO, users are already frustrated. Scale on queue depth trends rather than lagging latency indicators alone.

Handling cold starts

Cold starts in GPU serving hurt more than cold starts in traditional microservices. A typical web container might start in seconds; a GPU workload often has to provision a node, pull a large image, fetch model weights, and warm up the runtime before it can accept traffic.

Compare a warm GPU (already loaded and serving) with a cold GPU (starting from scratch):

The cold start has four distinct phases, each with its own mitigation strategy. The ranges below are illustrative planning inputs, not platform guarantees; record your own p50 and p95 timings:

Container streaming lets the container start executing before the entire image is downloaded. Only the required layers are fetched on demand, which cuts startup time for large inference images.

Model caching keeps frequently used model weights on high-speed local NVMe storage rather than pulling from remote object storage. For multi-node setups, shared network volumes like Amazon FSx for Lustre can serve weights at high speed to multiple nodes.

Some platforms also offer snapshot or restore features for a preloaded container or VM. Those can help, but they're more vendor-specific than warm pools plus local weight caches.

To mitigate cold start latency impact on users, engineers implement several proactive strategies:

• Over-provisioning: Keep a small headroom buffer to absorb sudden traffic spikes while new instances spin up.
• Predictive scaling: Use historical traffic patterns to scale up the cluster before anticipated usage spikes (e.g., scaling up just before the morning rush hour).
• Minimum warm pool: For high-priority workloads, retain enough measured warm capacity to protect the initial-burst TTFT target while cold replicas load.

You can turn measured phase durations into a readiness decision. In this example, predictive scaling four minutes before a known incident-review surge is sufficient for the measured path, while scaling two minutes before it's not:

Which cold-start phase should you optimize if pods are scheduled quickly but first token still takes minutes?

Answer

Look at container pull, model weight fetch, and runtime warmup. Fast scheduling only proves a node exists; the model still needs image layers, weights, and readiness warmups before serving traffic.

Cost optimization

Running top-end GPUs 24/7 is expensive, and idle replicas burn budget fast. To optimize costs without breaking Service Level Agreements (SLAs):

Spot instances and fault tolerance

Cloud providers offer spot instances at discounts, but they come with preemption risk. Inference can be easier to retry than long-running training, but reasoning runs and long generations may exceed a provider's warning window. Use measured request durations, a drain deadline, and retry behavior before assigning traffic to spot capacity.

For a code assistant, predictable launch-day or migration-week spikes can be candidates for spot overflow capacity when a stable baseline pool meets the promised SLO and interrupted work can drain or retry cleanly.

• Graceful shutdown: On SIGTERM, remove the replica from new admission, let completions that fit the remaining drain budget finish, and retry or fail over requests that can't finish before termination.
• Mixed pools: A common policy retains on-demand or reserved capacity for baseline SLO traffic and uses an autoscaling spot pool only for retryable overflow.

Why are spot GPUs often acceptable for inference but risky without drain logic?

Answer

Some inference requests finish within a preemption window or can be retried. Without drain logic, the load balancer keeps sending new work to a terminating node and active long generations are cut off without an explicit recovery path.

The drain policy must compare remaining work with a deadline rather than assume all requests finish. This example reserves twenty seconds for shutdown and transfer after a 120-second termination warning:

Scale-down cooldown & thrashing

A common (and expensive) mistake is scaling down too aggressively. If you terminate a GPU node because traffic dipped for 30 seconds, you'll pay the cold start penalty again when traffic returns a minute later. This "thrashing" can increase costs while degrading user experience.

It's like shutting down a warm inference replica because the queue cleared for 30 seconds, only to pay model-load latency again when the next burst arrives.

Scale-down rules:

• Cooldown period: Start with a cooldown longer than short observed lulls and tune it against measured cold-start cost, idle spend, and recurring traffic patterns.
• Drain before terminate: Give the autoscaler time to drain in-flight requests before removing a replica. Don't kill a GPU mid-generation.
• Scale down gradually: Remove one replica at a time and wait to see if queue depth remains low before removing more.

Why can aggressive scale-down increase both cost and latency?

Answer

It repeatedly destroys warm capacity, then pays node provisioning and model loading again when traffic rebounds. The service burns startup time and money without serving useful requests during the cold-start gap.

Deployment options

When deciding how to deploy GPU inference, teams face a build-vs-buy decision:

Serverless platforms like Modal and RunPod abstract away the Kubernetes layer entirely. They typically handle autoscaling and much of the instance lifecycle for you. This is ideal for teams without dedicated ML infrastructure engineers or for workloads with highly variable traffic.

Specialized GPU clouds sit in the middle. You usually get raw instances, storage, or managed Kubernetes primitives without the full DIY burden of the hyperscalers. Managed Kubernetes still gives you the most control over the full stack (vLLM versions, custom schedulers, quantization methods) and can be cheaper once utilization is high and predictable enough to pay back the platform engineering overhead.

Common pitfalls

Even experienced engineers trip over the same patterns when moving from web serving to GPU inference. These are the frequent failures to catch during design review, with their symptoms, root causes, and fixes.

Flapping: scaling up and down too fast

• Symptom: Your GPU node count oscillates wildly. Cloud bills spike, but user latency doesn't improve. The autoscaler logs show scale-up events followed by scale-down events within minutes.

• Cause: The autoscaler reacts to short traffic blips instead of sustained trends. A GPU node that just finished loading weights gets terminated before it serves enough requests to justify its startup cost.

• Fix: Choose a cooldown from traffic and startup measurements rather than a universal duration. Scale down gradually, and require queue depth near zero, low KV cache usage, and low running requests before removing capacity.

Ignoring tail latency

• Symptom: Your dashboard shows an average response time of 800 ms, but users complain about 30-second waits. The p95 or p99 latency is an order of magnitude worse than the mean.

• Cause: Average latency hides the users with long prompts or the batches where one straggler request keeps the GPU occupied. Static batching makes this worse, but even continuous batching can suffer if a single request with a 4,000-token prompt monopolizes a slot.

• Fix: Monitor TTFT and TPOT at the p95 or p99 percentile, not the mean. Set SLOs on tail latency. If long prompts are common, enable chunked prefill so they don't starve shorter decode requests.

No explicit queue or backpressure

• Symptom: Requests keep arriving even after the GPU workers are full. The service eventually times out randomly, and retry storms make the spike worse.

• Cause: The system accepts more work than the serving engine can schedule. Without a visible queue, admission policy, timeout budget, and retry contract, overload becomes invisible until users see failures.

• Fix: Put a queue or gateway in front of the inference engine. Track queue depth, queue age, and rejection rate. Return controlled overload responses before the GPU fleet collapses.

Treating every request as equal priority

• Symptom: A batch analytics job slows down interactive chat, or one tenant's long prompts make all tenants miss their TTFT target.

• Cause: The scheduler sees all requests as identical even though some workloads are interactive, some are batch, and some have stricter contractual latency limits.

• Fix: Separate traffic classes. Use priority queues, per-tenant rate limits, maximum prompt and generation budgets, and different pools when workload shapes are too different to share one scheduler fairly.

Underestimating long-context KV growth

• Symptom: A model that passed a short-prompt load test starts rejecting work or swapping under real chat history. GPU memory looks fine at startup and then collapses as conversations lengthen.

• Cause: The capacity plan counted weights but not enough KV cache. Long prompts, high concurrency, prefix-cache headroom, and larger generation limits all consume blocks during the request lifetime.

• Fix: Size with realistic prompt and output distributions, not maximum model weights alone. Load test with long conversations, watch vllm:kv_cache_usage_perc, and apply admission limits before the cache pool hits saturation.

Confusing PagedAttention with prefix caching

• Symptom: A team enables PagedAttention and expects repeated system prompts to become free, then sees no TTFT improvement on new replicas.

• Cause: PagedAttention is an allocation strategy for KV blocks. Prefix caching is a reuse strategy for shared prompt prefixes. One reduces fragmentation; the other avoids recomputing prefix KV state.

• Fix: Use both when the workload benefits from both. Treat PagedAttention as baseline memory management and prefix caching as a workload-specific optimization that needs stable shared prefixes and warm caches.

Cache blindness during scale-up

• Symptom: You scale from 2 to 10 GPUs during a traffic spike, but the new nodes serve requests slower than the old ones. TTFT increases right after scaling.

• Cause: The original nodes have been running for hours and have accumulated prefix cache hits (for example, a shared system prompt that every request includes). The brand-new nodes start with empty caches. They must recompute the full prefill from scratch, so their first tokens take much longer.

• Fix: Warm new nodes with a few synthetic requests that populate the common prefix before adding them to the load balancer rotation. Alternatively, use vLLM's prefix caching and send new nodes "seed" traffic to warm their KV cache before taking full production load.

Why might new replicas make TTFT worse right after a scale-up?

Answer

Old replicas may have hot prefix caches, while new replicas start cold. If traffic shifts immediately, the new replicas recompute shared prompt prefixes and first-token latency rises despite higher replica count.

Advanced techniques

Prefill-decode disaggregation

Prefill and decode stress GPUs differently. Prefill processes many prompt tokens in parallel and tends to be compute-heavy. Decode produces one token per active request and tends to be memory-bandwidth-heavy. If you run both phases on the same worker pool, a few long prompts can delay short streaming responses, and decode traffic can leave tensor cores underused.

Prefill-decode disaggregation, introduced for production serving by Splitwise^{[29]Reference 29Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677}, splits the serving fleet into two pools:

• Prefill workers ingest prompts, build the initial KV cache, and hand off cache state.
• Decode workers continue token generation for active requests with tighter scheduling around TPOT.

This isn't free. You now need cache transfer, placement logic, and backpressure between pools. But it can help high-traffic systems where long prompts and streaming generations compete for the same GPU budget. The autoscaling signals also become more specific: prefill workers scale on prompt-token backlog and TTFT, while decode workers scale on active sequences, KV-cache utilization, and TPOT.

NVIDIA Dynamo is one documented open-source control-plane example. Its documentation describes disaggregated prefill and decode deployments across backends including vLLM, SGLang, and TensorRT-LLM, with KV-aware routing and KV-transfer mechanisms for split deployments.^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/} This makes it a useful implementation to study, but the operational win still depends on measured prompt mix, transfer cost, and cache-hit behavior.

Don't start here. First tune continuous batching, chunked prefill, prefix caching, and queue policy. Disaggregation is a later optimization when one mixed pool can no longer hit both TTFT and TPOT targets, which is the same dual-SLO condition that defines goodput under load.

When does prefill-decode disaggregation become worth the extra system complexity?

Answer

Use it when long prompts and streaming decode traffic fight inside one worker pool after simpler fixes are exhausted. It helps when separate prefill and decode autoscaling signals are needed to restore goodput (request rate under both TTFT and decode SLOs).

Use measurements, not architecture fashion, to decide whether to split pools. This gate recommends investigation only when a tuned mixed pool breaches both service objectives under a long-prompt-heavy trace:

Speculative decoding

Speculative decoding speeds up generation by using a smaller "draft" model to predict multiple tokens ahead, then verifying them with the larger "target" model. Leviathan et al. report roughly 2-3x acceleration in their evaluated settings, not a universal serving guarantee.^{[30]Reference 30Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

The speedup comes from amortizing target-model work across several draft tokens. The draft model's predictions are treated as hypotheses; the target model evaluates them in one verification pass and keeps the valid prefix. When the draft model is reasonably accurate for the traffic pattern, this reduces the number of expensive target forward passes. It still pays draft-model compute and rejection overhead, so you measure it against your own workload instead of assuming it always helps.

For example, you might pair a small draft model with a Qwen3.6-27B target to accelerate decode-heavy traffic. The draft model generates candidate tokens quickly; the target model then verifies all candidates in a single forward pass, accepting all correct predictions up to the first mismatch.

When is speculative decoding most likely to help a serving fleet?

Answer

It helps decode-heavy, memory-bandwidth-bound workloads when a cheap draft model predicts a high fraction of target-model tokens correctly. It's less useful when prefill, retrieval, queueing, or network latency dominates.

Multi-region failover

GPU quota and available capacity vary by region and time. A resilient serving architecture should be able to route traffic to a secondary region when the primary is at capacity or experiencing issues.

This requires:

• Global load balancing: Route requests to the nearest healthy region
• Model sync: Keep model weights and LoRA adapters synchronized across regions
• Failover logic: Detect regional capacity constraints and automatically shift traffic

Regional failover is particularly important for spot instance workloads. If a regional spot fleet is reclaimed, your tested recovery-time objective determines whether secondary on-demand capacity can receive new requests before client timeout budgets expire. Long in-flight generations still need retry or resumption semantics.

What must be synchronized before multi-region failover is safe for tenant-specific models?

Answer

Synchronize base model versions, LoRA adapter versions, tokenizer/config files, safety policy versions, and routing metadata. Otherwise failover can silently serve a different model behavior than the primary region.

Multi-tenancy with LoRA

Instead of deploying a separate Qwen3.6-27B base for every customer fine-tune, use one shared base model and select LoRA (Low-Rank Adaptation)^{[31]Reference 31LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} adapters per request. Systems like S-LoRA^{[32]Reference 32S-LoRA: Serving Thousands of Concurrent LoRA Adapters.https://arxiv.org/abs/2311.03285} show how to batch requests across different adapters while keeping the base model weights shared in GPU memory. vLLM also supports per-request LoRA serving, with explicit warnings around separate runtime adapter-loading APIs in untrusted environments.^{[33]Reference 33LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/}

To implement multi-tenancy, map each tenant to an approved adapter and select that adapter per request. TenantAdapterRouter shows how a single base model instance can serve requests for different tenants by passing a LoRARequest to vLLM. While a true production environment would use vLLM's AsyncLLMEngine to handle concurrent requests without blocking, the synchronous LLM class is shown here for conceptual clarity. The function takes an incoming request containing a tenant ID, looks up the approved adapter path, and outputs the generated response with the selected adapter. This example doesn't expose vLLM's runtime adapter-loading APIs; those are a separate trusted-environment feature:

In production, prefer the async server path over a synchronous wrapper like this, and don't expose arbitrary adapter loading to untrusted tenants.^{[33]Reference 33LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/}

What is the main safety risk in LoRA multi-tenancy?

Answer

Runtime adapter loading can become a code and data isolation boundary. Only load approved adapter paths, bind tenant IDs to authorized adapters, version adapters immutably, and avoid accepting arbitrary adapter locations from requests.

Serving control loop recap

You started with a developer assistant that hit a release-freeze traffic spike and discovered why LLM serving isn't web serving. KV-cache capacity constrains active conversations; low-batch decode is often limited by memory bandwidth; and autoscaling needs queue, cache, and latency signals in addition to infrastructure telemetry.

A Kubernetes pattern using KEDA/HPA, a GPU node autoscaler, and a serving engine separates desired replicas from hardware provisioning. Continuous batching and PagedAttention improve GPU use, but a reliable fleet still needs measured capacity, a tuned cooldown, and a warmup policy justified by its latency target and cost.

GPU serving needs a different control loop than a web server. New nodes can run slower than old ones, and a thousand-user spike needs admission control, batching policy, and budget gates before it overruns the inference lane.