GPU Serving & Autoscaling

At 8:59, one warm GPU handles incident summaries comfortably. At 9:00, a release freeze sends a thousand requests through the same endpoint. Queue age climbs, first-token latency slips, and the replica count starts catching up after users are already waiting. Would you add replicas because GPU utilization is high? You need the request path and its memory state first.

Advanced MLOps and DevOps for AI gave you promotion gates, lineage, rollout policy, and rollback triggers. Apply that release discipline to the inference fleet: GPU serving and autoscaling must keep a promoted model inside latency and cost budgets as traffic changes.

A web server can finish a request and release most of its state. An LLM keeps model weights resident and grows per-request attention state as it generates. To produce the next token, it reads keys and values from earlier tokens. That state is the KV cache, and its memory footprint is one of the central capacity limits in serving.

Why does an LLM serving system run out of capacity differently from a normal web API?

Answer

A normal web API usually finishes a request and releases almost all state. An LLM keeps model weights resident and grows per-request KV cache state during generation, so active conversations consume scarce GPU memory until they finish.

Serving teams watch a small set of latency metrics, then attach service-level objectives (SLOs) (target bounds for acceptable user experience) to each:

1. Time To First Token (TTFT): The latency before the user sees the first word. This includes queue wait, cold readiness when relevant, and the prefill phase (processing the input prompt). Prefill is often compute-bound.
2. Inter-Token Latency (ITL), also called Time Between Tokens (TBT): The gap between consecutive streamed tokens after the first token. Each gap is a sample from the decode path. Tail SLOs (p95/p99 ITL) catch jitter that mean metrics hide.
3. Time Per Output Token (TPOT): The mean time per output token after the first token, usually total generation time after first token divided by (output tokens − 1). TPOT is useful for capacity planning and average perceived speed. It isn't the same as every individual gap: a request can have healthy TPOT while a few ITL spikes still feel broken.

Decode work is usually memory-bandwidth bound for large decoder models. When dashboards or papers say "token latency," check whether they mean per-gap ITL/TBT or mean TPOT before you set an alert.

The KV cache grows with every generated token. Fragmentation from clumsy allocation forces smaller batches and lower throughput.

A user says "first token is slow, but streaming is fast after that." Which phase is the first suspect?

Answer

Prefill. Slow first token points at prompt processing, cold cache, queueing, or model-load effects before generation starts. After the first token, watch ITL/TBT gaps for jitter and TPOT for mean decode speed.

Why GPU serving isn't web serving

The spike leaves two questions: what state does each request consume, and which signal says the fleet needs capacity? Four pieces answer them:

• The GPU is the scarce execution device. It's specialized, expensive, and you can't replace it with normal web servers.
• The model weights are the resident model state. Loading them takes time, and the worker can't decode until they are in GPU memory.
• The KV cache is per-request attention state. If a conversation started ten turns ago, its cache stays allocated until generation finishes. The cache pool fills up fast during a traffic spike.
• Autoscaling adds another ready worker. Bringing a new worker online takes time, and it can't help while the node, image, weights, and warmup are still loading.

Teams often scale because the GPU is busy. Scale when queue age, queue depth, KV-cache pressure, or latency shows that requests are waiting. A busy GPU can mean you're using the hardware well. A growing queue means users are waiting, and that's when you need more capacity.

That split between the execution device and the user-facing pressure signal is why the serving stack has two paths, not one web-style replica loop.

Why is "GPU utilization is high" not enough evidence to add replicas?

Answer

High GPU utilization can mean the current batch is healthy and full. Scale when user-facing pressure appears: queue depth rises, KV cache is near saturation, TTFT/TPOT SLOs degrade, or admission control starts rejecting work.

The GPU serving stack

A request crosses two loops before its first token arrives. The hot path queues work, schedules batches, runs the model, and streams tokens. The control path watches that work, asks for more replicas, and waits for GPU nodes when those replicas can't land. Keep that split in mind as you map the stack:

• Infrastructure layer: Kubernetes (EKS, GKE, or AKS) manages the GPU nodes. Unlike CPU nodes, GPU capacity is expensive and sometimes scarce, so a warm pool must be justified against latency targets and measured demand.

• Control layer: In a KEDA-based setup, KEDA (Kubernetes Event-Driven Autoscaling) can activate a workload from zero and configure a Horizontal Pod Autoscaler (HPA) to scale running replicas from metrics. Karpenter (AWS) or Cluster Autoscaler can then provision GPU nodes when new pods can't be scheduled on existing capacity.^{[1]Reference 1Scaling Deployments, StatefulSets & Custom Resourceshttps://keda.sh/docs/2.19/concepts/scaling-deployments/[2]Reference 2Conceptshttps://karpenter.sh/docs/concepts/}

• Serving layer: The actual inference engines (vLLM, SGLang, TensorRT-LLM) that load models into GPU memory and handle token generation. At datacenter scale, a control plane such as NVIDIA Dynamo can orchestrate disaggregated prefill/decode pools across many of these engine replicas.^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/}

• Metrics: CPU and RAM metrics alone don't describe LLM-serving capacity. NVIDIA's DCGM (Data Center GPU Manager) exposes GPU-specific telemetry, while user-facing scaling signals usually come from the serving layer: queue depth, KV cache usage, TTFT, and inter-token latency.^{[4]Reference 4Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

In this KEDA-based stack, which component decides pod count and which component adds GPU nodes?

Answer

KEDA and the generated HPA decide pod replica count from serving metrics. The Kubernetes scheduler tries to place those pods. Karpenter or Cluster Autoscaler adds GPU nodes only when pods can't be scheduled on existing capacity.

Serving frameworks

The stack named the engine. The next question is how that engine packs work so those GPU workers stay busy instead of waiting on the longest request.

Static batches wait for the longest reply

Early serving systems used static batching: waiting for $N$ requests to arrive, padding them to the same length, and processing them together. If one incident summary generates 100 tokens and another generates 1000, the GPU keeps the short request's empty slot until the long one finishes.

Modern frameworks use continuous batching (also known as iteration-level scheduling), pioneered by Orca^{[5]Reference 5Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu}. The scheduler operates at the granularity of a single token generation step. As soon as a request finishes, a new request from the queue can be inserted into its slot in the next iteration.

What problem does continuous batching solve that static batching can't?

Answer

Static batching waits for the longest request in the batch, leaving completed slots idle. Continuous batching repacks work every token step, so finished requests free capacity immediately for queued requests.

The small simulation below uses generation lengths for four assistant replies. A static batch doesn't refill its two slots until both original replies finish; a continuous scheduler admits the next queued reply as soon as a slot opens.

With the same replies and slots, the continuous scheduler finishes sooner because it wastes fewer slot-iterations.

vLLM and PagedAttention

One widely used open-source serving engine is vLLM, which introduced PagedAttention^{[6]Reference 6Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}. Before vLLM, inference systems pre-allocated a single contiguous block of GPU memory sized to the maximum sequence length a request might ever use. Because the actual length is revealed token-by-token during decoding, this approach created two problems:

• Internal fragmentation: most of the pre-allocated chunk sat empty for short requests.
• External fragmentation: free memory became scattered and couldn't satisfy new large contiguous allocations.

Profiling across real workloads showed that prior systems typically used only 20.4% to 38.2% of the KV cache memory they had reserved, wasting the rest on fragmentation.^{[6]Reference 6Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} The result: the effective batch size stayed small, and throughput suffered even when the GPU had spare VRAM.

PagedAttention treats the KV cache like virtual memory in an operating system (OS). It breaks the KV cache into fixed-size blocks (pages) that can be stored in non-contiguous memory.

Logical vs. physical blocks

PagedAttention introduces two key abstractions:

• Logical KV blocks: The view of the KV cache from the model's perspective. It appears as a contiguous sequence of tokens.
• Physical KV blocks: The actual fixed-size chunks of memory allocated in HBM (High Bandwidth Memory, the GPU's ultra-fast RAM) to store these tokens.

A Block Table maintains the mapping between logical and physical blocks. When a new token is generated, vLLM checks if the current physical block has space. If not, it allocates a new physical block from a pre-allocated pool and updates the mapping. The PagedAttention paper reports much tighter memory use and higher throughput because batches can pack around real sequence lengths instead of worst-case reservations.^{[6]Reference 6Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

PagedAttention gives the runtime fine-grained control over KV blocks. In current vLLM deployments, request-to-request reuse of a shared prompt prefix usually comes from automatic prefix caching, which reuses previously computed KV blocks when a new request shares the same prefix.^{[7]Reference 7Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

How are PagedAttention and prefix caching different?

Answer

PagedAttention is memory allocation: it stores each request's KV cache in fixed-size physical blocks to reduce fragmentation. Prefix caching is reuse: it avoids recomputing KV blocks when a new request shares an already-computed prefix.

The allocation arithmetic for four conversations makes the waste visible. The fixed reservation policy reserves each request for its maximum allowed length, while paging reserves blocks only for tokens currently stored:

Paging doesn't make attention state free. It stops short conversations from holding blocks for a maximum length they haven't used.

The overall request lifecycle coordinates these components. Client requests enter an API server queue, are packed by a continuous batching scheduler, and finally executed by a model engine using PagedAttention and, when enabled, prefix caching on the GPU cluster:

Compare engines against one request shape

Popularity doesn't tell you whether an engine fits the incident assistant. Hold request shape, SLOs, hardware, precision, and correctness checks fixed, then compare the execution paths:

The TGI documentation describes TGI as maintenance mode and directs users toward newer optimized engines such as vLLM and SGLang for new deployments.^{[10]Reference 10Text Generation Inference.https://huggingface.co/docs/text-generation-inference/index} These engines are the data plane that loads weights and generates tokens. At datacenter scale, a separate control plane such as NVIDIA Dynamo can sit above them to provide disaggregated prefill/decode scheduling and KV-aware routing across engine replicas (covered later under disaggregation).^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/}

Initialize vLLM for the 8B incident assistant from the opening. The configuration loads one GPU's worth of weights, sets a high memory-utilization budget for weights, KV blocks, and runtime buffers, and turns on chunked prefill so long prompts can share scheduling budget with decode traffic.^{[13]Reference 13Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html} Inputs are the model identifier and hardware constraints. Output is an engine ready to accept Summarize the rollback plan for incident INC-1842.

Before choosing a serving framework, what workload facts should you map?

Answer

Map latency SLO, expected concurrency, prompt length distribution, shared-prefix rate, GPU hardware, quantization tolerance, and operational ownership. A shared-system-prompt chatbot often fits vLLM well; an ultra-low-latency NVIDIA-only service may justify TensorRT-LLM complexity.

All of these knobs are workload-dependent. Tune them against TTFT and TPOT SLOs, rather than raw tokens/sec alone.

That SLO-first capacity measure is goodput: the highest offered request rate at which a predeclared fraction of requests stays within both a TTFT objective and a decode objective. DistServe uses mean TPOT for its decode constraint; if your product promises a tail ITL bound, declare that gate separately instead of silently treating TPOT as ITL.^{[14]Reference 14DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670} Use goodput when you size fleets, set autoscaling success criteria, and decide whether prefill-decode disaggregation is worth its transfer cost.

Benchmark the path you'll operate

A memory formula and a goodput target narrow the search, but neither tells you what one replica can serve. Measure the request path that users will hit. A list of identical 512-token prompts answers a toy throughput question; it won't expose queue growth, long-prefill interference, prefix-cache misses, or the cost of streaming tails.

Start with a versioned trace. Each row should carry an arrival time, prompt-token count, output-token cap, shared-prefix ID, tenant or priority class, and streaming flag. For the incident assistant, this small fixture makes the workload concrete without pretending to be a production result:

Warm the model, tokenizer, caches, and scheduler before recording steady-state runs. Sweep concurrency to expose queueing, then use a bounded request-rate sweep to find the saturation point. Keep the trace, arrival process, warmup rule, and stop condition fixed across candidates. If an offered rate outruns completion, an unbounded client queue can make every later latency number meaningless.

Record five parts with every result: the GPU model and count, interconnect and topology; engine, driver, CUDA, and model versions; precision, quantization, batching, and parallelism settings; the exact baseline configuration; and a correctness check. The baseline must receive the same trace, warmup, concurrency, and tuning effort. Correctness can combine deterministic token or logit checks where available with a golden request set that verifies citations, tool calls, refusal behavior, and output schema. A faster run that changes the answer contract isn't an improvement.

Report the full curve, not its best point: p50/p95/p99 TTFT, p95/p99 ITL, request-mean TPOT, completed requests per second, error and rejection rates, KV utilization, and cost per successful request. At each offered rate, show the fraction meeting the declared TTFT and decode SLOs. The highest sustainable rate that reaches the required attainment is the goodput point. This evidence tells you whether a new GPU, quantization setting, batch limit, or disaggregated data plane fixed the bottleneck.

GPU infrastructure

Sizing and selection

Turn that request shape into a hardware budget. Selecting the right GPU depends on model size, quantization, and expected traffic. The dominant factors are GPU memory capacity (often called VRAM; datacenter accelerators such as H100 use HBM) to fit weights plus KV cache, and memory bandwidth to serve tokens quickly.

To estimate memory requirements for capacity, use this formula:

$$\text{Memory} \approx \text{Weights} + \text{Growing KV} + \text{Hybrid State} + \text{Runtime Overhead}$$

Where:

• $\text{Weights} \approx \text{Params} \times \text{Precision (bytes)}$ (e.g., Qwen3.6-27B at BF16 is about $27B \times 2$ bytes = 54 GB)^{[15]Reference 15Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}
• $\text{Growing KV} \approx 2 \times \text{Full-Attention Layers} \times \text{KV Heads} \times \text{Head Dim} \times \text{Seq Len} \times \text{Concurrency} \times \text{Precision}$ (the factor of 2 accounts for both the key and value tensors)
• $\text{Hybrid State}$ covers fixed-size recurrent or linear-attention state per active request. It doesn't grow with sequence length, but it still consumes memory and must be added separately.

Two distinctions matter. The older shorthand using Hidden Dim assumes full multi-head attention. Many modern decoder models use grouped-query attention (GQA) or multi-query attention (MQA), so the number of KV heads is much smaller than the total attention head count. Hybrid models also mix full attention with linear or recurrent layers. Only full-attention layers append growing K/V tensors; linear layers keep a different state shape.

The function below estimates a memory-fit lower bound from model size and cache expectations. It takes the model size plus the architecture terms that control KV growth (num_kv_layers, num_kv_heads, head_dim, context length, and concurrency), then reports the minimum count of several GPU types by memory alone. Interconnect topology, kernel support, and measured latency still decide whether a configuration is deployable:

Qwen3.6-27B has 64 total language layers, but its published configuration alternates 48 linear-attention layers with 16 full-attention layers.^{[15]Reference 15Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Only those 16 full-attention layers contribute sequence-growing KV in this formula. The example separately adds about 0.15 GB of fixed recurrent matrix state per active request as a teaching lower bound; the real runtime may retain additional convolution state, allocator metadata, and temporary buffers. This low-concurrency 8k budget fits one H100 by the simplified memory check, but longer context and measured headroom still decide deployment.

Why does the growing-KV formula use num_kv_heads and only full-attention layers?

Answer

GQA and MQA share key/value heads across multiple query heads, so KV cache stores K/V for the smaller head count. In a hybrid model, only full-attention layers append those tensors at every token. Linear-attention layers keep separate fixed-size recurrent state, which belongs in another budget term.

As a reference, the following table separates sequence-growing KV from other recurrent state. The values are planning estimates, not vendor benchmarks. Precision, runtime layout, context policy, and multimodal state can move them.

The MoE rows are where fit conversations often go wrong. HBM residency includes the full expert checkpoint. Per-token decode traffic is different: it includes shared dense tensors plus the experts routed for that token, not every inactive expert byte. Keep residency math, active-byte bandwidth, attention state, and collective traffic separate. Cluster-scale models with million-token context or hundreds of experts need those same four budgets, plus a topology-specific runtime benchmark, before you treat any SKU as a fit.

Why can Qwen3.6-35B-A3B look cheap from active parameters but still be a tight one-GPU plan?

Answer

The A3B suffix is about activated parameters per token, not total checkpoint residency. The full expert pool, KV cache, runtime buffers, prefix-cache headroom, and traffic spikes still count against HBM.

Multi-Instance GPU (MIG)

For cost-efficient autoscaling of smaller models, Multi-Instance GPU (MIG) can partition a single GPU into hardware-isolated slices. Ampere- and Hopper-class parts such as A100, H100, and H200 can expose up to seven instances on supported SKUs.^{[20]Reference 20Supported GPUshttps://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-gpus.html} Instead of dedicating an entire large GPU to a single 7B model, you can run multiple replicas on isolated slices with dedicated compute and memory resources.

Find the decode ceiling

For low-batch decode of large decoder models, weight reads commonly make the system memory-bandwidth bound. Batching, cache traffic, kernels, and multi-GPU communication can change which limit dominates.

The theoretical maximum throughput ($TPS_{max}$) for a batch size of 1 is roughly (a simplified model derived from memory-bandwidth analyses such as Pope et al.)^{[21]Reference 21Efficiently Scaling Transformer Inference.https://arxiv.org/abs/2211.05102}:

$$TPS_{max} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$$

Reading the formula

In this simplified batch-size-one model, the GPU streams the relevant active weight footprint from HBM for each generated token. Little weight reuse happens across sequential steps (the KV cache reuses activations, not the static weights). For a dense model, the active footprint is approximately the full model. For an MoE model, use shared weights plus routed expert bytes measured for the token; full checkpoint residency isn't the decode denominator.

This is the reciprocal of the time required to load the relevant weight footprint from HBM once. An H100 SXM (3.35 TB/s HBM bandwidth) serving Qwen3.6-27B in BF16 (about 54 GB of dense weights) yields an upper bound of roughly 62 tokens/sec for a single sequence.^{[22]Reference 22H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/[15]Reference 15Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Real systems achieve lower throughput because:

• KV cache reads/writes, activations, and sampling also consume memory bandwidth
• Kernel launch, attention computation, and scheduler overhead
• The achieved bandwidth of fused kernels is rarely 100 % of the theoretical peak

This bound explains why quantization can improve decode throughput: halving only the weight-footprint term (FP16 to INT8, for example) doubles this simplified ceiling. Actual speedup is smaller or different when kernel support, KV-cache traffic, compute, or communication dominate. It also shows why large models may require tensor parallelism: the denominator grows while per-GPU bandwidth stays fixed.

The next calculation is deliberately a ceiling, not a benchmark. It isolates weight traffic so you can see what quantization changes before adding runtime overhead:

Why does weight quantization often improve decode speed besides memory footprint?

Answer

Decode repeatedly streams model weights from HBM. Smaller weights reduce bytes loaded per token, raising the memory-bandwidth upper bound for tokens per second.

Tensor parallelism

For models that don't fit comfortably on a single GPU, or that fit weights but not the target KV-cache and headroom budget, we use Tensor Parallelism (TP). TP splits individual weight matrices (e.g., $W_Q, W_K, W_V$) across multiple GPUs so each device computes one shard. This technique was pioneered in the Megatron-LM training system^{[23]Reference 23Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.https://arxiv.org/abs/1909.08053} and adapted for inference serving.

TP operates by slicing matrix computations, so participating GPUs exchange partial results through collective communication within each transformer layer. In the standard Megatron-style transformer block, budget two collectives per layer: one around the attention output projection and one around the feed-forward output projection. Exact collective type and placement depend on row/column partitioning and runtime, but counting only one synchronization per layer understates communication.

• Intra-node: TP works best when GPUs share a very fast fabric such as NVLink (NVIDIA high-bandwidth interconnect) or NVSwitch (NVIDIA multi-GPU switch).
• Inter-node: TP can extend across RDMA (Remote Direct Memory Access) or InfiniBand, but the communication tax rises quickly. For multi-node serving, teams often keep TP within a node when they can, and use Pipeline Parallelism (PP) or independent replicas across nodes instead. PP only communicates activations at stage boundaries, but it introduces pipeline bubbles and can worsen tail latency.

When should you prefer tensor parallelism within a node over spreading one model across nodes?

Answer

Prefer intra-node tensor parallelism when the GPUs share fast NVLink or NVSwitch and the model needs matrix shards on every layer. Crossing nodes adds communication latency; use pipeline parallelism or independent replicas when inter-node fabric would dominate tail latency.

This diagram shows where bytes move in each split:

Communication frequency isn't total cost: tensor-parallel messages may travel over faster links, while pipeline stages can create bubbles. This toy count makes the first screening question explicit:

Autoscaling strategies

Once the engine is measured, autoscaling must answer two different questions: are requests waiting for a ready worker, and is there enough GPU hardware to place another worker? CPU-based autoscaling alone misses the first question because CPU isn't usually the scarce inference resource. CPU can still reveal overloaded gateways or tokenizers. GPU utilization can also be misleading: a GPU might be busy with a healthy batch, or it might be memory-bound while compute duty cycle looks modest.

Key metrics

The most reliable metrics for autoscaling come from two sources: NVIDIA's DCGM (Data Center GPU Manager) for hardware-level visibility, and the serving framework itself for application-level signals. In vLLM, for example, the Prometheus endpoint exposes metrics such as vllm:num_requests_waiting, vllm:num_requests_running, vllm:kv_cache_usage_perc, vllm:time_to_first_token_seconds, and vllm:inter_token_latency_seconds.^{[4]Reference 4Metricshttps://docs.vllm.ai/en/stable/design/metrics/} Some dashboards label that last histogram TPOT. Treat the samples as ITL gaps; compute request-mean TPOT separately if your decode SLO follows DistServe.

DCGM (Data Center GPU Manager) exposes hardware-level GPU metrics like temperature, power, clock rates, engine activity, and memory traffic. While GPU utilization seems like an obvious scaling signal, it isn't a complete serving-capacity signal. High activity can reflect useful work or a bottleneck elsewhere in the memory path; low activity can reflect sparse or bursty work. Neither tells you how long requests are waiting or how many KV blocks remain schedulable.

That's why application-level metrics (queue depth, KV cache utilization, TTFT, ITL, TPOT) are more reliable scaling signals. They directly measure capacity constraints rather than hardware activity. Alert on both mean TPOT and the ITL histogram: a healthy request-mean can hide a few long gaps that still break streaming. Capacity success is goodput under those dual latency objectives, not GPU busy percent. The exact thresholds are workload-specific, so treat any numbers you see in dashboards or sample code as starting points, not universal defaults.

Which metric should page you first: average GPU utilization or sustained request queue depth?

Answer

Sustained request queue depth. It means users are waiting before admission. GPU utilization is supporting context, but it doesn't tell you whether the service has enough schedulable KV capacity for current demand.

KEDA and Karpenter integration

One control-loop design works as follows: the serving framework exposes metrics via Prometheus. KEDA can activate a scaled workload from zero and configure the generated HPA for running replica scaling. If the scheduler can't place new GPU pods, the node autoscaler layer (often Karpenter or Cluster Autoscaler) provisions nodes that satisfy pod requirements.^{[1]Reference 1Scaling Deployments, StatefulSets & Custom Resourceshttps://keda.sh/docs/2.19/concepts/scaling-deployments/[2]Reference 2Conceptshttps://karpenter.sh/docs/concepts/}

This two-layer approach separates desired pod count from hardware supply. Ready pods may scale quickly if spare GPU capacity exists; new nodes and model loading can take far longer. The useful warm buffer is a measured tradeoff between idle GPU cost and the latency damage of a cold burst.

Why is GPU autoscaling a two-loop problem?

Answer

Replica count changes quickly, but GPU node capacity appears slowly. Pod scaling decides desired serving capacity; node autoscaling supplies hardware only after unschedulable GPU pods reveal the cluster lacks the right nodes.

The autoscaling loop should separate replica pressure from hardware supply:

In a real Kubernetes deployment, KEDA and HPA compute the desired replica count for you. This Python snippet is a mental model for the control logic. It takes the current metrics and replica count as inputs, then returns the next target replica count. The thresholds are illustrative only:

The autoscaler determines a target; it doesn't erase startup time. This capacity calculation separates the eventual replica count from the traffic a warm buffer can accept immediately:

A worked example: scaling from zero

A concrete scenario gives the scaling policy something to answer. You're hosting an 8B developer-assistant model for incident summaries and deploy reviews. Traffic is nearly zero at night, but at 9 AM it jumps to 100 concurrent users. How do you protect the user experience without paying for a full night-time fleet?

Start with what users feel, not CPU percentage. A GPU worker can show little CPU pressure while its weights and KV blocks constrain admission. Watch request queue depth, queue age, KV cache utilization, and latency SLOs; retain CPU signals for gateway or preprocessing failures.

Next turn a benchmark into a capacity point. Suppose one GPU handles about 5 concurrent requests while keeping TPOT under 50 ms per token. That measured limit becomes the target capacity per replica, rather than a guessed utilization threshold.

The arithmetic is then simple: 100 concurrent requests divided by 5 requests per GPU requires 20 GPUs. If one instance is warm at 8:59 AM, the autoscaler must add 19 replicas. That is eventual capacity, not instant capacity.

The missing variable is readiness time. Node provisioning plus model loading can outlast the traffic's latency budget, so measure that path on your platform. If the morning rush is predictable, start warming nodes far enough ahead of the observed ready time, or keep a benchmarked buffer pool to absorb its first wave.

If one replica handles 5 concurrent requests and traffic jumps to 100 concurrent requests, what must scaling plan include beyond "set max replicas to 20"?

Answer

It must cover timing. The fleet needs about 20 replicas, but cold nodes may take minutes to provision and load weights, so use predictive warmup or a warm buffer before the spike reaches the queue.

Scaling only when latency spikes is late. By the time TTFT breaches your SLO, users are already frustrated. Scale on queue depth trends rather than lagging latency indicators alone.

Handling cold starts

Cold starts in GPU serving hurt more than cold starts in traditional microservices. A typical web container might start in seconds; a GPU workload often has to provision a node, pull a large image, fetch model weights, and warm up the runtime before it can accept traffic.

Compare a warm GPU (already loaded and serving) with a cold GPU (starting from scratch):

The cold start has four distinct phases, each with its own mitigation strategy. The ranges below are illustrative planning inputs, not platform guarantees; record your own p50 and p95 timings:

Container streaming lets the container start executing before the entire image is downloaded. Only the required layers are fetched on demand, which cuts startup time for large inference images.

Model caching keeps frequently used model weights on high-speed local NVMe storage rather than pulling from remote object storage. For multi-node setups, shared network volumes like Amazon FSx for Lustre can serve weights at high speed to multiple nodes.

Some platforms also offer snapshot or restore features for a preloaded container or VM. Those can help, but they're more vendor-specific than warm pools plus local weight caches.

To mitigate cold start latency impact on users, engineers implement several proactive strategies:

• Over-provisioning: Keep a small headroom buffer to absorb sudden traffic spikes while new instances spin up.
• Predictive scaling: Use historical traffic patterns to scale up the cluster before anticipated usage spikes (e.g., scaling up just before the morning rush hour).
• Minimum warm pool: For high-priority workloads, retain enough measured warm capacity to protect the initial-burst TTFT target while cold replicas load.

You can turn measured phase durations into a readiness decision. In this example, predictive scaling four minutes before a known incident-review surge is sufficient for the measured path, while scaling two minutes before it's not:

Which cold-start phase should you optimize if pods are scheduled quickly but first token still takes minutes?

Answer

Look at container pull, model weight fetch, and runtime warmup. Fast scheduling only proves a node exists; the model still needs image layers, weights, and readiness warmups before serving traffic.

Cost optimization

Measured capacity makes cost a decision boundary, not a guess. Running top-end GPUs 24/7 is expensive, and idle replicas burn budget fast. Keep the latency contract intact while choosing warm capacity, interruption risk, and platform ownership:

Spot instances and fault tolerance

Cloud providers offer spot instances at discounts, but they come with preemption risk. Inference can be easier to retry than long-running training, but reasoning runs and long generations may exceed a provider's warning window. Use measured request durations, a drain deadline, and retry behavior before assigning traffic to spot capacity.

For a code assistant, predictable launch-day or migration-week spikes can be candidates for spot overflow capacity when a stable baseline pool meets the promised SLO and interrupted work can drain or retry cleanly.

• Graceful shutdown: On SIGTERM, remove the replica from new admission, let completions that fit the remaining drain budget finish, and retry or fail over requests that can't finish before termination.
• Mixed pools: A common policy retains on-demand or reserved capacity for baseline SLO traffic and uses an autoscaling spot pool only for retryable overflow.

Why are spot GPUs often acceptable for inference but risky without drain logic?

Answer

Some inference requests finish within a preemption window or can be retried. Without drain logic, the load balancer keeps sending new work to a terminating node and active long generations are cut off without an explicit recovery path.

The drain policy must compare remaining work with a deadline rather than assume all requests finish. This example reserves twenty seconds for shutdown and transfer after a 120-second termination warning:

Scale-down cooldown & thrashing

A common (and expensive) mistake is scaling down too aggressively. If you terminate a GPU node because traffic dipped for 30 seconds, you'll pay the cold start penalty again when traffic returns a minute later. This "thrashing" can increase costs while degrading user experience.

It's like shutting down a warm inference replica because the queue cleared for 30 seconds, only to pay model-load latency again when the next burst arrives.

Scale-down rules:

• Cooldown period: Start with a cooldown longer than short observed lulls and tune it against measured cold-start cost, idle spend, and recurring traffic patterns.
• Drain before terminate: Give the autoscaler time to drain in-flight requests before removing a replica. Don't kill a GPU mid-generation.
• Scale down gradually: Remove one replica at a time and wait to see if queue depth remains low before removing more.

Why can aggressive scale-down increase both cost and latency?

Answer

It repeatedly destroys warm capacity, then pays node provisioning and model loading again when traffic rebounds. The service burns startup time and money without serving useful requests during the cold-start gap.

Deployment options

When deciding how to deploy GPU inference, teams face a build-vs-buy decision:

Serverless platforms like Modal and RunPod abstract away the Kubernetes layer entirely. They typically handle autoscaling and much of the instance lifecycle for you. This is ideal for teams without dedicated ML infrastructure engineers or for workloads with highly variable traffic.

Specialized GPU clouds sit in the middle. You usually get raw instances, storage, or managed Kubernetes primitives without the full DIY burden of the hyperscalers. Managed Kubernetes still gives you the most control over the full stack (vLLM versions, custom schedulers, quantization methods) and can be cheaper once utilization is high and predictable enough to pay back the platform engineering overhead.

Diagnose a stalled spike

When the incident assistant leaves a laptop demo and hits a freeze-driven queue, read each symptom as evidence about a different control point. The fixes below connect the visible failure to its likely cause and next measurement.

Flapping: scaling up and down too fast

• Symptom: Your GPU node count oscillates wildly. Cloud bills spike, but user latency doesn't improve. The autoscaler logs show scale-up events followed by scale-down events within minutes.

• Cause: The autoscaler reacts to short traffic blips instead of sustained trends. A GPU node that just finished loading weights gets terminated before it serves enough requests to justify its startup cost.

• Fix: Choose a cooldown from traffic and startup measurements rather than a universal duration. Scale down gradually, and require queue depth near zero, low KV cache usage, and low running requests before removing capacity.

Ignoring tail latency

• Symptom: Your dashboard shows an average response time of 800 ms, but users complain about 30-second waits. The p95 or p99 latency is an order of magnitude worse than the mean.

• Cause: Average latency hides the users with long prompts or the batches where one straggler request keeps the GPU occupied. Static batching makes this worse, but even continuous batching can suffer if a single request with a 4,000-token prompt monopolizes a slot.

• Fix: Monitor TTFT and TPOT at the p95 or p99 percentile, not the mean. Set SLOs on tail latency. If long prompts are common, enable chunked prefill so they don't starve shorter decode requests.

No explicit queue or backpressure

• Symptom: Requests keep arriving even after the GPU workers are full. The service eventually times out randomly, and retry storms make the spike worse.

• Cause: The system accepts more work than the serving engine can schedule. Without a visible queue, admission policy, timeout budget, and retry contract, overload becomes invisible until users see failures.

• Fix: Put a queue or gateway in front of the inference engine. Track queue depth, queue age, and rejection rate. Return controlled overload responses before the GPU fleet collapses.

The failure path is an explicit reject, not a timeout after the KV pool is already full:

A 429 with a retry budget is healthier than letting retries pile onto a saturated replica.

Treating every request as equal priority

• Symptom: A batch analytics job slows down interactive chat, or one tenant's long prompts make all tenants miss their TTFT target.

• Cause: The scheduler sees all requests as identical even though some workloads are interactive, some are batch, and some have stricter contractual latency limits.

• Fix: Separate traffic classes. Use priority queues, per-tenant rate limits, maximum prompt and generation budgets, and different pools when workload shapes are too different to share one scheduler fairly.

Underestimating long-context KV growth

• Symptom: A model that passed a short-prompt load test starts rejecting work or swapping under real chat history. GPU memory looks fine at startup and then collapses as conversations lengthen.

• Cause: The capacity plan counted weights but not enough KV cache. Long prompts, high concurrency, prefix-cache headroom, and larger generation limits all consume blocks during the request lifetime.

• Fix: Size with realistic prompt and output distributions, not maximum model weights alone. Load test with long conversations, watch vllm:kv_cache_usage_perc, and apply admission limits before the cache pool hits saturation.

Confusing PagedAttention with prefix caching

• Symptom: A team enables PagedAttention and expects repeated system prompts to become free, then sees no TTFT improvement on new replicas.

• Cause: PagedAttention is an allocation strategy for KV blocks. Prefix caching is a reuse strategy for shared prompt prefixes. One reduces fragmentation; the other avoids recomputing prefix KV state.

• Fix: Use both when the workload benefits from both. Treat PagedAttention as baseline memory management and prefix caching as a workload-specific optimization that needs stable shared prefixes and warm caches.

Cache blindness during scale-up

• Symptom: You scale from 2 to 10 GPUs during a traffic spike, but the new nodes serve requests slower than the old ones. TTFT increases right after scaling.

• Cause: The original nodes have been running for hours and have accumulated prefix cache hits (for example, a shared system prompt that every request includes). The brand-new nodes start with empty caches. They must recompute the full prefill from scratch, so their first tokens take much longer.

• Fix: Warm new nodes with a few synthetic requests that populate the common prefix before adding them to the load balancer rotation. Alternatively, use vLLM's prefix caching and send new nodes "seed" traffic to warm their KV cache before taking full production load.

Why might new replicas make TTFT worse right after a scale-up?

Answer

Old replicas may have hot prefix caches, while new replicas start cold. If traffic shifts immediately, the new replicas recompute shared prompt prefixes and first-token latency rises despite higher replica count.

When the baseline isn't enough

The baseline handles many fleets. The options below change the data plane or the failure boundary, so measure the simpler path first and add one only when the trace names a bottleneck.

Prefill-decode disaggregation

Prefill and decode stress GPUs differently. Prefill processes many prompt tokens in parallel and tends to be compute-heavy. Decode produces one token per active request and tends to be memory-bandwidth-heavy. If you run both phases on the same worker pool, a few long prompts can delay short streaming responses, and decode traffic can leave tensor cores underused.

Prefill-decode disaggregation, studied for production serving by Splitwise and DistServe, splits the serving fleet into two pools:^{[24]Reference 24Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[14]Reference 14DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

• Prefill workers ingest prompts, build the initial KV cache, and hand off cache state.
• Decode workers continue token generation for active requests with tighter scheduling around TPOT.

This isn't free. You now need cache transfer, placement logic, and backpressure between pools. But it can help high-traffic systems where long prompts and streaming generations compete for the same GPU budget. The autoscaling signals also become more specific: prefill workers scale on prompt-token backlog and TTFT, while decode workers scale on active sequences, KV-cache utilization, and TPOT.

Follow one request across the data plane

Disaggregation changes request ownership, not only replica count. The frontend or router first chooses a compatible prefill worker, using load and, when useful, prefix or cache locality. That worker processes the prompt and builds the initial KV blocks. A transfer layer then moves those blocks and the metadata needed to interpret them to a decode worker. Both sides must agree on model, dtype, block size, and KV layout, or the handoff can't be trusted.^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/}

The decode worker admits the sequence, runs token steps, and streams the answer. Cancellation and timeout must release state on both sides, while backpressure must reach the router before either pool accumulates work it can't finish. Placement matters too: a split that sends large KV transfers across a slow link can give back the goodput it was meant to recover.

NVIDIA Dynamo is one documented open-source control-plane example. Its documentation describes disaggregated prefill and decode deployments across backends including vLLM, SGLang, and TensorRT-LLM, with KV-aware routing and KV-transfer mechanisms for split deployments.^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/} This makes it a useful implementation to study, but the operational win still depends on measured prompt mix, transfer cost, and cache-hit behavior.

Don't start here. First tune continuous batching, chunked prefill, prefix caching, and queue policy. Disaggregation is a later optimization when one mixed pool can no longer hit both TTFT and TPOT targets, which is the same dual-SLO condition that defines goodput under load.

When does prefill-decode disaggregation become worth the extra system complexity?

Answer

Use it when long prompts and streaming decode traffic fight inside one worker pool after simpler fixes are exhausted. It helps when separate prefill and decode autoscaling signals are needed to restore goodput (request rate under both TTFT and decode SLOs).

Use measurements, not architecture fashion, to decide whether to split pools. This gate recommends investigation only when a tuned mixed pool breaches both service objectives under a long-prompt-heavy trace:

Speculative decoding

Speculative decoding speeds up generation by using a smaller "draft" model to predict multiple tokens ahead, then verifying them with the larger "target" model. Leviathan et al. report roughly 2-3x acceleration in their evaluated settings, not a universal serving guarantee.^{[25]Reference 25Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

The speedup comes from amortizing target-model work across several draft tokens. The draft model's predictions are treated as hypotheses; the target model evaluates them in one verification pass and keeps the valid prefix. When the draft model is reasonably accurate for the traffic pattern, this reduces the number of expensive target forward passes. It still pays draft-model compute and rejection overhead, so you measure it against your own workload instead of assuming it always helps.

For example, you might pair a small draft model with a Qwen3.6-27B target to accelerate decode-heavy traffic. The draft model generates candidate tokens quickly; the target model then verifies all candidates in a single forward pass, accepting all correct predictions up to the first mismatch.

When is speculative decoding most likely to help a serving fleet?

Answer

It helps decode-heavy, memory-bandwidth-bound workloads when a cheap draft model predicts a high fraction of target-model tokens correctly. It's less useful when prefill, retrieval, queueing, or network latency dominates.

Multi-region failover

GPU quota and available capacity vary by region and time. If the primary region is at capacity or unhealthy, the incident assistant still needs a path to a secondary region.

This requires:

• Global load balancing: Route requests to the nearest healthy region
• Model sync: Keep model weights and LoRA adapters synchronized across regions
• Failover logic: Detect regional capacity constraints and automatically shift traffic

Regional failover is particularly important for spot instance workloads. If a regional spot fleet is reclaimed, your tested recovery-time objective determines whether secondary on-demand capacity can receive new requests before client timeout budgets expire. Long in-flight generations still need retry or resumption semantics.

What must be synchronized before multi-region failover is safe for tenant-specific models?

Answer

Synchronize base model versions, LoRA adapter versions, tokenizer/config files, safety policy versions, and routing metadata. Otherwise failover can silently serve a different model behavior than the primary region.

Multi-tenancy with LoRA

Instead of deploying a separate Qwen3.6-27B base for every customer fine-tune, use one shared base model and select LoRA (Low-Rank Adaptation)^{[26]Reference 26LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} adapters per request. Systems like S-LoRA^{[27]Reference 27S-LoRA: Serving Thousands of Concurrent LoRA Adapters.https://arxiv.org/abs/2311.03285} show how to batch requests across different adapters while keeping the base model weights shared in GPU memory. vLLM also supports per-request LoRA serving, with explicit warnings around separate runtime adapter-loading APIs in untrusted environments.^{[28]Reference 28LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/}

To implement multi-tenancy, map each tenant to an approved adapter and select that adapter per request. TenantAdapterRouter shows how a single base model instance can serve requests for different tenants by passing a LoRARequest to vLLM. While a true production environment would use vLLM's AsyncLLMEngine to handle concurrent requests without blocking, the synchronous LLM class is shown here for conceptual clarity. The function takes an incoming request containing a tenant ID, looks up the approved adapter path, and outputs the generated response with the selected adapter. This example doesn't expose vLLM's runtime adapter-loading APIs; those are a separate trusted-environment feature:

Prefer the async server path over a synchronous wrapper like this, and don't expose arbitrary adapter loading to untrusted tenants.^{[28]Reference 28LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/}

What is the main safety risk in LoRA multi-tenancy?

Answer

Runtime adapter loading can become a code and data isolation boundary. Only load approved adapter paths, bind tenant IDs to authorized adapters, version adapters immutably, and avoid accepting arbitrary adapter locations from requests.

Choose the next capacity change

The release-freeze spike now has a decision path. Trace a request from arrival through queueing, prefill, cache allocation, decode, and streaming. Then measure the same path under a realistic arrival process, with declared TTFT and decode SLOs.

If the queue grows before the first token, inspect admission, cold readiness, prefill, and cache pressure. If token gaps grow after streaming starts, inspect decode bandwidth, active sequences, and batching. A Kubernetes pattern using KEDA/HPA, a GPU node autoscaler, and a serving engine can act on those signals, but only after measured capacity gives it a meaningful threshold.

Continuous batching and PagedAttention improve GPU use, yet they don't remove cold starts or capacity variance between nodes. Choose warm capacity, quantization, a batch limit, or a split prefill/decode data plane only when the evidence shows which constraint moved. That keeps a thousand-user spike inside its latency and budget gates.