GPU Serving & Autoscaling

Advanced MLOps and DevOps for AI gave you the release machinery: promotion gates, lineage, rollout policy, and rollback triggers. Now apply that machinery to the inference fleet. GPU serving and autoscaling decide whether a promoted model can survive real traffic without burning budget or missing latency goals.

A developer assistant that becomes the default way to summarize incidents, review diffs, and explain failing deploys can overload quickly. At 8 AM, ten people are using it. By 9 AM, a release freeze sends a thousand requests through the same endpoint. If you try serving the model like a normal web API, it'll quickly grind to a halt. A web server fetches pre-computed data and forgets the user immediately. An LLM has to process every prompt and retain attention state as generation proceeds. That short-term memory is a central capacity constraint alongside compute, memory bandwidth, and queueing delay.

Standard web servers don't maintain growing state for each active user. An LLM does. To generate the next token, the model must access computed attention states from previous tokens. In LLMs, this state is the KV cache, and managing its memory footprint is one of the central challenges of production serving.

Engineers designing serving systems optimize for two latency metrics. Teams usually attach service-level objectives (SLOs), target boundaries for acceptable user experience, to each:

1. Time To First Token (TTFT): The latency before the user sees the first word. This is dominated by the prefill phase (processing the input prompt) and is often compute-bound.
2. Time Per Output Token (TPOT), also called Time Between Tokens (TBT): The latency between subsequent tokens. This determines the perceived generation speed. This is the decode phase and is usually memory-bandwidth bound for large decoder models.

The Key-Value (KV) cache (the stored attention computations for past tokens) grows dynamically with every generated token. Poor memory management leads to fragmentation, which forces smaller batch sizes and lower throughput.

GPU serving mental model

To see why GPU serving is so different from normal web scaling, keep the serving path concrete:

• The GPU is the scarce execution device. It's specialized, expensive, and you can't replace it with normal web servers.
• The model weights are the resident model state. Loading them takes time, and the worker can't decode until they are in GPU memory.
• The KV cache is per-request attention state. If a conversation started ten turns ago, its cache stays allocated until generation finishes. The cache pool fills up fast during a traffic spike.
• Autoscaling adds another ready worker. Bringing a new worker online takes time, and it can't help while the node, image, weights, and warmup are still loading.

Teams often scale because the GPU is busy. Scale when queue age, queue depth, KV-cache pressure, or latency shows that requests are waiting. A busy GPU can mean you're using the hardware well. A growing queue means users are waiting, and that's when you need more capacity.

The GPU serving stack

Start with one common production GPU serving architecture before choosing a framework. It consists of several layers:

• Infrastructure layer: Kubernetes (EKS, GKE, or AKS) manages the GPU nodes. Unlike CPU nodes, GPU capacity is expensive and sometimes scarce, so a warm pool must be justified against latency targets and measured demand.

• Control layer: In a KEDA-based setup, KEDA (Kubernetes Event-Driven Autoscaling) can activate a workload from zero and configure a Horizontal Pod Autoscaler (HPA) to scale running replicas from metrics. Karpenter (AWS) or Cluster Autoscaler can then provision GPU nodes when new pods can't be scheduled on existing capacity.^{[1]Reference 1Scaling Deployments, StatefulSets & Custom Resourceshttps://keda.sh/docs/2.19/concepts/scaling-deployments/[2]Reference 2Conceptshttps://karpenter.sh/docs/concepts/}

• Serving layer: The actual inference engines (vLLM, SGLang, TensorRT-LLM) that load models into GPU memory and handle token generation. At datacenter scale, a control plane such as NVIDIA Dynamo can orchestrate disaggregated prefill/decode pools across many of these engine replicas.^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/}

• Metrics: CPU and RAM metrics alone don't describe LLM-serving capacity. NVIDIA's DCGM (Data Center GPU Manager) exposes GPU-specific telemetry, while user-facing scaling signals usually come from the serving layer: queue depth, KV cache usage, TTFT, and inter-token latency.^{[4]Reference 4Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

Serving frameworks

Now zoom into the serving engine: batching policy, KV-cache allocation, framework choice, and request lifecycle.

The evolution of batching

Early serving systems used static batching: waiting for $N$ requests to arrive, padding them to the same length, and processing them together. It's inefficient because requests finish at different times. If one request generates 100 tokens and another generates 1000, the GPU must continue processing the batch until the longest request finishes, leaving slots idle for the shorter requests.

Modern frameworks use continuous batching (also known as iteration-level scheduling), pioneered by Orca^{[5]Reference 5Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu}. The scheduler operates at the granularity of a single token generation step. As soon as a request finishes, a new request from the queue can be inserted into its slot in the next iteration.

The small simulation below uses generation lengths for four assistant replies. A static batch doesn't refill its two slots until both original replies finish; a continuous scheduler admits the next queued reply as soon as a slot opens.

With the same replies and slots, the continuous scheduler finishes sooner because it wastes fewer slot-iterations.

vLLM and PagedAttention

One widely used open-source serving engine is vLLM, which introduced PagedAttention^{[6]Reference 6Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}. Before vLLM, inference systems pre-allocated a single contiguous block of GPU memory sized to the maximum sequence length a request might ever use. Because the actual length is revealed token-by-token during decoding, this approach created two problems:

• Internal fragmentation: most of the pre-allocated chunk sat empty for short requests.
• External fragmentation: free memory became scattered and couldn't satisfy new large contiguous allocations.

Profiling across real workloads showed that prior systems typically used only 20 to 38% of the KV cache memory they had reserved, wasting 62 to 80%.^{[6]Reference 6Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} The result: the effective batch size stayed small, and throughput suffered even when the GPU had spare VRAM.

PagedAttention treats the KV cache like virtual memory in an operating system (OS). It breaks the KV cache into fixed-size blocks (pages) that can be stored in non-contiguous memory.

Logical vs. physical blocks

PagedAttention introduces two key abstractions:

• Logical KV blocks: The view of the KV cache from the model's perspective. It appears as a contiguous sequence of tokens.
• Physical KV blocks: The actual fixed-size chunks of memory allocated in HBM (High Bandwidth Memory, the GPU's ultra-fast RAM) to store these tokens.

A Block Table maintains the mapping between logical and physical blocks. When a new token is generated, vLLM checks if the current physical block has space. If not, it allocates a new physical block from a pre-allocated pool and updates the mapping. The PagedAttention paper reports much tighter memory use and higher throughput because batches can pack around real sequence lengths instead of worst-case reservations.^{[6]Reference 6Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

PagedAttention gives the runtime fine-grained control over KV blocks. In current vLLM deployments, request-to-request reuse of a shared prompt prefix usually comes from automatic prefix caching, which reuses previously computed KV blocks when a new request shares the same prefix.^{[7]Reference 7Automatic Prefix Cachinghttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/}

The allocation arithmetic for four conversations makes the waste visible. The fixed reservation policy reserves each request for its maximum allowed length, while paging reserves blocks only for tokens currently stored:

Paging doesn't make attention state free. It stops short conversations from holding blocks for a maximum length they haven't used.

The overall request lifecycle coordinates these components. Client requests enter an API server queue, are packed by a continuous batching scheduler, and finally executed by a model engine using PagedAttention and, when enabled, prefix caching on the GPU cluster:

Framework comparison

Choosing the right serving engine depends on your specific constraints:

The TGI documentation describes TGI as maintenance mode and directs users toward newer optimized engines such as vLLM and SGLang for new deployments.^{[10]Reference 10Text Generation Inference.https://huggingface.co/docs/text-generation-inference/index} These engines are the data plane that loads weights and generates tokens. At datacenter scale, a separate control plane such as NVIDIA Dynamo can sit above them to provide disaggregated prefill/decode scheduling and KV-aware routing across engine replicas (covered later under disaggregation).^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/}

To see how this works in practice, here's an example of initializing vLLM with settings you would benchmark before deployment. The configuration below loads a large model across multiple GPUs using tensor parallelism and sets a high memory-utilization budget for weights, KV blocks, and runtime buffers. It also uses chunked prefill so long prompts can share scheduling budget with decode traffic.^{[13]Reference 13Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html} It takes the model identifier and hardware constraints as inputs, and outputs an initialized engine ready to accept inference requests:

All of these knobs are workload-dependent. Tune them against TTFT and TPOT SLOs, rather than raw tokens/sec alone.

GPU infrastructure

Sizing and selection

Selecting the right GPU depends on model size, quantization, and expected traffic. The dominant factors are GPU memory capacity (often called VRAM; datacenter accelerators such as H100 use HBM) to fit weights plus KV cache, and memory bandwidth to serve tokens quickly.

To estimate memory requirements for capacity, use this formula:

$$\text{Memory} \approx \text{Weights} + \text{Growing KV} + \text{Hybrid State} + \text{Runtime Overhead}$$

Where:

• $\text{Weights} \approx \text{Params} \times \text{Precision (bytes)}$ (e.g., Qwen3.6-27B at BF16 is about $27B \times 2$ bytes = 54 GB)^{[14]Reference 14Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}
• $\text{Growing KV} \approx 2 \times \text{Full-Attention Layers} \times \text{KV Heads} \times \text{Head Dim} \times \text{Seq Len} \times \text{Concurrency} \times \text{Precision}$ (the factor of 2 accounts for both the key and value tensors)
• $\text{Hybrid State}$ covers fixed-size recurrent or linear-attention state per active request. It doesn't grow with sequence length, but it still consumes memory and must be added separately.

Two distinctions matter. The older shorthand using Hidden Dim assumes full multi-head attention. Many modern decoder models use grouped-query attention (GQA) or multi-query attention (MQA), so the number of KV heads is much smaller than the total attention head count. Hybrid models also mix full attention with linear or recurrent layers. Only full-attention layers append growing K/V tensors; linear layers keep a different state shape.

Here's a practical Python function that estimates a memory-fit lower bound from model size and cache expectations. It takes the model size plus the architecture terms that control KV growth (num_kv_layers, num_kv_heads, head_dim, context length, and concurrency), then reports the minimum count of several GPU types by memory alone. Interconnect topology, kernel support, and measured latency still decide whether a configuration is deployable:

Qwen3.6-27B has 64 total language layers, but its published configuration alternates 48 linear-attention layers with 16 full-attention layers.^{[14]Reference 14Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Only those 16 full-attention layers contribute sequence-growing KV in this formula. The example separately adds about 0.15 GB of fixed recurrent matrix state per active request as a teaching lower bound; the real runtime may retain additional convolution state, allocator metadata, and temporary buffers. This low-concurrency 8k budget fits one H100 by the simplified memory check, but longer context and measured headroom still decide deployment.

As a reference, the following table separates sequence-growing KV from other recurrent state. The values are planning estimates, not vendor benchmarks. Precision, runtime layout, context policy, and multimodal state can move them.

The MoE rows are where fit conversations often go wrong. HBM residency includes the full expert checkpoint. Per-token decode traffic is different: it includes shared dense tensors plus the experts routed for that token, not every inactive expert byte. Keep residency math and active-byte bandwidth math separate.

Multi-Instance GPU (MIG)

For cost-efficient autoscaling of smaller models, Multi-Instance GPU (MIG) can partition a single GPU into hardware-isolated slices. Ampere- and Hopper-class parts such as A100, H100, and H200 can expose up to seven instances on supported SKUs.^{[19]Reference 19Supported GPUshttps://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-gpus.html} Instead of dedicating an entire large GPU to a single 7B model, you can run multiple replicas on isolated slices with dedicated compute and memory resources.

Throughput & bandwidth constraints

For low-batch decode of large decoder models, weight reads commonly make the system memory-bandwidth bound. Batching, cache traffic, kernels, and multi-GPU communication can change which limit dominates.

The theoretical maximum throughput ($TPS_{max}$) for a batch size of 1 is roughly (a simplified model derived from memory-bandwidth analyses such as Pope et al.)^{[20]Reference 20Efficiently Scaling Transformer Inference.https://arxiv.org/abs/2211.05102}:

$$TPS_{max} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$$

Reading the formula

In this simplified batch-size-one model, the GPU streams the relevant active weight footprint from HBM for each generated token. Little weight reuse happens across sequential steps (the KV cache reuses activations, not the static weights). For a dense model, the active footprint is approximately the full model. For an MoE model, use shared weights plus routed expert bytes measured for the token; full checkpoint residency isn't the decode denominator.

$$TPS_{max} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$$

This is the reciprocal of the time required to load the relevant weight footprint from HBM once. An H100 SXM (3.35 TB/s HBM bandwidth) serving Qwen3.6-27B in BF16 (about 54 GB of dense weights) yields an upper bound of roughly 62 tokens/sec for a single sequence.^{[21]Reference 21H100 GPUhttps://www.nvidia.com/en-us/data-center/h100/[14]Reference 14Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Real systems achieve lower throughput because:

• KV cache reads/writes, activations, and sampling also consume memory bandwidth
• Kernel launch, attention computation, and scheduler overhead
• The achieved bandwidth of fused kernels is rarely 100 % of the theoretical peak

This bound explains why quantization can improve decode throughput: halving only the weight-footprint term (FP16 to INT8, for example) doubles this simplified ceiling. Actual speedup is smaller or different when kernel support, KV-cache traffic, compute, or communication dominate. It also shows why large models may require tensor parallelism: the denominator grows while per-GPU bandwidth stays fixed.

The next calculation is deliberately a ceiling, not a benchmark. It isolates weight traffic so you can see what quantization changes before adding runtime overhead:

Tensor parallelism

For models that don't fit comfortably on a single GPU, or that fit weights but not the target KV-cache and headroom budget, we use Tensor Parallelism (TP). TP splits individual weight matrices (e.g., $W_Q, W_K, W_V$) across multiple GPUs so each device computes one shard. This technique was pioneered in the Megatron-LM training system^{[22]Reference 22Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.https://arxiv.org/abs/1909.08053} and adapted for inference serving.

TP operates by slicing matrix computations, so participating GPUs exchange partial results through collective communication within each transformer layer. In the standard Megatron-style transformer block, budget two collectives per layer: one around the attention output projection and one around the feed-forward output projection. Exact collective type and placement depend on row/column partitioning and runtime, but counting only one synchronization per layer understates communication.

• Intra-node: TP works best when GPUs share a very fast fabric such as NVLink (NVIDIA high-bandwidth interconnect) or NVSwitch (NVIDIA multi-GPU switch).
• Inter-node: TP can extend across RDMA (Remote Direct Memory Access) or InfiniBand, but the communication tax rises quickly. For multi-node serving, teams often keep TP within a node when they can, and use Pipeline Parallelism (PP) or independent replicas across nodes instead. PP only communicates activations at stage boundaries, but it introduces pipeline bubbles and can worsen tail latency.

This diagram shows where bytes move in each split:

Communication frequency isn't total cost: tensor-parallel messages may travel over faster links, while pipeline stages can create bubbles. This toy count makes the first screening question explicit:

Autoscaling strategies

CPU-based autoscaling alone is insufficient for LLM serving because CPU isn't usually the scarce inference resource. CPU can still reveal overloaded gateways or tokenizers. GPU utilization can also be misleading: a GPU might be busy with a healthy batch, or it might be memory-bound while compute duty cycle looks modest.

Key metrics

The most reliable metrics for autoscaling come from two sources: NVIDIA's DCGM (Data Center GPU Manager) for hardware-level visibility, and the serving framework itself for application-level signals. In vLLM, for example, the Prometheus endpoint exposes metrics such as vllm:num_requests_waiting, vllm:num_requests_running, vllm:kv_cache_usage_perc, vllm:time_to_first_token_seconds, and vllm:inter_token_latency_seconds.^{[4]Reference 4Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

DCGM (Data Center GPU Manager) exposes hardware-level GPU metrics like temperature, power, clock rates, engine activity, and memory traffic. While GPU utilization seems like an obvious scaling signal, it isn't a complete serving-capacity signal. High activity can reflect useful work or a bottleneck elsewhere in the memory path; low activity can reflect sparse or bursty work. Neither tells you how long requests are waiting or how many KV blocks remain schedulable.

That's why application-level metrics (queue depth, KV cache utilization, TTFT, TPOT) are more reliable scaling signals. They directly measure capacity constraints rather than hardware activity. The exact thresholds are workload-specific, so treat any numbers you see in dashboards or sample code as starting points, not universal defaults.

KEDA and Karpenter integration

One control-loop design works as follows: the serving framework exposes metrics via Prometheus. KEDA can activate a scaled workload from zero and configure the generated HPA for running replica scaling. If the scheduler can't place new GPU pods, the node autoscaler layer (often Karpenter or Cluster Autoscaler) provisions nodes that satisfy pod requirements.^{[1]Reference 1Scaling Deployments, StatefulSets & Custom Resourceshttps://keda.sh/docs/2.19/concepts/scaling-deployments/[2]Reference 2Conceptshttps://karpenter.sh/docs/concepts/}

This two-layer approach separates desired pod count from hardware supply. Ready pods may scale quickly if spare GPU capacity exists; new nodes and model loading can take far longer. The useful warm buffer is a measured tradeoff between idle GPU cost and the latency damage of a cold burst.

The autoscaling loop should separate replica pressure from hardware supply:

In a real Kubernetes deployment, KEDA and HPA compute the desired replica count for you. This Python snippet is a mental model for the control logic. It takes the current metrics and replica count as inputs, then returns the next target replica count. The thresholds are illustrative only:

The autoscaler determines a target; it doesn't erase startup time. This capacity calculation separates the eventual replica count from the traffic a warm buffer can accept immediately:

A worked example: scaling from zero

A concrete scenario exposes the scaling policy. You're hosting an 8B developer-assistant model for incident summaries and deploy reviews. Traffic is nearly zero at night, but at 9 AM it jumps to 100 concurrent users. How do you scale without blowing the budget or the user experience?

Step 1: Pick the right metric. Scaling only on CPU percentage misses engine capacity. A GPU worker can show little CPU pressure while its weights and KV blocks constrain admission. Use request queue depth, queue age, KV cache utilization, and latency SLOs; retain CPU signals for gateway or preprocessing failures.

Step 2: Set a threshold. Through benchmarking, you determine that one GPU can comfortably handle about 5 concurrent requests while keeping TPOT under 50 ms per token. That's your target capacity per replica.

Step 3: Do the math. With 100 concurrent requests and 5 requests per GPU, you need 100 / 5 = 20 GPUs. If you're starting from one warm instance at 8:59 AM, the autoscaler must add 19 more replicas quickly.

Step 4: Solve the cold-start gap. Even if a replica target changes promptly, node provisioning plus model loading can outlast the traffic's latency budget. Measure that ready time on your platform. If the morning rush is predictable, start warming nodes far enough ahead of the observed ready time, or keep a benchmarked buffer pool that absorbs its first wave.

Scaling only when latency spikes is late. By the time TTFT breaches your SLO, users are already frustrated. Scale on queue depth trends rather than lagging latency indicators alone.

Handling cold starts

Cold starts in GPU serving hurt more than cold starts in traditional microservices. A typical web container might start in seconds; a GPU workload often has to provision a node, pull a large image, fetch model weights, and warm up the runtime before it can accept traffic.

Compare a warm GPU (already loaded and serving) with a cold GPU (starting from scratch):

The cold start has four distinct phases, each with its own mitigation strategy. The ranges below are illustrative planning inputs, not platform guarantees; record your own p50 and p95 timings:

Container streaming lets the container start executing before the entire image is downloaded. Only the required layers are fetched on demand, which cuts startup time for large inference images.

Model caching keeps frequently used model weights on high-speed local NVMe storage rather than pulling from remote object storage. For multi-node setups, shared network volumes like Amazon FSx for Lustre can serve weights at high speed to multiple nodes.

Some platforms also offer snapshot or restore features for a preloaded container or VM. Those can help, but they're more vendor-specific than warm pools plus local weight caches.

To mitigate cold start latency impact on users, engineers implement several proactive strategies:

• Over-provisioning: Keep a small headroom buffer to absorb sudden traffic spikes while new instances spin up.
• Predictive scaling: Use historical traffic patterns to scale up the cluster before anticipated usage spikes (e.g., scaling up just before the morning rush hour).
• Minimum warm pool: For high-priority workloads, retain enough measured warm capacity to protect the initial-burst TTFT target while cold replicas load.

You can turn measured phase durations into a readiness decision. In this example, predictive scaling four minutes before a known incident-review surge is sufficient for the measured path, while scaling two minutes before it's not:

Cost optimization

Running top-end GPUs 24/7 is expensive, and idle replicas burn budget fast. To optimize costs without breaking Service Level Agreements (SLAs):

Spot instances and fault tolerance

Cloud providers offer spot instances at discounts, but they come with preemption risk. Inference can be easier to retry than long-running training, but reasoning runs and long generations may exceed a provider's warning window. Use measured request durations, a drain deadline, and retry behavior before assigning traffic to spot capacity.

For a code assistant, predictable launch-day or migration-week spikes can be candidates for spot overflow capacity when a stable baseline pool meets the promised SLO and interrupted work can drain or retry cleanly.

• Graceful shutdown: On SIGTERM, remove the replica from new admission, let completions that fit the remaining drain budget finish, and retry or fail over requests that can't finish before termination.
• Mixed pools: A common policy retains on-demand or reserved capacity for baseline SLO traffic and uses an autoscaling spot pool only for retryable overflow.

The drain policy must compare remaining work with a deadline rather than assume all requests finish. This example reserves twenty seconds for shutdown and transfer after a 120-second termination warning:

Scale-down cooldown & thrashing

A common (and expensive) mistake is scaling down too aggressively. If you terminate a GPU node because traffic dipped for 30 seconds, you'll pay the cold start penalty again when traffic returns a minute later. This "thrashing" can increase costs while degrading user experience.

It's like shutting down a warm inference replica because the queue cleared for 30 seconds, only to pay model-load latency again when the next burst arrives.

Scale-down rules:

• Cooldown period: Start with a cooldown longer than short observed lulls and tune it against measured cold-start cost, idle spend, and recurring traffic patterns.
• Drain before terminate: Give the autoscaler time to drain in-flight requests before removing a replica. Don't kill a GPU mid-generation.
• Scale down gradually: Remove one replica at a time and wait to see if queue depth remains low before removing more.

Deployment options

When deciding how to deploy GPU inference, teams face a build-vs-buy decision:

Serverless platforms like Modal and RunPod abstract away the Kubernetes layer entirely. They typically handle autoscaling and much of the instance lifecycle for you. This is ideal for teams without dedicated ML infrastructure engineers or for workloads with highly variable traffic.

Specialized GPU clouds sit in the middle. You usually get raw instances, storage, or managed Kubernetes primitives without the full DIY burden of the hyperscalers. Managed Kubernetes still gives you the most control over the full stack (vLLM versions, custom schedulers, quantization methods) and can be cheaper once utilization is high and predictable enough to pay back the platform engineering overhead.

Common pitfalls

Even experienced engineers trip over the same patterns when moving from web serving to GPU inference. These are the frequent failures to catch during design review, with their symptoms, root causes, and fixes.

Flapping: scaling up and down too fast

• Symptom: Your GPU node count oscillates wildly. Cloud bills spike, but user latency doesn't improve. The autoscaler logs show scale-up events followed by scale-down events within minutes.

• Cause: The autoscaler reacts to short traffic blips instead of sustained trends. A GPU node that just finished loading weights gets terminated before it serves enough requests to justify its startup cost.

• Fix: Choose a cooldown from traffic and startup measurements rather than a universal duration. Scale down gradually, and require queue depth near zero, low KV cache usage, and low running requests before removing capacity.

Ignoring tail latency

• Symptom: Your dashboard shows an average response time of 800 ms, but users complain about 30-second waits. The p95 or p99 latency is an order of magnitude worse than the mean.

• Cause: Average latency hides the users with long prompts or the batches where one straggler request keeps the GPU occupied. Static batching makes this worse, but even continuous batching can suffer if a single request with a 4,000-token prompt monopolizes a slot.

• Fix: Monitor TTFT and TPOT at the p95 or p99 percentile, not the mean. Set SLOs on tail latency. If long prompts are common, enable chunked prefill so they don't starve shorter decode requests.

No explicit queue or backpressure

• Symptom: Requests keep arriving even after the GPU workers are full. The service eventually times out randomly, and retry storms make the spike worse.

• Cause: The system accepts more work than the serving engine can schedule. Without a visible queue, admission policy, timeout budget, and retry contract, overload becomes invisible until users see failures.

• Fix: Put a queue or gateway in front of the inference engine. Track queue depth, queue age, and rejection rate. Return controlled overload responses before the GPU fleet collapses.

Treating every request as equal priority

• Symptom: A batch analytics job slows down interactive chat, or one tenant's long prompts make all tenants miss their TTFT target.

• Cause: The scheduler sees all requests as identical even though some workloads are interactive, some are batch, and some have stricter contractual latency limits.

• Fix: Separate traffic classes. Use priority queues, per-tenant rate limits, maximum prompt and generation budgets, and different pools when workload shapes are too different to share one scheduler fairly.

Underestimating long-context KV growth

• Symptom: A model that passed a short-prompt load test starts rejecting work or swapping under real chat history. GPU memory looks fine at startup and then collapses as conversations lengthen.

• Cause: The capacity plan counted weights but not enough KV cache. Long prompts, high concurrency, prefix-cache headroom, and larger generation limits all consume blocks during the request lifetime.

• Fix: Size with realistic prompt and output distributions, not maximum model weights alone. Load test with long conversations, watch vllm:kv_cache_usage_perc, and apply admission limits before the cache pool hits saturation.

Confusing PagedAttention with prefix caching

• Symptom: A team enables PagedAttention and expects repeated system prompts to become free, then sees no TTFT improvement on new replicas.

• Cause: PagedAttention is an allocation strategy for KV blocks. Prefix caching is a reuse strategy for shared prompt prefixes. One reduces fragmentation; the other avoids recomputing prefix KV state.

• Fix: Use both when the workload benefits from both. Treat PagedAttention as baseline memory management and prefix caching as a workload-specific optimization that needs stable shared prefixes and warm caches.

Cache blindness during scale-up

• Symptom: You scale from 2 to 10 GPUs during a traffic spike, but the new nodes serve requests slower than the old ones. TTFT increases right after scaling.

• Cause: The original nodes have been running for hours and have accumulated prefix cache hits (for example, a shared system prompt that every request includes). The brand-new nodes start with empty caches. They must recompute the full prefill from scratch, so their first tokens take much longer.

• Fix: Warm new nodes with a few synthetic requests that populate the common prefix before adding them to the load balancer rotation. Alternatively, use vLLM's prefix caching and send new nodes "seed" traffic to warm their KV cache before taking full production load.

Advanced techniques

Prefill-decode disaggregation

Prefill and decode stress GPUs differently. Prefill processes many prompt tokens in parallel and tends to be compute-heavy. Decode produces one token per active request and tends to be memory-bandwidth-heavy. If you run both phases on the same worker pool, a few long prompts can delay short streaming responses, and decode traffic can leave tensor cores underused.

Prefill-decode disaggregation, introduced for production serving by Splitwise^{[23]Reference 23Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677}, splits the serving fleet into two pools:

• Prefill workers ingest prompts, build the initial KV cache, and hand off cache state.
• Decode workers continue token generation for active requests with tighter scheduling around TPOT.

This isn't free. You now need cache transfer, placement logic, and backpressure between pools. But it can help high-traffic systems where long prompts and streaming generations compete for the same GPU budget. The autoscaling signals also become more specific: prefill workers scale on prompt-token backlog and TTFT, while decode workers scale on active sequences, KV-cache utilization, and TPOT.

NVIDIA Dynamo is one documented open-source control-plane example. Its documentation describes disaggregated prefill and decode deployments across backends including vLLM, SGLang, and TensorRT-LLM, with KV-aware routing and KV-transfer mechanisms for split deployments.^{[3]Reference 3NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/} This makes it a useful implementation to study, but the operational win still depends on measured prompt mix, transfer cost, and cache-hit behavior.

Don't start here. First tune continuous batching, chunked prefill, prefix caching, and queue policy. Disaggregation is a later optimization when one mixed pool can no longer hit both TTFT and TPOT targets.

Use measurements, not architecture fashion, to decide whether to split pools. This gate recommends investigation only when a tuned mixed pool breaches both service objectives under a long-prompt-heavy trace:

Speculative decoding

Speculative decoding speeds up generation by using a smaller "draft" model to predict multiple tokens ahead, then verifying them with the larger "target" model. Leviathan et al. report roughly 2-3x acceleration in their evaluated settings, not a universal serving guarantee.^{[24]Reference 24Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

The speedup comes from amortizing target-model work across several draft tokens. The draft model's predictions are treated as hypotheses; the target model evaluates them in one verification pass and keeps the valid prefix. When the draft model is reasonably accurate for the traffic pattern, this reduces the number of expensive target forward passes. It still pays draft-model compute and rejection overhead, so you measure it against your own workload instead of assuming it always helps.

For example, you might pair a small draft model with a Qwen3.6-27B target to accelerate decode-heavy traffic. The draft model generates candidate tokens quickly; the target model then verifies all candidates in a single forward pass, accepting all correct predictions up to the first mismatch.

Multi-region failover

GPU quota and available capacity vary by region and time. A resilient serving architecture should be able to route traffic to a secondary region when the primary is at capacity or experiencing issues.

This requires:

• Global load balancing: Route requests to the nearest healthy region
• Model sync: Keep model weights and LoRA adapters synchronized across regions
• Failover logic: Detect regional capacity constraints and automatically shift traffic

Regional failover is particularly important for spot instance workloads. If a regional spot fleet is reclaimed, your tested recovery-time objective determines whether secondary on-demand capacity can receive new requests before client timeout budgets expire. Long in-flight generations still need retry or resumption semantics.

Multi-tenancy with LoRA

Instead of deploying a separate Qwen3.6-27B base for every customer fine-tune, use one shared base model and select LoRA (Low-Rank Adaptation)^{[25]Reference 25LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} adapters per request. Systems like S-LoRA^{[26]Reference 26S-LoRA: Serving Thousands of Concurrent LoRA Adapters.https://arxiv.org/abs/2311.03285} show how to batch requests across different adapters while keeping the base model weights shared in GPU memory. vLLM also supports per-request LoRA serving, with explicit warnings around separate runtime adapter-loading APIs in untrusted environments.^{[27]Reference 27LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/}

To implement multi-tenancy, map each tenant to an approved adapter and select that adapter per request. TenantAdapterRouter shows how a single base model instance can serve requests for different tenants by passing a LoRARequest to vLLM. While a true production environment would use vLLM's AsyncLLMEngine to handle concurrent requests without blocking, the synchronous LLM class is shown here for conceptual clarity. The function takes an incoming request containing a tenant ID, looks up the approved adapter path, and outputs the generated response with the selected adapter. This example doesn't expose vLLM's runtime adapter-loading APIs; those are a separate trusted-environment feature:

In production, prefer the async server path over a synchronous wrapper like this, and don't expose arbitrary adapter loading to untrusted tenants.^{[27]Reference 27LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/}

Evaluation rubric

Explain a serving design from first principles instead of only naming tools:

• Compare vLLM, TensorRT-LLM, TGI, SGLang, and llama.cpp in terms of throughput, latency, ecosystem fit, hardware coupling, and operational complexity.
• Explain how continuous batching avoids the straggler waste of static batching.
• Describe how PagedAttention maps logical KV blocks to physical GPU blocks and why that reduces fragmentation.
• Choose scaling signals from queue depth, KV cache utilization, TTFT, and TPOT instead of CPU percent alone.
• Recommend MIG for smaller models only when the target GPU SKU supports it and the slice size still fits weights plus KV cache.
• Plan cold-start mitigation with warm pools, predictive scaling, cached weights, and readiness warmups.
• Explain why quantization reduces both VRAM pressure and the decode bandwidth denominator.
• Separate full-checkpoint residency from active MoE bytes per token, and count two standard tensor-parallel collectives per transformer layer.
• Use LoRA adapters for multi-tenant fine-tunes without turning adapter loading into an untrusted file-access path.
• Say when to stay on one mixed pool versus move to prefill-decode disaggregation (Splitwise-style, operated by frameworks like NVIDIA Dynamo) once one pool can't hit both TTFT and TPOT.

Follow-up questions

How do you handle cold starts for GPU instances in production? Cold starts are dominated by node provisioning, image pulls, weight loading, and runtime warmup. Keep a small warm pool for high-priority traffic, pre-warm ahead of predictable spikes, store weights close to the node, and run readiness warmups before sending real users to a replica. Snapshot or restore features can help on specific platforms, but warm capacity and local weight caches are the common starting point.

When would you choose TensorRT-LLM over vLLM? Choose TensorRT-LLM when measured latency or throughput on NVIDIA hardware justifies the build and deployment complexity. It fits tightly controlled fleets where the model, GPU type, precision mode, and engine build process are stable. vLLM can be easier to operate for general serving and rapid iteration; benchmark either claim on your workload.

How does continuous batching improve throughput compared with static batching? Static batching waits for the longest request in the batch. Continuous batching repacks active work at token-step boundaries, so a finished request frees its slot for queued work immediately. That matters because decode produces one token at a time and request lengths vary widely.

Which metrics matter most for scaling LLM inference clusters? Start with request queue depth and queue age, KV cache utilization, TTFT, and TPOT. Hardware metrics such as GPU duty cycle, memory bandwidth, temperature, and power are useful supporting signals, but they don't tell you whether users are waiting or whether the scheduler has enough KV blocks left.

Try it: three practice labs

To internalize GPU serving concepts, simulate the decisions yourself. Each lab below builds on the article's examples.

Lab A: the architect

Set up a dummy inference service and configure an autoscaler to scale based on queue depth. Use a tool like KEDA with a Prometheus metric source, or simulate the control loop in Python using the GPUAutoscaler class above. Feed it a synthetic traffic trace (flat at night, spike at 9 AM) and plot the replica count over time. Compare a cooldown shorter than your measured node-ready time with one longer than a typical lull.

Lab B: the optimizer

Measure how long it takes to load Gemma 4 12B versus Qwen3.6-27B into GPU memory on your hardware (or using cloud instance startup logs). Calculate the "cost of a cold start" in dollars: multiply measured loading time by hourly GPU cost. Compare that cost and queue impact with keeping one ready replica through a known low-traffic period.

Lab C: the diagnostician

A user reports that the first token takes 5 seconds, but subsequent tokens arrive quickly. Which part of your stack is likely at fault: the autoscaler or the model engine? Write down your reasoning, then check it against the cold-start phases and the prefill-vs-decode discussion above.

Serving control loop recap

You started with a developer assistant that hit a release-freeze traffic spike and discovered why LLM serving isn't web serving. KV-cache capacity constrains active conversations; low-batch decode is often limited by memory bandwidth; and autoscaling needs queue, cache, and latency signals in addition to infrastructure telemetry.

A Kubernetes pattern using KEDA/HPA, a GPU node autoscaler, and a serving engine separates desired replicas from hardware provisioning. Continuous batching and PagedAttention improve GPU use, but a reliable fleet still needs measured capacity, a tuned cooldown, and a warmup policy justified by its latency target and cost.

GPU serving needs a different control loop than a web server. New nodes can run slower than old ones, and a thousand-user spike needs admission control, batching policy, and budget gates before it overruns the inference lane.