GPU Serving & Autoscaling

Advanced MLOps and DevOps for AI gave you the release machinery: promotion gates, lineage, rollout policy, and rollback triggers. This chapter turns that machinery toward the inference fleet. GPU serving and autoscaling decide whether a promoted model can survive real traffic without burning budget or missing latency goals.

Imagine your merchant-support chat app goes viral. At 8 AM, ten customers are asking about order delays. By 9 AM, there are a thousand. If you try serving your AI model on a normal web server, it'll quickly grind to a halt. That's because serving a Large Language Model (LLM) isn't like serving a website. A web server fetches pre-computed data and forgets the user immediately. An LLM has to process every prompt and retain attention state as generation proceeds. That short-term memory is a central capacity constraint alongside compute, memory bandwidth, and queueing delay.

Standard web servers don't maintain growing state for each active user. An LLM does. To generate the next token, the model must access computed attention states from previous tokens. In LLMs, this state is the KV cache, and managing its memory footprint is one of the central challenges of production serving.

Engineers designing serving systems must optimize for two conflicting metrics:

1. Time To First Token (TTFT): The latency before the user sees the first word. This is dominated by the prefill phase (processing the input prompt) and is often compute-bound.
2. Time Per Output Token (TPOT), also called Time Between Tokens (TBT): The latency between subsequent tokens. This determines the perceived generation speed. This is the decode phase and is usually memory-bandwidth bound for large decoder models.

The core difficulty is that the Key-Value (KV) cache (the stored attention computations for past tokens) grows dynamically with every generated token. Poor memory management leads to fragmentation, which forces smaller batch sizes and lower throughput.

The fulfillment-center analogy

To see why GPU serving is so different from normal web scaling, picture a fulfillment center during a return-label surge:

• The GPU is the automated sorting lane. It's specialized, expensive, and you can't replace it with normal web servers.
• The model weights are the routing map loaded into the sorter. Loading it takes time, and the lane can't process parcels until the map is resident.
• The KV cache is the set of active parcel traces on the belt. If a customer conversation started ten turns ago, its trace stays allocated until generation finishes. The trace pool fills up fast during a traffic spike.
• Autoscaling is opening another sorting lane. Bringing a new lane online takes time, and it can't help while the routing map is still loading.

A common mistake is scaling because the sorter is busy. You scale when the queue at the dock is too long. A busy sorter means you're using the hardware. A long queue means customers are waiting, and that's when you need more capacity.

The GPU serving stack

Before diving into specific frameworks, let's look at one common production GPU serving architecture. It consists of several layers:

Infrastructure Layer: Kubernetes (EKS, GKE, or AKS) manages the GPU nodes. Unlike CPU nodes, GPU capacity is expensive and sometimes scarce, so a warm pool must be justified against latency targets and measured demand.

Control Layer: In a KEDA-based setup, KEDA (Kubernetes Event-Driven Autoscaling) can activate a workload from zero and configure a Horizontal Pod Autoscaler (HPA) to scale running replicas from metrics. Karpenter (AWS) or Cluster Autoscaler can then provision GPU nodes when new pods can't be scheduled on existing capacity.^[1][2]

Serving Layer: The actual inference engines (vLLM, SGLang, TensorRT-LLM) that load models into VRAM and handle token generation. At datacenter scale, a control plane such as NVIDIA Dynamo can orchestrate disaggregated prefill/decode pools across many of these engine replicas.^[3]

Metrics: CPU and RAM metrics alone don't describe LLM-serving capacity. NVIDIA's DCGM (Data Center GPU Manager) exposes GPU-specific telemetry, while user-facing scaling signals usually come from the serving layer: queue depth, KV cache usage, TTFT, and inter-token latency.^[4]

Serving frameworks

The rest of this section zooms into the serving engine: batching policy, KV-cache allocation, framework choice, and request lifecycle.

The evolution of batching

Early serving systems used static batching: waiting for $N$ requests to arrive, padding them to the same length, and processing them together. It's inefficient because requests finish at different times. If one request generates 100 tokens and another generates 1000, the GPU must continue processing the batch until the longest request finishes, leaving slots idle for the shorter requests.

Modern frameworks use continuous batching (also known as iteration-level scheduling), pioneered by Orca^[5]. The scheduler operates at the granularity of a single token generation step. As soon as a request finishes, a new request from the queue can be inserted into its slot in the next iteration.

The small simulation below uses generation lengths for four order-support replies. A static batch doesn't refill its two slots until both original replies finish; a continuous scheduler admits the next queued reply as soon as a slot opens.

With the same replies and slots, the continuous scheduler finishes sooner because it wastes fewer slot-iterations.

vLLM and PagedAttention

One widely used open-source serving engine is vLLM, which introduced PagedAttention^[6]. Before vLLM, inference systems pre-allocated a single contiguous block of GPU memory sized to the maximum sequence length a request might ever use (often 4k-32k tokens). Because the actual length is revealed token-by-token during decoding, this approach created two problems:

• Internal fragmentation: most of the pre-allocated chunk sat empty for short requests.
• External fragmentation: free memory became scattered and couldn't satisfy new large contiguous allocations.

Profiling across real workloads showed that prior systems typically used only 20–38% of the KV cache memory they had reserved, wasting 62–80%.^[6] The result: the effective batch size stayed small, and throughput suffered even when the GPU had spare VRAM.

PagedAttention treats the KV cache like virtual memory in an operating system (OS). It breaks the KV cache into fixed-size blocks (pages) that can be stored in non-contiguous memory.

Logical vs. physical blocks

PagedAttention introduces two key abstractions:

• Logical KV blocks: The view of the KV cache from the model's perspective. It appears as a contiguous sequence of tokens.
• Physical KV blocks: The actual fixed-size chunks of memory allocated in HBM (High Bandwidth Memory, the GPU's ultra-fast RAM) to store these tokens.

A Block Table maintains the mapping between logical and physical blocks. When a new token is generated, vLLM checks if the current physical block has space. If not, it allocates a new physical block from a pre-allocated pool and updates the mapping. The PagedAttention paper reports much tighter memory use and higher throughput because batches can pack around real sequence lengths instead of worst-case reservations.^[6]

PagedAttention gives the runtime fine-grained control over KV blocks. In current vLLM deployments, request-to-request reuse of a shared prompt prefix usually comes from automatic prefix caching, which reuses previously computed KV blocks when a new request shares the same prefix.^[7]

Here is the allocation arithmetic for four conversations. The fixed reservation policy reserves each request for its maximum allowed length, while paging reserves blocks only for tokens currently stored:

Paging doesn't make attention state free. It stops short conversations from holding blocks for a maximum length they haven't used.

The overall request lifecycle coordinates these components. Client requests enter an API server queue, are packed by a continuous batching scheduler, and finally executed by a model engine using PagedAttention and, when enabled, prefix caching on the GPU cluster:

Framework comparison

Choosing the right serving engine depends on your specific constraints:

The TGI documentation describes TGI as maintenance mode and directs users toward newer optimized engines such as vLLM and SGLang for new deployments.^[10] These engines are the data plane that loads weights and generates tokens. At datacenter scale, a separate control plane such as NVIDIA Dynamo can sit above them to provide disaggregated prefill/decode scheduling and KV-aware routing across engine replicas (covered later under disaggregation).^[3]

To see how this works in practice, here's an example of initializing vLLM with settings you would benchmark before deployment. The configuration below loads a large model across multiple GPUs using tensor parallelism and sets a high memory-utilization budget for weights, KV blocks, and runtime buffers. It also uses chunked prefill so long prompts can share scheduling budget with decode traffic.^[13] It takes the model identifier and hardware constraints as inputs, and outputs an initialized engine ready to accept inference requests:

All of these knobs are workload-dependent. Tune them against TTFT and TPOT SLOs, not just raw tokens/sec.

GPU infrastructure

Sizing and selection

Selecting the right GPU depends on model size, quantization, and expected traffic. The dominant factors are VRAM (Video Random Access Memory) capacity (to fit the model + KV cache) and Memory Bandwidth (to serve tokens fast).

To estimate memory requirements for capacity, use this formula:

$$\text{Memory} \approx \text{Weights} + \text{KV Cache} + \text{Activation Overhead}$$

Where:

• $\text{Weights} \approx \text{Params} \times \text{Precision (bytes)}$ (e.g., 70B $\times$ 2 bytes = 140GB)
• $\text{KV Cache} \approx 2 \times \text{Layers} \times \text{KV Heads} \times \text{Head Dim} \times \text{Seq Len} \times \text{Concurrency} \times \text{Precision}$ (the factor of 2 accounts for both the key and value tensors)

That distinction matters. The older shorthand using Hidden Dim assumes full multi-head attention. Many modern decoder models use grouped-query attention (GQA) or multi-query attention (MQA), so the number of KV heads is much smaller than the total attention head count. If you ignore that, you'll often overestimate KV cache size by a wide margin.

Here's a practical Python function that estimates the required GPU memory based on model size and cache expectations. It takes the model size plus the architecture terms that control KV growth (num_layers, num_kv_heads, head_dim, context length, and concurrency), then returns a dictionary detailing the number of specific GPU models needed. This helps engineers plan capacity before deploying:

Notice how the 70B BF16 case needs more than the theoretical two-GPU weight fit once you add four active 8k contexts and operational headroom. The table below is a minimum-fit reference; production sizing starts from measured prompt and concurrency distributions.

As a reference, the following table gives order-of-magnitude sizing for common dense decoder deployments. The KV numbers assume GQA-style architectures and FP16/BF16 KV cache. Full multi-head attention or larger KV precision pushes the cache higher.

That 70B INT8 row is a tight fit. In practice, many teams still use two GPUs so they have room for longer prompts, prefix caching, and higher concurrency.

Multi-Instance GPU (MIG)

For cost-efficient autoscaling of smaller models, Multi-Instance GPU (MIG) allows you to partition a single GPU into hardware-isolated slices. Ampere- and Hopper-class parts such as A100, H100, and H200 can expose up to seven instances on supported SKUs.^[14] Instead of dedicating an entire large GPU to a single 7B model, you can run multiple replicas on isolated slices with dedicated compute and memory resources.

Throughput & bandwidth constraints

For low-batch decode of large decoder models, weight reads commonly make the system memory-bandwidth bound. Batching, cache traffic, kernels, and multi-GPU communication can change which limit dominates.

The theoretical maximum throughput ($TPS_{max}$) for a batch size of 1 is roughly (a simplified model derived from memory-bandwidth analyses such as Pope et al.)^[15]:

$$TPS_{max} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$$

Reading the formula

In this simplified batch-size-one model, the GPU streams the weight footprint from HBM for each generated token. There is little weight reuse across sequential steps (the KV cache reuses activations, not the static weights). Memory bandwidth therefore gives a useful ceiling:

$$TPS_{max} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$$

This is the reciprocal of the time required to load the entire model from HBM once. An H100 SXM (3.35 TB/s HBM bandwidth) serving a 7B dense model in FP16 (≈14 GB of weights) yields an upper bound of roughly 239 tokens/sec for a single sequence.^[16] Real systems achieve lower throughput because:

• KV cache reads/writes, activations, and sampling also consume memory bandwidth
• Kernel launch, attention computation, and scheduler overhead
• The achieved bandwidth of fused kernels is rarely 100 % of the theoretical peak

This bound explains why quantization can improve decode throughput: halving only the weight-footprint term (FP16 to INT8, for example) doubles this simplified ceiling. Actual speedup is smaller or different when kernel support, KV-cache traffic, compute, or communication dominate. It also shows why large models may require tensor parallelism: the denominator grows while per-GPU bandwidth stays fixed.

The next calculation is deliberately a ceiling, not a benchmark. It isolates weight traffic so you can see what quantization changes before adding runtime overhead:

Tensor parallelism

For models that don't fit on a single GPU (for example, a 70B dense model in BF16), we use Tensor Parallelism (TP). TP splits the individual weight matrices (e.g., $W_Q, W_K, W_V$) across multiple GPUs so that the computation is distributed evenly. This technique was pioneered in the Megatron-LM training system^[17] and adapted for inference serving.

TP operates by slicing the matrix computations, meaning all participating GPUs must communicate their partial results at each layer before the model can proceed to the next layer. This constant, high-volume communication requires immense bandwidth.

• Intra-node: TP works best when GPUs share a very fast fabric such as NVLink (NVIDIA high-bandwidth interconnect) or NVSwitch (NVIDIA multi-GPU switch).
• Inter-node: TP can extend across RDMA (Remote Direct Memory Access) or InfiniBand, but the communication tax rises quickly. For multi-node serving, teams often keep TP within a node when they can, and use Pipeline Parallelism (PP) or independent replicas across nodes instead. PP only communicates activations at stage boundaries, but it introduces pipeline bubbles and can worsen tail latency.

The following diagram illustrates the structural difference between intra-node Tensor Parallelism (where computation is split and synchronized at every layer) and inter-node Pipeline Parallelism (where computation is chained sequentially across nodes):

Communication frequency is not total cost: tensor-parallel messages may travel over faster links, while pipeline stages can create bubbles. This toy count makes the first screening question explicit:

Autoscaling strategies

CPU-based autoscaling alone is insufficient for LLM serving because CPU isn't usually the scarce inference resource. CPU can still reveal overloaded gateways or tokenizers. GPU utilization can also be misleading: a GPU might be busy with a healthy batch, or it might be memory-bound while compute duty cycle looks modest.

Key metrics

The most reliable metrics for autoscaling come from two sources: NVIDIA's DCGM (Data Center GPU Manager) for hardware-level visibility, and the serving framework itself for application-level signals. In vLLM, for example, the Prometheus endpoint exposes metrics such as vllm:num_requests_waiting, vllm:num_requests_running, vllm:kv_cache_usage_perc, vllm:time_to_first_token_seconds, and vllm:inter_token_latency_seconds.^[4]

DCGM (Data Center GPU Manager) exposes hardware-level GPU metrics like temperature, power, clock rates, and utilization. While GPU utilization seems like an obvious scaling signal, it's often misleading for LLM inference. A GPU can show 100% "utilization" while sitting idle waiting for memory (memory-bound), or show low utilization during long context prefill phases.

That's why application-level metrics (queue depth, KV cache utilization, TTFT, TPOT) are more reliable scaling signals. They directly measure capacity constraints rather than hardware activity. The exact thresholds are workload-specific, so treat any numbers you see in dashboards or sample code as starting points, not universal defaults.

KEDA and Karpenter integration

One control-loop design works as follows: the serving framework exposes metrics via Prometheus. KEDA can activate a scaled workload from zero and configure the generated HPA for running replica scaling. If the scheduler can't place new GPU pods, the node autoscaler layer (often Karpenter or Cluster Autoscaler) provisions nodes that satisfy pod requirements.^[1][2]

This two-layer approach separates desired pod count from hardware supply. Ready pods may scale quickly if spare GPU capacity exists; new nodes and model loading can take far longer. The useful warm buffer is a measured tradeoff between idle GPU cost and the latency damage of a cold burst.

The autoscaling control loop continuously monitors queue depth, KV cache utilization, and latency metrics to make scaling decisions. As the diagram below shows, the metric path and the node-provisioning path are separate:

In a real Kubernetes deployment, KEDA and HPA compute the desired replica count for you. The following Python snippet is just a mental model for the control logic. It takes the current metrics and replica count as inputs, then returns the next target replica count. The thresholds are illustrative only:

The autoscaler determines a target; it doesn't erase startup time. This capacity calculation separates the eventual replica count from the traffic a warm buffer can accept immediately:

A worked example: scaling from zero

Let's walk through a concrete scenario. You're hosting a Llama-3-8B model for your merchant-support chat app. Traffic is nearly zero at night, but at 9 AM it jumps to 100 concurrent users. How do you scale without blowing the budget or the user experience?

Step 1: pick the right metric. Scaling only on CPU percentage misses engine capacity. A GPU worker can show little CPU pressure while its weights and KV blocks constrain admission. Use request queue depth, queue age, KV cache utilization, and latency SLOs; retain CPU signals for gateway or preprocessing failures.

Step 2: set a threshold. Through benchmarking, you determine that one GPU can comfortably handle about 5 concurrent requests while keeping TPOT under 50 ms per token. That's your target capacity per replica.

Step 3: do the math. With 100 concurrent requests and 5 requests per GPU, you need 100 / 5 = 20 GPUs. If you're starting from one warm instance at 8:59 AM, the autoscaler must add 19 more replicas quickly.

Step 4: solve the cold-start gap. Even if a replica target changes promptly, node provisioning plus model loading can outlast the traffic's latency budget. Measure that ready time on your platform. If the morning rush is predictable, start warming nodes far enough ahead of the observed ready time, or keep a benchmarked buffer pool that absorbs its first wave.

Scaling only when latency spikes is late. By the time TTFT breaches your SLO, users are already frustrated. Scale on queue depth trends, not only on lagging latency indicators.

Handling cold starts

Cold starts in GPU serving environments are significantly more disruptive than in traditional microservices. While a typical web container might start in seconds, a GPU workload often has to provision a node, pull a large image, fetch model weights, and then warm up the runtime before it can accept traffic.

Here's the difference between a warm GPU (already loaded and serving) and a cold GPU (starting from scratch):

The cold start has four distinct phases, each with its own mitigation strategy. The ranges below are illustrative planning inputs, not platform guarantees; record your own p50 and p95 timings:

Container streaming lets the container start executing before the entire image is downloaded. Only the required layers are fetched on demand, which cuts startup time for large inference images.

Model caching keeps frequently used model weights on high-speed local NVMe storage rather than pulling from remote object storage. For multi-node setups, shared network volumes like Amazon FSx for Lustre can serve weights at high speed to multiple nodes.

Some platforms also offer snapshot or restore features for a preloaded container or VM. Those can help, but they're more vendor-specific than warm pools plus local weight caches.

To mitigate cold start latency impact on users, engineers implement several proactive strategies:

• Over-provisioning: Keep a small headroom buffer to absorb sudden traffic spikes while new instances spin up.
• Predictive scaling: Use historical traffic patterns to scale up the cluster before anticipated usage spikes (e.g., scaling up just before the morning rush hour).
• Minimum warm pool: For critical workloads, retain enough measured warm capacity to protect the initial-burst TTFT target while cold replicas load.

You can turn measured phase durations into a readiness decision. In this example, predictive scaling four minutes before a known carrier-status surge is sufficient for the measured path, while scaling two minutes before it is not:

Cost optimization

Running top-end GPUs 24/7 is expensive, and idle replicas burn budget fast. To optimize costs without breaking Service Level Agreements (SLAs):

Spot instances and fault tolerance

Cloud providers offer spot instances at discounts, but they come with preemption risk. Inference can be easier to retry than long-running training, but reasoning runs and long generations may exceed a provider's warning window. Use measured request durations, a drain deadline, and retry behavior before assigning traffic to spot capacity.

In retail, seasonal spikes such as holiday rushes can be candidates for spot overflow capacity when a stable baseline pool meets the promised SLO and interrupted work can drain or retry cleanly.

• Graceful shutdown: On SIGTERM, remove the replica from new admission, let completions that fit the remaining drain budget finish, and retry or fail over requests that cannot finish before termination.
• Mixed pools: A common policy retains on-demand or reserved capacity for baseline SLO traffic and uses an autoscaling spot pool only for retryable overflow.

The drain policy must compare remaining work with a deadline rather than assume all requests finish. This example reserves twenty seconds for shutdown and transfer after a 120-second termination warning:

Scale-down cooldown & thrashing

A common (and expensive) mistake is scaling down too aggressively. If you terminate a GPU node because traffic dipped for 30 seconds, you'll pay the cold start penalty again when traffic returns a minute later. This "thrashing" can actually increase costs while degrading user experience.

In e-commerce logistics, this is like closing a return-processing lane because the conveyor cleared for 30 seconds during a lull, only to have the post-holiday rush hit again before the lane can reopen.

Best practices for scale-down:

• Cooldown period: Start with a cooldown longer than short observed lulls and tune it against measured cold-start cost, idle spend, and recurring traffic patterns.
• Drain before terminate: Give the autoscaler time to drain in-flight requests before removing a replica. Don't kill a GPU mid-generation.
• Scale down gradually: Remove one replica at a time and wait to see if queue depth remains low before removing more.

Deployment options

When deciding how to deploy GPU inference, teams face a build-vs-buy decision:

Serverless platforms like Modal and RunPod abstract away the Kubernetes layer entirely. They typically handle autoscaling and much of the instance lifecycle for you. This is ideal for teams without dedicated ML infrastructure engineers or for workloads with highly variable traffic.

Specialized GPU clouds sit in the middle. You usually get raw instances, storage, or managed Kubernetes primitives without the full DIY burden of the hyperscalers. Managed Kubernetes still gives you the most control over the full stack (vLLM versions, custom schedulers, quantization methods) and can be cheaper once utilization is high and predictable enough to pay back the platform engineering overhead.

Common pitfalls

Even experienced engineers trip over the same patterns when moving from web serving to GPU inference. Here are the frequent failures to catch during design review, with their symptoms, root causes, and fixes.

Flapping: scaling up and down too fast

Symptom: Your GPU node count oscillates wildly. Cloud bills spike, but user latency doesn't improve. The autoscaler logs show scale-up events followed by scale-down events within minutes.

Cause: The autoscaler reacts to short traffic blips instead of sustained trends. A GPU node that just finished loading weights gets terminated before it serves enough requests to justify its startup cost.

Fix: Choose a cooldown from traffic and startup measurements rather than a universal duration. Scale down gradually, and require queue depth near zero, low KV cache usage, and low running requests before removing capacity.

Ignoring tail latency

Symptom: Your dashboard shows an average response time of 800 ms, but support tickets complain about 30-second waits. The p95 or p99 latency is an order of magnitude worse than the mean.

Cause: Average latency hides the users with long prompts or the batches where one straggler request keeps the GPU occupied. Static batching makes this worse, but even continuous batching can suffer if a single request with a 4,000-token prompt monopolizes a slot.

Fix: Monitor TTFT and TPOT at the p95 or p99 percentile, not the mean. Set SLOs on tail latency. If long prompts are common, enable chunked prefill so they don't starve shorter decode requests.

No explicit queue or backpressure

Symptom: Requests keep arriving even after the GPU workers are full. The service eventually times out randomly, and retry storms make the spike worse.

Cause: The system accepts more work than the serving engine can schedule. Without a visible queue, admission policy, timeout budget, and retry contract, overload becomes invisible until users see failures.

Fix: Put a queue or gateway in front of the inference engine. Track queue depth, queue age, and rejection rate. Return controlled overload responses before the GPU fleet collapses.

Treating every request as equal priority

Symptom: A batch analytics job slows down interactive customer support, or one tenant's long prompts make all tenants miss their TTFT target.

Cause: The scheduler sees all requests as identical even though some workloads are interactive, some are batch, and some have stricter contractual latency limits.

Fix: Separate traffic classes. Use priority queues, per-tenant rate limits, maximum prompt and generation budgets, and different pools when workload shapes are too different to share one scheduler fairly.

Underestimating long-context KV growth

Symptom: A model that passed a short-prompt load test starts rejecting work or swapping under real chat history. GPU memory looks fine at startup and then collapses as conversations lengthen.

Cause: The capacity plan counted weights but not enough KV cache. Long prompts, high concurrency, prefix-cache headroom, and larger generation limits all consume blocks during the request lifetime.

Fix: Size with realistic prompt and output distributions, not only maximum model weights. Load test with long conversations, watch vllm:kv_cache_usage_perc, and apply admission limits before the cache pool hits saturation.

Confusing PagedAttention with prefix caching

Symptom: A team enables PagedAttention and expects repeated system prompts to become free, then sees no TTFT improvement on new replicas.

Cause: PagedAttention is an allocation strategy for KV blocks. Prefix caching is a reuse strategy for shared prompt prefixes. One reduces fragmentation; the other avoids recomputing prefix KV state.

Fix: Use both when the workload benefits from both. Treat PagedAttention as baseline memory management and prefix caching as a workload-specific optimization that needs stable shared prefixes and warm caches.

Cache blindness during scale-up

Symptom: You scale from 2 to 10 GPUs during a traffic spike, but the new nodes serve requests slower than the old ones. TTFT actually increases right after scaling.

Cause: The original nodes have been running for hours and have accumulated prefix cache hits (for example, a shared system prompt that every request includes). The brand-new nodes start with empty caches. They must recompute the full prefill from scratch, so their first tokens take much longer.

Fix: Warm new nodes with a few synthetic requests that populate the common prefix before adding them to the load balancer rotation. Alternatively, use vLLM's prefix caching and ensure new nodes receive "seed" traffic to warm their KV cache before taking full production load.

Advanced techniques

Prefill-decode disaggregation

Prefill and decode stress GPUs differently. Prefill processes many prompt tokens in parallel and tends to be compute-heavy. Decode produces one token per active request and tends to be memory-bandwidth-heavy. If you run both phases on the same worker pool, a few long prompts can delay short streaming responses, and decode traffic can leave tensor cores underused.

Prefill-decode disaggregation, introduced for production serving by Splitwise^[18], splits the serving fleet into two pools:

• Prefill workers ingest prompts, build the initial KV cache, and hand off cache state.
• Decode workers continue token generation for active requests with tighter scheduling around TPOT.

This is not free. You now need cache transfer, placement logic, and backpressure between pools. But it can help high-traffic systems where long prompts and streaming generations compete for the same GPU budget. The autoscaling signals also become more specific: prefill workers scale on prompt-token backlog and TTFT, while decode workers scale on active sequences, KV-cache utilization, and TPOT.

NVIDIA Dynamo is one documented open-source control-plane example. Its documentation describes disaggregated prefill and decode deployments across backends including vLLM, SGLang, and TensorRT-LLM, with KV-aware routing and KV-transfer mechanisms for split deployments.^[3] This makes it a useful implementation to study, but the operational win still depends on measured prompt mix, transfer cost, and cache-hit behavior.

Don't start here. First tune continuous batching, chunked prefill, prefix caching, and queue policy. Disaggregation is a later optimization when one mixed pool can no longer hit both TTFT and TPOT targets.

Use measurements, not architecture fashion, to decide whether to split pools. The following gate recommends investigation only when a tuned mixed pool breaches both service objectives under a long-prompt-heavy trace:

Speculative decoding

Speculative decoding speeds up generation by using a smaller "draft" model to predict multiple tokens ahead, then verifying them with the larger "target" model. Leviathan et al. report roughly 2-3x acceleration in their evaluated settings, not a universal serving guarantee.^[19]

The speedup comes from amortizing target-model work across several draft tokens. The draft model's predictions are treated as hypotheses; the target model evaluates them in one verification pass and keeps the valid prefix. When the draft model is reasonably accurate for the traffic pattern, this reduces the number of expensive target forward passes. It still pays draft-model compute and rejection overhead, so you measure it against your own workload instead of assuming it always helps.

For example, you might pair a small draft model with a 70B target model to accelerate decode-heavy traffic. The draft model generates candidate tokens quickly; the target model then verifies all candidates in a single forward pass, accepting all correct predictions up to the first mismatch.

Multi-region failover

GPU availability varies by region and time. During peak demand (like major AI product launches), entire regions can run out of H100 capacity. A resilient serving architecture should be able to "overflow" traffic to a secondary region when the primary is at capacity or experiencing issues.

This requires:

• Global load balancing: Route requests to the nearest healthy region
• Model sync: Keep model weights and LoRA adapters synchronized across regions
• Failover logic: Detect regional capacity constraints and automatically shift traffic

Regional failover is particularly important for spot instance workloads. If a regional spot fleet is reclaimed, your tested recovery-time objective determines whether secondary on-demand capacity can receive new requests before client timeout budgets expire. Long in-flight generations still need retry or resumption semantics.

Multi-tenancy with LoRA

Instead of deploying a separate 70B model for every customer fine-tune, use a shared base model and hot-swap LoRA (Low-Rank Adaptation)^[20] adapters. Systems like S-LoRA^[21] show how to batch requests across different adapters while keeping the base model weights shared in GPU memory. vLLM also supports per-request LoRA serving, with explicit warnings around runtime adapter loading in untrusted environments.^[22]

To implement multi-tenancy, we can dynamically load LoRA adapters per request. The class below shows how a single base model instance can serve requests for different tenants by applying the corresponding tenant's adapter on the fly. While a true production environment would use vLLM's AsyncLLMEngine to handle concurrent requests without blocking, the synchronous LLM class is shown here for conceptual clarity. The function takes the incoming HTTP request containing a tenant ID, looks up the adapter path, and outputs the generated response using the dynamically merged weights:

In production, prefer the async server path over a synchronous wrapper like this, and don't expose arbitrary adapter loading to untrusted tenants.^[22]

Evaluation rubric

At this point, you should be able to explain a serving design from first principles, not only name tools:

• Compare vLLM, TensorRT-LLM, TGI, SGLang, and llama.cpp in terms of throughput, latency, ecosystem fit, hardware coupling, and operational complexity.
• Explain how continuous batching avoids the straggler waste of static batching.
• Describe how PagedAttention maps logical KV blocks to physical GPU blocks and why that reduces fragmentation.
• Choose scaling signals from queue depth, KV cache utilization, TTFT, and TPOT instead of CPU percent alone.
• Recommend MIG for smaller models only when the target GPU SKU supports it and the slice size still fits weights plus KV cache.
• Plan cold-start mitigation with warm pools, predictive scaling, cached weights, and readiness warmups.
• Explain why quantization reduces both VRAM pressure and the decode bandwidth denominator.
• Use LoRA adapters for multi-tenant fine-tunes without turning adapter loading into an untrusted file-access path.
• Say when to stay on one mixed pool versus move to prefill-decode disaggregation (Splitwise-style, operated by frameworks like NVIDIA Dynamo) once one pool cannot hit both TTFT and TPOT.

Follow-up questions

How do you handle cold starts for GPU instances in production?

Cold starts are dominated by node provisioning, image pulls, weight loading, and runtime warmup. Keep a small warm pool for critical traffic, pre-warm ahead of predictable spikes, store weights close to the node, and run readiness warmups before sending real users to a replica. Snapshot or restore features can help on specific platforms, but warm capacity and local weight caches are the common starting point.

When would you choose TensorRT-LLM over vLLM?

Choose TensorRT-LLM when measured latency or throughput on NVIDIA hardware justifies the build and deployment complexity. It fits tightly controlled fleets where the model, GPU type, precision mode, and engine build process are stable. vLLM can be easier to operate for general serving and rapid iteration; benchmark either claim on your workload.

How does continuous batching improve throughput compared with static batching?

Static batching waits for the longest request in the batch. Continuous batching repacks active work at token-step boundaries, so a finished request frees its slot for queued work immediately. That matters because decode produces one token at a time and request lengths vary widely.

Which metrics matter most for scaling LLM inference clusters?

Start with request queue depth and queue age, KV cache utilization, TTFT, and TPOT. Hardware metrics such as GPU duty cycle, memory bandwidth, temperature, and power are useful supporting signals, but they don't tell you whether users are waiting or whether the scheduler has enough KV blocks left.

Try it: three practice labs

The best way to internalize GPU serving concepts is to simulate the decisions yourself. Each lab below builds on the article's examples.

Lab A: the architect

Set up a dummy inference service and configure an autoscaler to scale based on queue depth. Use a tool like KEDA with a Prometheus metric source, or simulate the control loop in Python using the GPUAutoscaler class from this article. Feed it a synthetic traffic trace (flat at night, spike at 9 AM) and plot the replica count over time. Compare a cooldown shorter than your measured node-ready time with one longer than a typical lull.

Lab B: the optimizer

Measure how long it takes to load a 7B model versus a 70B model into GPU memory on your hardware (or using cloud instance startup logs). Calculate the "cost of a cold start" in dollars: multiply measured loading time by hourly GPU cost. Compare that cost and queue impact with keeping one ready replica through a known low-traffic period.

Lab C: the interviewer

A user reports that the first token takes 5 seconds, but subsequent tokens arrive quickly. Which part of your stack is likely at fault: the autoscaler or the model engine? Write down your reasoning, then check it against the cold-start phases and the prefill-vs-decode discussion in this article.

Key takeaways

You started with a merchant chat app that went viral and discovered why LLM serving isn't web serving. KV-cache capacity constrains active conversations; low-batch decode is often limited by memory bandwidth; and autoscaling needs queue, cache, and latency signals in addition to infrastructure telemetry.

A Kubernetes pattern using KEDA/HPA, a GPU node autoscaler, and a serving engine separates desired replicas from hardware provisioning. Continuous batching and PagedAttention improve GPU use, but a reliable fleet still needs measured capacity, a tuned cooldown, and a warmup policy justified by its latency target and cost.

After this chapter, you can explain why a GPU serving lane needs a different control loop than a web server, why new nodes sometimes run slower than old ones, and how to keep a thousand-user spike from overrunning your inference budget.