LLM Inference Engines Compared: vLLM, SGLang, Ollama & llama.cpp

Fifty engineers open an internal assistant at once. The first ten get fast answers, and the rest wait even though the GPU dashboard shows 60% utilization.

You have two plausible fixes: buy a larger GPU or change the serving runtime, the software that schedules model work. The dashboard doesn't tell you which one addresses the wait. Requests can pile up when work is admitted in the wrong order, memory is reserved poorly, or prompts vary in length.

Start with the deployment surface

First separate a packaged local product from a library embedded in your application and a shared GPU server. Then check whether each candidate supports your exact model. Those filters remove incompatible choices before any speed comparison.

Selection rule: Deployment surface narrows the field. Exact checkpoint support decides what can run. Your latency service-level objective (SLO), throughput, and operating cost decide what should run.

Ollama packages model download, lifecycle, a local daemon, command-line interface (CLI), and application programming interfaces (APIs) into one local product. One supported execution backend is llama.cpp. Ollama also provides an MLX (Apple's machine-learning framework) path on Apple Silicon. The March 2026 announcement introduced it as a preview; check the installed release and model rather than assuming every Ollama model uses the same backend.^{[1]Reference 1Ollama GitHub Repositoryhttps://github.com/ollama/ollama[2]Reference 2Ollama is now powered by MLX on Apple Silicon (preview)https://ollama.com/blog/mlx}

llama.cpp is the lower-level runtime and library. It reads GGUF model files (a format for model tensors and metadata), builds across CPU and GPU backends, and includes llama-server, an OpenAI-compatible HTTP server.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp} Ollama packages the local workflow. llama.cpp gives an application more control over embedding the runtime, builds, and runtime flags. Here, embedding means incorporating a library, not generating an embedding vector. Local LLM Deployment covers hardware and memory sizing; Run Gemma 4 Locally with Ollama works through local model tags.

The server-oriented engines expose more cluster scheduling, parallelism, and model-serving controls. That extra control only helps when the deployment needs it, so choose this branch because you have shared traffic to serve, not because the product name sounds more serious.

Their release state changes quickly. Pin a version before comparing behavior.

Runtime licenses cover the engine software. Review checkpoint, tokenizer, dataset, and remote model-code terms separately.

Use these stable tags to inspect the documented behavior. vLLM also publishes v0.33.0rc3; its release-candidate suffix keeps it outside this stable comparison. Prereleases, nightlies, and unversioned documentation can describe different behavior. Benchmark the target workload before upgrading, and record the deployed tag and container digest alongside the project name.^{[4]Reference 4vLLM v0.32.0 releasehttps://github.com/vllm-project/vllm/releases/tag/v0.32.0}

Release numbers still don't prove compatibility. Validate the exact checkpoint, quantization, attention backend, tool parser, structured-output path, and parallel layout. "Supports the architecture" can still mean your chosen feature combination is untested.

Compatibility still doesn't explain why those fifty requests wait while the GPU looks only partly busy. What does the runtime decide for a single request?

What an inference engine controls

Follow one request through the runtime. First, prefill processes all prompt tokens and writes the request's key-value (KV) state, the attention data reused at each output step. Then ordinary autoregressive decode reads that state to produce the next token. Speculative decoding can draft and verify several positions in one target-model pass, so a runtime step need not deliver exactly one token.

That split gives us two useful clocks. Client-observed time to first token (TTFT) includes network transit, queueing, scheduling, prefill, and delivery of the first output token. Time per output token (TPOT) averages the remaining decode duration over the remaining output tokens. For 101 output tokens whose first and last arrivals are 5 seconds apart, TPOT is 5/100 = 0.05 seconds, or 50 ms. Its reciprocal is 20 tokens/s for that request after the first token. Aggregate output tokens per second also depends on concurrent requests and batching.

An average can hide pauses. Inter-token latency measures the gaps between successive streamed tokens. If the client receives chunks rather than token-level events, record that limitation. A smooth-looking average TPOT doesn't prove that every token arrived smoothly.

The Inference: TTFT, TPS & KV Cache lesson derives those metrics and the cache-size formula. For this comparison, remember that a runtime controls four coupled resources:

1. Admission and scheduling: which prefills and decodes enter each step.
2. KV memory: which request owns each block and when blocks can be reused.
3. Kernels and parallelism: how model work maps to target hardware.
4. API and operations: model loading, streaming, metrics, failure recovery, and rollout artifacts.

View larger

Now return to the 60% dashboard. Low utilization can come from a host-side scheduler gap, a long prefill blocking decode, insufficient queued work, collective communication, or memory admission. GPU utilization alone can't distinguish them.

Use a request trace to choose an engine. The name on the server matters less than which part of this path is holding the queue.

Count work that meets the latency target

Suppose the assistant's target is TTFT at most 500 ms and average TPOT at most 50 ms. Treat these as example requirements and set production limits from the product's latency budget. Compare two fictional 10-second runs on the same hardware and request mix, each with ten offered requests. Every completed response contains 100 output tokens; failed requests emit none in this simplified example.

B produces more tokens, but A completes more requests within the target. Goodput counts work that satisfies a declared success contract. Here the contract includes completion and both latency limits; an actual deployment must also check answer quality and endpoint correctness. Neither result automatically justifies release: A's 20% failure rate may violate a separate reliability gate.

View larger

The short calculation below reconstructs the table from per-request records. Each tuple contains completion status, a nonnegative integer output-token count, TTFT in milliseconds, and average TPOT in milliseconds. Missing timing fields on failures never qualify as success. All ten offered requests remain in the accounting.

from math import isfinite

def summarize(rows, seconds, ttft_limit=500, tpot_limit=50):
    if not isfinite(seconds) or seconds <= 0:
        raise ValueError("Measurement duration must be positive and finite")
    completed = good = tokens = good_tokens = 0
    for ok, output_tokens, ttft_ms, tpot_ms in rows:
        if type(output_tokens) is not int or output_tokens < 0:
            raise ValueError("Output-token count must be a nonnegative integer")
        tokens += output_tokens  # Includes partial output from failed requests.
        completed += int(ok)
        timing_valid = all(
            value is not None and isfinite(value) and value >= 0
            for value in (ttft_ms, tpot_ms)
        )
        if ok and timing_valid and ttft_ms <= ttft_limit and tpot_ms <= tpot_limit:
            good += 1
            good_tokens += output_tokens
    return {
        "offered": len(rows), "completed": completed,
        "failed": len(rows) - completed, "good": good,
        "output_tps": tokens / seconds,
        "good_output_tps": good_tokens / seconds,
        "good_rps": good / seconds,
    }

runs = {
    "A": [(True, 100, 400, 40)] * 6
         + [(True, 100, 700, 40)] * 2
         + [(False, 0, None, None)] * 2,
    "B": [(True, 100, 400, 40)] * 4
         + [(True, 100, 400, 70)] * 6,
}
for name, rows in runs.items():
    result = summarize(rows, seconds=10)
    print(f"{name}: completed={result['completed']}/10, good={result['good']}/10, "
          f"output={result['output_tps']:.0f} tok/s, "
          f"good_output={result['good_output_tps']:.0f} tok/s, "
          f"good_requests={result['good_rps']:.1f} req/s")

A: completed=8/10, good=6/10, output=80 tok/s, good_output=60 tok/s, good_requests=0.6 req/s
B: completed=10/10, good=4/10, output=100 tok/s, good_output=40 tok/s, good_requests=0.4 req/s

For a real run, choose a measurement-window policy before collecting data. If requests cross window boundaries, don't divide whole-request token counts by an interval that contains only part of their execution. Use token timestamps within a steady-state window, or a complete cohort with its full elapsed duration. Report offered, completed, failed, and cancelled requests alongside the throughput numbers.

Benchmark boundary: Same model name isn't enough. Match model revision, precision, tokenizer, prompt template, output limits, hardware, workload, and correctness checks before attributing a difference to the runtime.

Those fifty engineers may also share a long runbook prefix. Unique prompts, repeated prefixes, and memory pressure exercise different parts of the serving path. Before comparing cache hit rates, separate two mechanisms: packing request memory and reusing work across requests.

Why block allocation changes capacity

Return to the shared runbook. If many requests repeat the same long prefix, the runtime needs enough KV memory to keep them active at once. vLLM's original PagedAttention work made that allocation problem a first-class serving concern. Instead of reserving one maximum-length contiguous region per request, the runtime assigns fixed-size physical blocks as tokens arrive and keeps a logical block table.^{[17]Reference 17Efficient Memory Management for Large Language Model Serving with PagedAttentionhttps://arxiv.org/abs/2309.06180}

Use the same 18-block pool on both sides. A fixed policy reserves six blocks for each request. Requests A, B, and C currently need three, four, and two blocks: nine used, nine reserved but empty, and no unreserved capacity. On-demand allocation gives them only their nine live blocks. It can then admit D and E using three and two blocks, leaving four blocks free.

The diagram compares two hypothetical allocation policies. Paged allocation can improve utilization even when requests share no prefix. Prefix caching is a separate optimization that reuses already computed KV blocks across matching requests. Real allocators also face partially filled tails, metadata, eviction, and future growth. Four free blocks may still leave five growing requests competing for capacity and trigger preemption. KV Cache & PagedAttention derives those capacity constraints.

View larger

With the memory problem visible, the engine comparison has a sharper question: which runtime gives this workload the scheduling, caching, and hardware behavior it needs?

What each runtime optimizes

vLLM

Start the shared-server shortlist with vLLM when you need broad model coverage, familiar APIs, several accelerator paths, and a large operational ecosystem.^{[18]Reference 18vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionhttps://github.com/vllm-project/vllm}

Its design combines paged KV memory, continuous batching, chunked prefill, parallel execution, and automatic prefix caching (APC). APC reuses matching prompt KV state to skip prefill work. The engine still decodes each new answer.^{[19]Reference 19Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/} Both vLLM and SGLang support shared-prefix reuse.

A fair comparison therefore measures both engines with prefix caching enabled, compatible cache isolation, and cache-aware routing. Prefix Caching and Prompt Caching covers the hit-rate mechanics. Use Deep Dive - vLLM when scheduler, block-pool, or preemption behavior is part of the decision.

SGLang

SGLang combines a server runtime with a programming frontend for multi-call language-model programs. Its founding work introduced RadixAttention, a radix-tree design that reuses exact token prefixes across requests.^{[20]Reference 20SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104}

That mechanism can help chat, retrieval-augmented generation (RAG), few-shot prompts, and agent loops that repeat large prefixes. Its benefit depends on stable templates, identifiers, replica routing, and shared branches. Changes in those conditions can erase reuse.

SGLang also offers broad hardware paths, structured generation, speculative decoding, and distributed serving. "Prefix specialist" is now too narrow a label.^{[11]Reference 11SGLang Documentationhttps://docs.sglang.io/} Deep Dive - SGLang traces those mechanisms through its runtime.

TensorRT-LLM

TensorRT-LLM is an NVIDIA-focused serving stack with optimized attention, quantization, CUDA graphs (reusable GPU execution plans), parallelism, KV-cache reuse, speculative decoding, and disaggregated-serving paths that place prefill and decode on separate workers.^{[13]Reference 13NVIDIA TensorRT-LLM Documentation.https://docs.nvidia.com/tensorrt-llm/[21]Reference 21Paged Attention, IFB, and Request Scheduling.https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html}

Choose TensorRT-LLM by the exact support-matrix fit for your model and hardware. NVIDIA's current migration guide says the TensorRT engine backend has been removed: the PyTorch backend loads supported Hugging Face checkpoints without the old trtllm-build step.^{[22]Reference 22TensorRT Backend Removalhttps://nvidia.github.io/TensorRT-LLM/legacy/tensorrt-backend-removal.html} The rolling guide describes current migration behavior; the table pins stable tag 1.2.1. Check release notes before applying a legacy compiled-engine recipe to a newer release.

Pin one line, check its exact model and feature matrix, then benchmark it against vLLM and SGLang on the same NVIDIA system. Benchmark each architecture, quantization, and GPU generation separately.

Ollama and llama.cpp

Ollama packages a managed local daemon, model tags, REST APIs, and OpenAI-compatible local endpoints.^{[15]Reference 15OpenAI compatibility - Ollamahttps://docs.ollama.com/api/openai-compatibility} It can import GGUF artifacts, but model conversion and quantization quality remain separate decisions.^{[23]Reference 23Importing a Model - Ollamahttps://docs.ollama.com/import}

Ollama also offers cloud models. A cloud model selected through the local daemon still executes remotely, so check the chosen tag before treating a local endpoint as a local-compute or data-residency boundary.^{[24]Reference 24Cloud - Ollamahttps://docs.ollama.com/cloud}

Choose llama.cpp directly when you need an embedded C/C++ library, a custom build, explicit backend control, or its lightweight HTTP server. Choose Ollama when model lifecycle and local integration matter more than exposing every runtime knob.

Both remain options for a small service. Measure concurrency, memory behavior, authentication boundary, cancellation handling, and endpoint compatibility under that service's load.

Exact checkpoints still need a recipe

Architecture support needs a matching vendor launch recipe for the checkpoint you actually downloaded.

GLM-5.2 documents a 1M-token context and lists official SGLang (v0.5.13.post1+) and vLLM (v0.23.0+) recipes on its model card.^{[25]Reference 25GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2} Use those recipes as the first compatibility tests.

DeepSeek-V4-Flash-0731 ships a DSpark speculative module in the same checkpoint. Its published recipes name vLLM method: dspark and SGLang --speculative-algorithm DSPARK explicitly.^{[26]Reference 26DeepSeek-V4-Flash-0731 Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731} The V4 technical report frames that family around million-token context.^{[27]Reference 27DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencehttps://arxiv.org/abs/2606.19348}

Generic Transformers support or an older DeepSeek run cannot establish that sparse attention, speculative heads, quantization, and tokenizer behavior work together. Start from the vendor recipe, pin the runtime and checkpoint, then verify load, tool calls, long context, and throughput on the GPUs you'll actually use. For a community GGUF, size the downloaded artifact, offloaded weights, KV cache, and runtime workspace before assessing laptop fit.

Recipe gate: Pin the vendor command, then verify load and correctness on your GPUs.

Keep adjacent layers separate

Text Generation Inference (TGI) is in maintenance mode. Hugging Face directs new work toward engines including vLLM, SGLang, llama.cpp, and MLX.^{[28]Reference 28Text Generation Inference.https://huggingface.co/docs/text-generation-inference/index} Evaluate TGI as a migration path, with its maintenance status included in the decision.

NVIDIA Dynamo sits above an engine. It can coordinate TensorRT-LLM, vLLM, or SGLang across workers and supports patterns such as disaggregated prefill and decode.^{[29]Reference 29NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/} Comparing Dynamo with one single-node engine mixes orchestration and execution layers, so put it in a separate architecture decision.

Run a bake-off that can reverse the decision

Once the candidate set is small, write down the workload before running commands. This contract keeps a benchmark from quietly changing the question.

First record:

• exact model revision, tokenizer, chat template, quantization, and allowed remote code;
• target GPU or accelerator, driver, container, engine version, and parallel topology;
• prompt and output length distributions, not one average;
• concurrency, arrival bursts, shared-prefix fraction, tool or schema usage, and cancellation rate;
• latency SLOs for p50 and tail TTFT, TPOT or inter-token latency, and end-to-end completion;
• quality, correctness, memory headroom, deployment time, rollback time, and staff effort.

Run at least three traffic shapes: mostly unique prompts, repeated prefixes, and the real production mix. Unique prompts isolate general serving efficiency. Shared prefixes test cache reuse. The production mix tells you whether either result matters to users.

Distinguish a fixed-concurrency test, where a new request follows a completion, from an arrival-rate test, where requests arrive on a schedule. Fixed concurrency reduces offered traffic when a server slows; it can hide queue growth under a real burst. Sweep offered load, show tail latency and failures at each point, and report repeated runs rather than selecting the best run.

Warm each engine the same way and hold model semantics constant. Test cold starts and cold caches separately from warmed operation. Report output tokens per second separately from total tokens per second, and pair aggregate throughput with SLO-compliant throughput. Keep failed and cancelled requests in the offered count, and report preemptions as a separate operational signal. Compare cost per successful request only after the candidates pass the same quality and reliability gates.

Reversal test: Write down what result would make you keep the incumbent before running the benchmark. A migration without a reversal threshold is a preference disguised as measurement.

Which engine to start with

The table turns the earlier trade-offs into a starting rule. It doesn't replace the bake-off: every row still asks you to prove compatibility, SLO behavior, and operational fit.

Separate a local package from a shared server. Reject a comparison that reports only raw tokens per second, and name which part of the request path a runtime still has to prove. The engine is only one control surface. Scaling LLM Inference connects that choice to batching, KV-cache sizing, quantization, disaggregation, and the throughput-latency-cost operating point.