vLLM vs SGLang vs TensorRT-LLM vs Ollama: Choosing an Inference Engine in 2026

Fifty engineers open an internal assistant at once. The first ten get fast answers, and the rest wait even though the GPU dashboard shows 60% utilization.

That symptom doesn't pick a runtime. Inspect queueing, prefill scheduling, decode batches, and key-value (KV) cache pressure before buying a larger GPU.

Use Ollama when you want a packaged local experience. Use llama.cpp when you want a portable, embeddable GGUF runtime. For a shared GPU service, shortlist vLLM and SGLang, then add TensorRT-LLM when its exact NVIDIA model-feature matrix fits. Choose from a workload-matched bake-off, not a framework leaderboard.

Bottom line: Deployment surface narrows the field. Exact checkpoint support decides what can run. Your latency service-level objective (SLO), throughput, and operating cost decide what should run.

Start with the deployment surface

Ollama doesn't occupy the same layer as the three server engines. It packages model download, lifecycle, a local daemon, command-line interface (CLI), and application programming interfaces (APIs). One supported execution backend is llama.cpp. Ollama also runs an MLX engine on Apple Silicon. The March 2026 announcement called that path a preview; later 0.32 releases use it for several models.^{[1]Reference 1Ollama GitHub Repositoryhttps://github.com/ollama/ollama[2]Reference 2Ollama is now powered by MLX on Apple Silicon (preview)https://ollama.com/blog/mlx}

llama.cpp is the lower-level runtime and library underneath that local path. It reads GGUF model files, builds across many CPU and GPU backends, and includes llama-server, an OpenAI-compatible HTTP server.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp} Ollama is usually the easier local product surface; llama.cpp gives an application more control over embedding, builds, and runtime flags. Local LLM Deployment covers the hardware and memory sizing behind that choice. For a worked local tag-and-memory walkthrough, see Run Gemma 4 Locally with Ollama.

The server-oriented engines expose more cluster scheduling, parallelism, and model-serving controls. Their current release state changes quickly, so pin a version before comparing behavior.

These are code licenses, not model licenses. A runtime's license doesn't grant permission to use a checkpoint, tokenizer, dataset, or remote model code.

Treat these pins as inspection snapshots. Ollama's 0.32 line and llama.cpp's build tags move especially fast, so put the tag you actually run in the deploy manifest.

Release numbers also don't prove compatibility. Validate the exact checkpoint, quantization, attention backend, tool parser, structured-output path, and parallel layout. "Supports the architecture" can still mean your chosen feature combination is untested.

What an inference engine controls

Autoregressive generation has two phases. Prefill processes the prompt and writes its KV state. Decode reads that state while producing one new token per request step. Time to first token (TTFT) includes queueing, scheduling, and prefill. Time per output token (TPOT) summarizes decode pacing after the first token. Per-request decode rate is roughly the inverse of TPOT when units match, while aggregate output tokens per second also depends on concurrent work and batching.

The Inference: TTFT, TPS & KV Cache lesson derives those metrics and the cache-size formula. For this comparison, remember that a runtime controls four coupled resources:

1. Admission and scheduling: which prefills and decodes enter each step.
2. KV memory: which request owns each block and when blocks can be reused.
3. Kernels and parallelism: how model work maps to target hardware.
4. API and operations: model loading, streaming, metrics, failure recovery, and rollout artifacts.

Low utilization can come from a host-side scheduler gap, a long prefill blocking decode, insufficient queued work, collective communication, or memory admission. GPU utilization alone can't distinguish them.

Read one benchmark as a dated experiment

A March 2026 Spheron test compared vLLM v0.18.0, SGLang v0.5.9, and TensorRT-LLM v1.2.0 on one H100 SXM5 80 GB with Llama 3.3 70B Instruct in FP8. It used 200 unique prompts, average input length 512 tokens, average output length 256 tokens, concurrency 1, 10, 50, and 100, a 60-second warm-up, and three-minute measurement windows.^{[18]Reference 18vLLM vs TensorRT-LLM vs SGLang: H100 Benchmarks (2026)https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/}

Reported output tokens per second:

Latency and first-use cost on the same run:

TensorRT-LLM led this run. That statement is deliberately narrow. The source doesn't publish a runnable benchmark repository, identify the sampled prompt dataset, report repeated-run variance, or provide confidence intervals. Its unique-prompt workload also gives prefix caching little opportunity to help. Treat the table as a useful case study, not an engine ranking.

Version drift is already material. vLLM and SGLang have moved several releases beyond the tested versions. TensorRT-LLM's 1.3.0rc documentation removes the compiled TensorRT engine backend and makes PyTorch the sole execution backend, with Hugging Face checkpoints loaded directly.^{[14]Reference 14NVIDIA TensorRT-LLM Documentation.https://docs.nvidia.com/tensorrt-llm/} The 28-minute build belongs to Spheron's 1.2.0 compiled path. It isn't the default operating model for the upcoming 1.3 line.

Benchmark boundary: Never mix a compiled 1.2 TensorRT-LLM result with a 1.3 PyTorch deployment, or an old vLLM/SGLang row with current defaults, and call the delta an engine effect.

If those fifty engineers share a long incident-runbook prefix, this unique-prompt chart is the wrong test. The next question is whether the runtime can pack KV memory tightly enough to keep more of those requests live.

Why block allocation changes capacity

vLLM's original PagedAttention work made KV-cache allocation a first-class serving problem. Instead of reserving one maximum-length contiguous region per request, the runtime assigns fixed-size physical blocks as tokens arrive and keeps a logical block table.^{[19]Reference 19Efficient Memory Management for Large Language Model Serving with PagedAttentionhttps://arxiv.org/abs/2309.06180}

Suppose 6 GB remains after model weights and runtime reserve. A fixed policy that reserves 1 GB per request admits six requests. If the average live request needs 400 MB, fine-grained on-demand allocation can hold about fifteen such requests before overhead. The simplified ratio is $15/6 = 2.5$ times as many active requests.

This isn't a throughput promise. Real blocks have partially filled tails, metadata, prefix ownership, and eviction costs. It does show why allocator policy can leave compute idle while requests wait. KV Cache & PagedAttention walks through the block table and the capacity math.

What each runtime optimizes

vLLM

vLLM is a strong first server candidate when you need broad model coverage, familiar APIs, several accelerator paths, and a large operational ecosystem.^{[20]Reference 20vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionhttps://github.com/vllm-project/vllm} Its design combines paged KV memory, continuous batching, chunked prefill, parallel execution, and automatic prefix caching (APC).^{[21]Reference 21Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

APC means shared-prefix reuse isn't exclusive to SGLang. A current comparison has to measure both engines with prefix caching enabled, compatible cache isolation, and cache-aware routing. Prefix Caching and Prompt Caching covers the hit-rate mechanics. Use Deep Dive - vLLM when scheduler, block-pool, or preemption behavior is part of the decision.

SGLang

SGLang combines a high-performance server runtime with a programming frontend for multi-call language-model programs. Its founding work introduced RadixAttention, a radix-tree design that reuses exact token prefixes across requests.^{[22]Reference 22SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104}

That mechanism can help chat, retrieval-augmented generation (RAG), few-shot prompts, and agent loops that repeat large prefixes. It doesn't guarantee a win. Template changes, random identifiers near the prompt front, replica routing, or early branch divergence can erase reuse. SGLang also offers broad hardware paths, structured generation, speculative decoding, and distributed serving, so "prefix specialist" is now too narrow a label.^{[11]Reference 11SGLang Documentationhttps://docs.sglang.io/} Deep Dive - SGLang traces those mechanisms through its runtime.

TensorRT-LLM

TensorRT-LLM is an NVIDIA-focused serving stack with optimized attention, quantization, CUDA graphs, parallelism, KV-cache reuse, speculative decoding, and disaggregated-serving paths.^{[14]Reference 14NVIDIA TensorRT-LLM Documentation.https://docs.nvidia.com/tensorrt-llm/[23]Reference 23Paged Attention, IFB, and Request Scheduling.https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html}

Its current decision boundary is support-matrix fit, not a universal compile-versus-flexibility story. Stable 1.2.1 and pre-release 1.3 have different backend lifecycles. Pin one line, check its exact model and feature matrix, then benchmark it against vLLM and SGLang on the same NVIDIA system. A strong NVIDIA result for one dense model doesn't transfer automatically to another architecture, quantization, or GPU generation.

Ollama and llama.cpp

Ollama is the shortest path to a managed local daemon, model tags, REST APIs, and OpenAI-compatible local endpoints.^{[16]Reference 16OpenAI compatibility - Ollamahttps://docs.ollama.com/api/openai-compatibility} It can import GGUF artifacts, but model conversion and quantization quality remain separate decisions.^{[24]Reference 24Importing a Model - Ollamahttps://docs.ollama.com/import}

Choose llama.cpp directly when you need an embedded C/C++ library, a custom build, explicit backend control, or its lightweight HTTP server. Choose Ollama when model lifecycle and local integration matter more than exposing every runtime knob. Neither is automatically disqualified from a small service, but this Spheron H100 benchmark says nothing about either one.

Exact checkpoints still need a recipe

"Supports the architecture" is not the same as a vendor launch command for the checkpoint you actually downloaded.

GLM-5.2 documents a 1M-token context and lists official SGLang (v0.5.13.post1+) and vLLM (v0.23.0+) recipes on its model card.^{[25]Reference 25GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2} DeepSeek-V4-Flash-0731 ships a DSpark speculative module in the same checkpoint, with explicit vLLM method: dspark and SGLang --speculative-algorithm DSPARK launch commands.^{[26]Reference 26DeepSeek-V4-Flash-0731 Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731} The V4 technical report frames that family around million-token context.^{[27]Reference 27DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencehttps://arxiv.org/abs/2606.19348}

Generic Transformers support, or "we served an older DeepSeek," isn't proof that sparse attention, speculative heads, quantization, and the tokenizer path work together. Start from the vendor recipe, pin the runtime and checkpoint, then verify load, tool calls, long context, and throughput on the GPUs you'll actually use. Ollama can import community GGUF files, but these checkpoints don't belong in the laptop row above.

Recipe gate: Architecture support is not a serving recipe. Pin the vendor command, then prove load and correctness on your GPUs.

Keep adjacent layers separate

Text Generation Inference (TGI) is in maintenance mode. Hugging Face directs new work toward engines including vLLM, SGLang, llama.cpp, and MLX.^{[28]Reference 28Text Generation Inference.https://huggingface.co/docs/text-generation-inference/index}

NVIDIA Dynamo sits above an engine. It can coordinate TensorRT-LLM, vLLM, or SGLang across workers and supports patterns such as disaggregated prefill and decode.^{[29]Reference 29NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Modelshttps://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/} Comparing Dynamo with one single-node engine mixes orchestration and execution layers.

Run a bake-off that can reverse the decision

First record a workload contract:

• exact model revision, tokenizer, chat template, quantization, and allowed remote code;
• target GPU or accelerator, driver, container, engine version, and parallel topology;
• prompt and output length distributions, not one average;
• concurrency, arrival bursts, shared-prefix fraction, tool or schema usage, and cancellation rate;
• latency SLOs for p50 and tail TTFT, TPOT or inter-token latency, and end-to-end completion;
• quality, correctness, memory headroom, deployment time, rollback time, and staff effort.

Then run at least three traffic shapes: mostly unique prompts, repeated prefixes, and the real production mix. Warm each engine the same way. Hold model semantics constant. Report output tokens per second separately from total tokens per second, and pair aggregate throughput with SLO-compliant throughput. Include failed requests and preemptions rather than dropping them from the denominator.

Reversal test: Write down what result would make you keep the incumbent before running the benchmark. A migration without a reversal threshold is a preference disguised as measurement.

Which engine to start with

The engine is only one control surface. Scaling LLM Inference connects runtime choice to batching, KV-cache sizing, quantization, disaggregation, and the throughput-latency-cost operating point.