Local LLM Deployment

Quantization shrinks the model artifact. Local large language model (LLM) deployment turns that smaller artifact into a service with hardware limits, runtime choices, health checks, eval gates, and rollback.

Local LLM deployment isn't "download a model and hope." It's a serving plan.

Sometimes local deployment is the right answer: source code or incident data can't leave your boundary, latency needs to stay inside an on-prem network, the workload is steady enough to justify hardware, or the product needs offline operation. Other times, a hosted API is cheaper and safer. The engineering task is to measure the trade-off.

What is the first decision in local LLM deployment?

Answer

Define the job and its evaluation target before choosing model size. A local model is successful when it meets the workflow's quality, latency, privacy, and reliability requirements.

Start from the job

Pick the smallest local model that passes the job's evaluation, not the largest model your GPU can barely load.

For a developer or incident-assistant workflow, define the job:

The model choice follows the evaluation. If Gemma 4 12B handles incident-status questions with high accuracy, don't deploy Qwen3.6-35B-A3B for that route just because it fits somewhere.^{[1]Reference 1gemma4https://ollama.com/library/gemma4[2]Reference 2Qwen3.6-35B-A3Bhttps://huggingface.co/Qwen/Qwen3.6-35B-A3B}

GLM-5.2 and DeepSeek V4 Flash 0731 belong on the open-weight shortlist, but not in this workstation lane. GLM-5.2 is a 744B-A40B model with roughly 1.51 TB of raw BF16 weights before runtime overhead. Flash 0731 has a 284B/13B core and a mixed-precision artifact with an attached speculative module. Both offer 1M text context and MIT-licensed weights; their released artifacts demand datacenter-scale, multi-GPU serving plans. Test them as private cluster or hosted routes, not as replacements for the small local examples below.^{[3]Reference 3GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2[4]Reference 4GLM-5 Repositoryhttps://github.com/zai-org/GLM-5[5]Reference 5DeepSeek-V4-Flash-0731 Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731}

Why is "largest model that fits" the wrong default?

Answer

It spends memory, latency, power, and operational complexity before proving a quality need. Start with the smallest model that passes the route's eval, then escalate for measured failures.

Hardware budget

A local model consumes memory in several buckets:

1. Model weights.
2. KV cache for active requests.
3. Runtime buffers.
4. Activations during prefill.
5. Fragmentation and safety margin.

Gemma 4 12B may fit a local workstation route, while Qwen3.6-35B-A3B can require high-memory accelerators once reserve, KV cache, and runtime buffers are counted.^{[1]Reference 1gemma4https://ollama.com/library/gemma4[2]Reference 2Qwen3.6-35B-A3Bhttps://huggingface.co/Qwen/Qwen3.6-35B-A3B} Context length changes the math because KV cache grows with sequence length and concurrency.

Use two rough formulas before trusting any fit claim:

$\text{weight GB} \approx \frac{\text{parameters} \times \text{bits per weight}}{8 \times 10^9}$

$$\begin{aligned} \text{KV GB} \approx {} & \frac{2 \times \text{layers} \times \text{KV heads}}{10^9} \\ & \times \text{head dim} \times \text{tokens} \\ & \times \text{concurrent sequences} \times \text{bytes per value} \end{aligned}$$

The second formula is architecture-specific. Grouped-query and multi-query attention reduce KV heads, while more tokens, longer context, and more concurrent requests push the number up quickly. A hybrid model needs one term per attention type rather than one set of dimensions for every layer.

Gemma 4 12B's official config lists 48 layers: 40 sliding-attention layers with 8 KV heads of width 256, plus 8 full-attention layers with 1 global KV head of width 512. Its sliding window is 1,024 tokens.^{[6]Reference 6Gemma 4 12B Configurationhttps://huggingface.co/google/gemma-4-12B/blob/main/config.json} The estimate below makes every cache assumption explicit:

• Four independent sequences, each at 16,384 text tokens.
• Two bytes per cached scalar, matching BF16 or FP16 storage. The official checkpoint config uses BF16.
• Sliding layers evict old entries and retain at most 1,024 tokens. Full-attention layers retain all 16,384 tokens.
• Keys and values occupy separate tensors. This stays conservative even though the config declares attention_k_eq_v; a runtime that aliases them may allocate less.
• No prefix sharing or KV-cache quantization. The number is raw payload only, excluding page metadata, padding, allocator fragmentation, runtime buffers, and prefill activations.

Under those assumptions, sliding layers use 1.342 decimal GB and full-attention layers use 1.074 decimal GB across four sequences, for 2.416 GB raw K+V. Treat 2.42 GB as raw payload under these assumptions, not a runtime allocation, then measure the selected runtime's actual memory use. This chapter stays in decimal GB (1e9). If you copied GiB numbers from the quantization chapter, convert or re-run the decimal formulas here (2.416e9 / 1024**3 ≈ 2.25 GiB).

Fit needs a reserve policy, not a total under device capacity alone:

Ask these questions before buying or reserving hardware:

Local tends to be cheaper when utilization is high enough. A GPU sitting idle for most of the day is a fixed cost.

Which memory bucket do beginners most often forget when sizing local serving?

Answer

They often count model weights but forget KV cache, runtime buffers, activations during prefill, fragmentation, and safety margin. Long context and concurrency can make the KV cache the limiting term.

Where memory lives, and when bandwidth limits speed

Fit is only half the question. The other half is how fast the runtime serves your workload. During low-batch autoregressive decode, a runtime commonly streams much of the active weight set while generating one token at a time. When that weight traffic dominates, decode is memory-bandwidth-limited. Longer prompts, larger batches, cache traffic, offload, and different kernels can move the bottleneck, so treat this as a diagnostic model rather than a guarantee.

A useful upper-bound diagnostic for tokens per second during a weight-traffic-dominated decode step is:

$\text{max tokens/sec} \approx \frac{\text{memory bandwidth (GB/s)}}{\text{bytes read per token (GB)}}$

Bytes read per token can be close to the active weight footprint in this regime, but measured throughput is lower when other traffic and kernel overhead matter. Reducing weight bytes may improve decode throughput; it doesn't promise a fixed speedup.

Three hardware paths exist, and they trade capacity against bandwidth:

On Apple Silicon, MLX uses the unified-memory architecture so arrays can be operated on across CPU and GPU without copying between separate CPU and GPU pools.^{[7]Reference 7MLX: An array framework for Apple siliconhttps://github.com/ml-explore/mlx} That can make a large-memory machine a useful local candidate, but a model still needs room for artifact metadata, KV cache, runtime allocations, and the operating system. Benchmark the exact artifact and context requirement.

A sizing worksheet is more reliable than a fixed "what fits" table:

Keep the hot path resident when possible

Runtimes such as llama.cpp support CPU/GPU layer offload so a model larger than VRAM can still run.^{[8]Reference 8llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp} Fit isn't the same as speed. If each generated token crosses a CPU/GPU boundary, slower system-memory reads, PCIe transfers, and synchronization can dominate the faster GPU work. Treat hybrid offload as a measured fallback, not proof that the larger model meets latency.

Prefer this decision order:

1. Fit the selected model, KV cache, and runtime reserve fully on the target device.
2. If it doesn't fit, lower context or concurrency, choose a smaller model, or test stronger quantization.
3. Use partial offload only when no other viable path remains, then benchmark prompt processing and token generation separately on the exact hardware.

CPU thread count needs the same discipline. Setting threads above the number of physical cores can oversubscribe memory bandwidth, caches, and scheduling, lowering token-generation throughput even though more logical CPUs appear busy. Hyperthreads may help prompt processing on some machines, so don't hardcode a universal count. Sweep thread counts around the physical-core count and keep the fastest stable setting for each workload phase.

Why can a model that technically fits through CPU/GPU offload still fail a local deployment gate?

Answer

Decode may repeatedly cross the CPU/GPU boundary, making system-memory and PCIe synchronization the bottleneck. Prefer full residency or a smaller or more strongly quantized model, and sweep both offload layers and CPU thread counts with real prompt and generation benchmarks.

When is the memory-bandwidth tokens-per-second estimate useful?

Answer

It's an upper-bound diagnostic for low-batch decode when active-weight reads dominate traffic. It doesn't replace a benchmark: KV-cache traffic, kernels, batching, offload, and prefill can change the bottleneck.

Quantization, pruning, and sparsity

Quantization stores selected weights with fewer bits. GGUF, GPTQ, and AWQ are common deployment terms covered in Model Quantization, whose ideal sizes use GiB. Keep the decimal-GB worksheets here separate. A supported low-bit artifact can reduce weight storage and weight traffic; quality and end-to-end latency still need evaluation.

Four-bit is a memory format, not a losslessness certificate. A quantized model can keep acceptable perplexity or average benchmark results while failing a different set of hard examples. Studies of mathematical reasoning report capability-specific degradation under aggressive AWQ and GPTQ configurations,^{[9]Reference 9Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoninghttps://arxiv.org/abs/2501.03035} while an automated-program-repair study found that similar aggregate repair counts could hide little overlap in which bugs full-precision and quantized models repaired.^{[10]Reference 10Smaller Models, Unexpected Costs: Trade-offs in LLM Quantization for Automated Program Repairhttps://arxiv.org/abs/2606.27205} The effect depends on model, quantizer, runtime, and task, so don't generalize one artifact's result to every Q4 build.

Evaluate the actual route with code patches, multi-step reasoning, exact policy values, and long-context cases that matter in production. Perplexity is useful as one broad signal, but it doesn't prove that hard-tail behavior survived quantization.

Pruning and sparsity attack a different part of the problem. Pruning removes weights or structures. Sparsity means many values are zero or skipped. The lottery-ticket line of work showed why sparse subnetworks are scientifically interesting,^{[11]Reference 11Linear Mode Connectivity and the Lottery Ticket Hypothesis.https://arxiv.org/abs/1912.05671} but production speedups depend on runtime support. A sparse checkpoint that the runtime treats like a dense matrix may save little or nothing at inference time.

Use this rule:

For local deployment, a supported weight-quantized artifact is commonly the first compression candidate to test. Pruning and sparsity are worth tracking, but they don't create inference speed without a runtime path that uses their structure.

Why can a sparse checkpoint fail to speed up local inference?

Answer

Sparsity helps when the runtime and hardware skip the zero or pruned work efficiently. If the runtime still executes dense kernels, the checkpoint may be smaller or more interesting scientifically without making generation faster.

Runtime choice

Common local runtimes solve different jobs:

Ollama documents compatibility with parts of the OpenAI API and GGUF imports for local integrations.^{[12]Reference 12OpenAI compatibility - Ollamahttps://docs.ollama.com/api/openai-compatibility[13]Reference 13Importing a Model - Ollamahttps://docs.ollama.com/import} llama.cpp supports GGUF workflows and a server path, with device/offload options depending on build and hardware.^{[8]Reference 8llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp} LM Studio documents local model serving and supported runtimes for its releases.^{[14]Reference 14LM Studio - Local AI on your computerhttps://lmstudio.ai/} MLX targets Apple Silicon's unified-memory architecture.^{[7]Reference 7MLX: An array framework for Apple siliconhttps://github.com/ml-explore/mlx} vLLM and TensorRT-LLM are serving candidates when GPU throughput and batching are requirements.^{[15]Reference 15Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[16]Reference 16TensorRT-LLM: A High-Performance Inference Framework for LLMs.https://github.com/NVIDIA/TensorRT-LLM} Runtime support and performance move with versions, so pin the runtime, inspect supported artifact formats, and benchmark the chosen pair.

The runtime should match the operational target. A developer laptop assistant and a production incident-review service shouldn't use the same acceptance bar.

When should you prefer vLLM or TensorRT-LLM over Ollama or llama.cpp?

Answer

Evaluate vLLM or TensorRT-LLM when the target is shared GPU serving with batching and throughput requirements. Evaluate Ollama, llama.cpp, LM Studio, or MLX for their supported local, desktop, GGUF, or Apple Silicon workflows. Pin the runtime and accept it only after artifact compatibility, quality, and latency tests pass.

Containers and repeatability

Containerization closes the gap between "works on my machine" and "can be deployed again." Docker gives you a repeatable package for the runtime, model mount, tokenizer files, server flags, health checks, and dependency versions.^{[17]Reference 17Docker Documentation.https://docs.docker.com/}

A minimal local serving plan should include:

1. Container image or pinned runtime install.
2. Model artifact location and checksum.
3. Startup command with context length and quantization settings.
4. Health endpoint.
5. Metrics endpoint.
6. Eval smoke test.
7. Rollback path to previous model version.

For example, a local rollback-runbook assistant should run the same golden questions after every model or quantization change. If the model starts missing paging requirements or inventing runbook windows, the deployment fails even if the server starts.

Why isn't a health endpoint enough for local model deployment?

Answer

A health endpoint proves the process is alive, not that the model, tokenizer, quantization, prompt template, retrieval path, and context settings still answer correctly. You need a smoke eval tied to the actual workflow.

Local deployment decision

Choose local for explicit requirements: privacy, offline use, latency control, customization, or high steady utilization. Choose hosted when you need higher available capability, elastic traffic handling, or low operational burden.

Many mature systems use both: a local model handles sensitive or routine workflows, while a hosted model handles rare high-complexity cases through a gateway that records the trade-off.

Local deployment is a product commitment. Treat the model like a service: version it, containerize it, monitor it, evaluate it, and keep a fallback ready.

When should local and hosted models coexist?

Answer

Use local models for sensitive, offline, low-latency, or routine high-utilization paths. Route rare, hard, bursty, or capability-sensitive cases to a hosted model when that gives better quality or elasticity.

Work a sizing check

Use rough math before you buy hardware. Suppose you want Gemma 4 12B for an on-prem runbook assistant; these are illustrative planning numbers:

That can fit on modest hardware for a single route if the KV and runtime budget stays resident. Now change the requirement to Qwen3.6-35B-A3B in BF16, long context, and 20 concurrent users. The full checkpoint, KV cache, and interconnect requirements change the project shape. A credible answer may be quantization, a smaller model, fewer concurrent local users, sharding, or a hosted fallback for the rare hard cases.

What changes when a requirement moves from Gemma 4 12B for one secure runbook route to Qwen3.6-35B-A3B with long context and 20 concurrent users?

Answer

Total checkpoint residency becomes much larger, KV cache grows with context and concurrency, and multi-GPU or expert-aware placement may become part of the design. The decision may shift from "buy one GPU" to quantize harder, reduce local concurrency, choose a smaller model, shard the deployment, or add a hosted fallback.

Why is a local model not automatically cheaper than a hosted API?

Answer

Local serving adds hardware, power, maintenance, observability, capacity planning, and staff time. It wins when privacy, latency control, offline operation, or high steady utilization justify those costs. For bursty traffic, hosted APIs often stay cheaper because you only pay when requests arrive.

Where do pruning and sparsity fit relative to quantization?

Answer

Quantization stores each weight with fewer bits. Pruning and sparsity remove or skip weights and activations. Sparse models need runtime and hardware support to turn that structure into speed, so they are less plug-and-play than standard quantized checkpoints.

Common deployment failures

Choosing the largest local model first

Symptom: The route starts with Qwen3.6-35B-A3B before anyone defines the quality target. Cause: Model size is treated as the goal instead of a cost paid only when smaller models fail. Fix: Start with the workflow eval, try the smallest model that passes, then escalate only for measured failure cases.

Counting weights and forgetting KV cache

Symptom: A quantized checkpoint fits on paper but crashes or thrashes once context length and concurrency increase. Cause: The sizing plan counts model weights but omits KV cache, runtime buffers, activations, fragmentation, and margin. Fix: Budget weights, KV cache, buffers, and 20-30 percent headroom before buying hardware or setting num_ctx.

Treating offload and threads as free capacity

Symptom: The model fits after partial GPU offload, or all logical CPUs are busy, but token generation slows sharply. Cause: The hot path crosses CPU/GPU memory boundaries or threads contend for the same bandwidth and cache. Fix: Prefer full residency or a smaller or more strongly quantized artifact. Sweep offload layers and thread counts around the physical-core count with separate prefill and decode measurements.

Calling Q4 lossless after one aggregate score

Symptom: Perplexity and average accuracy look acceptable, but code repair or multi-step reasoning regresses. Cause: Aggregate metrics hid which hard examples changed under quantization. Fix: Compare full-precision and quantized artifacts on route-specific hard cases and reject the artifact if those acceptance criteria regress.

Deploying without a rollback path

Symptom: The local model starts, but a bad quantization or prompt-template change breaks real incident answers. Cause: Deployment only checks process health and lacks a previous model artifact, routing fallback, and smoke eval. Fix: Keep the old artifact, checksum both versions, route through a gateway, and require golden-case evals before traffic.

Testing only startup

Symptom: The local server starts, health check passes, and the first live engineer gets a wrong rollback answer. Cause: Startup checks prove the process is alive. They don't prove the tokenizer, quantization, prompt template, context length, or policy retrieval path still produce correct answers. Fix: Add a smoke eval to deployment. Ask five golden questions covering rollback status, paging windows, missing deploy IDs, source-sensitive abstention, and citation quality. The deploy passes when the local model gives acceptable answers, citations, and latency on those cases.

What should a deployment smoke eval include for a local rollback-runbook assistant?

Answer

It should cover the real failure modes: rollback status, paging windows, missing deploy IDs, sensitive-source abstention, citation quality, and latency. Startup alone doesn't prove the route is safe to serve.