Local LLM Deployment

Quantization shrinks the model artifact. Local large language model (LLM) deployment turns that smaller artifact into a service with hardware limits, runtime choices, health checks, eval gates, and rollback.

Local LLM deployment isn't "download a model and hope." It's a serving plan.

Sometimes local deployment is the right answer: source code or incident data can't leave your boundary, latency needs to stay inside an on-prem network, the workload is steady enough to justify hardware, or the product needs offline operation. Other times, a hosted API is cheaper and safer. The engineering task is to measure the trade-off.

Start from the job

Pick the smallest local model that passes the job's evaluation, not the largest model your GPU can barely load.

For a developer or incident-assistant workflow, define the job:

The model choice follows the evaluation. If Gemma 4 12B handles incident-status questions with high accuracy, don't deploy Qwen3.6-35B-A3B for that route just because it fits somewhere.^[1][2]

Hardware budget

A local model consumes memory in several buckets:

1. Model weights.
2. KV cache for active requests.
3. Runtime buffers.
4. Activations during prefill.
5. Fragmentation and safety margin.

Gemma 4 12B may fit a local workstation route, while Qwen3.6-35B-A3B can require high-memory accelerators once reserve, KV cache, and runtime buffers are counted.^[1][2] Context length changes the math because KV cache grows with sequence length and concurrency.

Use two rough formulas before trusting any fit claim:

$\text{weight GB} \approx \frac{\text{parameters} \times \text{bits per weight}}{8 \times 10^9}$

$$\begin{aligned} \text{KV GB} \approx {} & \frac{2 \times \text{layers} \times \text{KV heads}}{10^9} \\ & \times \text{head dim} \times \text{tokens} \\ & \times \text{concurrent sequences} \times \text{bytes per value} \end{aligned}$$

The second formula is architecture-specific. Grouped-query and multi-query attention reduce KV heads, while longer context and more concurrent requests push the number up quickly.

Fit needs a reserve policy, not a total under device capacity alone:

Ask these questions before buying or reserving hardware:

Local tends to be cheaper when utilization is high enough. A GPU sitting idle for most of the day is a fixed cost.

Where memory lives, and when bandwidth limits speed

Fit is only half the question. The other half is how fast the runtime serves your workload. During low-batch autoregressive decode, a runtime commonly streams much of the active weight set while generating one token at a time. When that weight traffic dominates, decode is memory-bandwidth-limited. Longer prompts, larger batches, cache traffic, offload, and different kernels can move the bottleneck, so treat this as a diagnostic model rather than a guarantee.

A useful upper-bound diagnostic for tokens per second during a weight-traffic-dominated decode step is:

$\text{max tokens/sec} \approx \frac{\text{memory bandwidth (GB/s)}}{\text{bytes read per token (GB)}}$

Bytes read per token can be close to the active weight footprint in this regime, but measured throughput is lower when other traffic and kernel overhead matter. Reducing weight bytes may improve decode throughput; it doesn't promise a fixed speedup.

Three hardware paths exist, and they trade capacity against bandwidth:

On Apple Silicon, MLX uses the unified-memory architecture so arrays can be operated on across CPU and GPU without copying between separate CPU and GPU pools.^[3] That can make a large-memory machine a useful local candidate, but a model still needs room for artifact metadata, KV cache, runtime allocations, and the operating system. Benchmark the exact artifact and context requirement.

A sizing worksheet is more reliable than a fixed "what fits" table:

Quantization, pruning, and sparsity

Quantization stores selected weights with fewer bits. GGUF, GPTQ, and AWQ are common deployment terms you saw in the quantization chapter. A supported low-bit artifact can reduce weight storage and weight traffic; quality and end-to-end latency still need evaluation.

Pruning and sparsity attack a different part of the problem. Pruning removes weights or structures. Sparsity means many values are zero or skipped. The lottery-ticket line of work showed why sparse subnetworks are scientifically interesting,^[4] but production speedups depend on runtime support. A sparse checkpoint that the runtime treats like a dense matrix may save little or nothing at inference time.

Use this rule:

For local deployment, a supported weight-quantized artifact is commonly the first compression candidate to test. Pruning and sparsity are worth tracking, but they don't create inference speed without a runtime path that uses their structure.

Runtime choice

Common local runtimes solve different jobs:

Ollama documents compatibility with parts of the OpenAI API and GGUF imports for local integrations.^[5][6] llama.cpp supports GGUF workflows and a server path, with device/offload options depending on build and hardware.^[7] LM Studio documents local model serving and supported runtimes for its releases.^[8] MLX targets Apple Silicon's unified-memory architecture.^[3] vLLM and TensorRT-LLM are serving candidates when GPU throughput and batching are requirements.^[9][10] Runtime support and performance move with versions, so pin the runtime, inspect supported artifact formats, and benchmark the chosen pair.

The runtime should match the operational target. A developer laptop assistant and a production incident-review service shouldn't use the same acceptance bar.

Containers and repeatability

Containerization closes the gap between "works on my machine" and "can be deployed again." Docker gives you a repeatable package for the runtime, model mount, tokenizer files, server flags, health checks, and dependency versions.^[11]

A minimal local serving plan should include:

1. Container image or pinned runtime install.
2. Model artifact location and checksum.
3. Startup command with context length and quantization settings.
4. Health endpoint.
5. Metrics endpoint.
6. Eval smoke test.
7. Rollback path to previous model version.

For example, a local rollback-runbook assistant should run the same golden questions after every model or quantization change. If the model starts missing paging requirements or inventing runbook windows, the deployment fails even if the server starts.

Local deployment decision

Choose local for explicit requirements: privacy, offline use, latency control, customization, or high steady utilization. Choose hosted when you need higher available capability, elastic traffic handling, or low operational burden.

Many mature systems use both: a local model handles sensitive or routine workflows, while a hosted model handles rare high-complexity cases through a gateway that records the trade-off.

Local deployment is a product commitment. Treat the model like a service: version it, containerize it, monitor it, evaluate it, and keep a fallback ready.

Work a sizing check

Use rough math before you buy hardware. Suppose you want Gemma 4 12B for an on-prem runbook assistant; these are illustrative planning numbers:

That can fit on modest hardware for a single route if the KV and runtime budget stays resident. Now change the requirement to Qwen3.6-35B-A3B in BF16, long context, and 20 concurrent users. The full checkpoint, KV cache, and interconnect requirements change the project shape. A credible answer may be quantization, a smaller model, fewer concurrent local users, sharding, or a hosted fallback for the rare hard cases.

What to check before moving on

Check that you can:

• Choose a local model and quantization level from explicit latency, privacy, and quality targets.
• Explain when local deployment is about data control or offline operation rather than lower cost.
• Build a repeatable serving plan with pinned runtime, model checksum, container, health check, smoke eval, and rollback.
• Estimate weight memory and KV-cache memory before claiming a model fits.
• Explain why local decode speed is bounded by memory bandwidth, and estimate tokens per second from bandwidth and model size.
• Compare discrete GPU VRAM, Apple Silicon unified memory, and CPU paths, then build a reserve-aware fit worksheet for the candidate machine.
• Decide whether Ollama, llama.cpp, LM Studio, MLX, vLLM, or TensorRT-LLM matches the deployment target.

Production questions

Why is a local model not automatically cheaper than a hosted API?

Local serving adds hardware, power, maintenance, observability, capacity planning, and staff time. It wins when privacy, latency control, offline operation, or high steady utilization justify those costs. For bursty traffic, hosted APIs often stay cheaper because you only pay when requests arrive.

Where do pruning and sparsity fit relative to quantization?

Quantization stores each weight with fewer bits. Pruning and sparsity remove or skip weights and activations. Sparse models need runtime and hardware support to turn that structure into speed, so they are less plug-and-play than standard quantized checkpoints.

Common pitfalls

Choosing the largest local model first

• Symptom: The route starts with Qwen3.6-35B-A3B before anyone defines the quality target.

• Cause: Model size is treated as the goal instead of a cost paid only when smaller models fail.

• Fix: Start with the workflow eval, try the smallest model that passes, then escalate only for measured failure cases.

Counting weights and forgetting KV cache

• Symptom: A quantized checkpoint fits on paper but crashes or thrashes once context length and concurrency increase.

• Cause: The sizing plan counts model weights but omits KV cache, runtime buffers, activations, fragmentation, and margin.

• Fix: Budget weights, KV cache, buffers, and 20-30 percent headroom before buying hardware or setting num_ctx.

Deploying without a rollback path

• Symptom: The local model starts, but a bad quantization or prompt-template change breaks real incident answers.

• Cause: Deployment only checks process health and lacks a previous model artifact, routing fallback, and smoke eval.

• Fix: Keep the old artifact, checksum both versions, route through a gateway, and require golden-case evals before traffic.

Testing only startup

• Symptom: The local server starts, health check passes, and the first live engineer gets a wrong rollback answer.

• Cause: Startup checks prove the process is alive. They don't prove the tokenizer, quantization, prompt template, context length, or policy retrieval path still produce correct answers.

• Fix: Add a smoke eval to deployment. Ask five golden questions covering rollback status, paging windows, missing deploy IDs, source-sensitive abstention, and citation quality. The deploy passes when the local model gives acceptable answers, citations, and latency on those cases.