Local LLM Deployment

Imagine an internal runbook assistant whose quantized download is 7.6 GB. It loads on a 24 GB GPU and answers a short question. Four engineers then submit long incident logs at once, and a request fails with out-of-memory (OOM). The download fitted. The workload did not.

That is a hypothetical incident, not a benchmark. Before diagnosing it, ask what else was allocated, on which device, and at what point in the request. Startup success alone answers none of those questions.

Our assistant retrieves authorized post-mortems and drafts rollback commands for human review. Credentials remain in the credential store rather than becoming retrieval documents. Organizational policy requires incident data to stay inside the approved network. If local inference fails, the fallback is approved local documentation or a human engineer, with no automatic public API failover.

Model Quantization explains weight compression. KV Cache and PagedAttention explains attention history. This lesson turns those mechanisms into a sizing worksheet, a placement check, and a deployment experiment you can reproduce on your own hardware.

Follow the bytes before counting the cores

Single-sequence dense-model decode often has low arithmetic intensity: a matrix must supply many weight bytes for one new token. Memory bandwidth can dominate. That is a workload-dependent bottleneck, not a law that all local inference is strictly bandwidth-bound. Batching reuses weights across sequences; attention, unpacking, launches, CPU work, and long contexts can change the limiting resource.

Discrete GPU: capacity and bandwidth are different limits

The GeForce RTX 4090 is a useful named example, rather than the current consumer flagship. NVIDIA specifies 24 GB GDDR6X, a 384-bit interface, and 1,008 GB/s peak memory bandwidth.^{[1]Reference 1NVIDIA Ada GPU Architecture Whitepaperhttps://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf} Peak bandwidth is not a measured decode rate. Four-bit stored weights also do not imply every multiplication uses native INT4 instructions: W4A16 kernels can unpack into floating-point arithmetic.

Host RAM does not become extra ordinary CUDA device memory just because the machine has plenty of it. A runtime can intentionally execute some layers on CPU, transfer data, or use a supported managed-memory path. An ordinary device allocation can instead fail. Establish which path your installed backend uses before explaining a slowdown.

Apple Silicon: shared storage still has a working-set limit

Unified memory lets CPU and GPU access the same physical memory pool without copying an array across a discrete GPU's PCIe link. Shared physical storage does not require identical CPU/GPU virtual addresses, eliminate synchronization, or make all installed RAM freely available to the model.

As checked September 22, 2026, Apple's Mac Studio specifications list M5 Max variants at 460 or 614 GB/s and M5 Ultra at 1.2 TB/s, with supported memory configurations reaching 128 GB and 512 GB respectively.^{[2]Reference 2Mac Studio Technical Specificationshttps://www.apple.com/mac-studio/specs/} These are advertised hardware specifications. They establish neither a universal model-fit guarantee nor a token rate.

Include macOS, other applications, runtime caches, and the Metal working-set constraints in your budget. There is no universal rule that macOS reserves exactly 20–25%. In MLX, supported execution devices are CPU and GPU; the framework does not expose the Neural Engine as an execution device.^{[3]Reference 3MLX: An array framework for Apple siliconhttps://github.com/ml-explore/mlx[4]Reference 4MLX Unified Memoryhttps://ml-explore.github.io/mlx/build/html/usage/unified_memory.html} Evaluate MLX-LM against another compatible runner using the same artifact quality, prompt, and concurrency before claiming one is faster.

CPU: channels, locality, and workload shape

For a hypothetical dual-channel DDR5-5600 system, the transfer-rate calculation is $2 \times 5.6\text{ GT/s} \times 8\text{ bytes} = 89.6\text{ GB/s}$ before practical losses. Eight such channels give 358.4 GB/s theoretically. These are arithmetic examples, not limits shared by every EPYC or Xeon platform. Check the processor's supported channels, DIMM population, transfer rate, and NUMA placement.

More threads help until another resource dominates; they can also increase contention. Measure a thread-count sweep instead of prescribing a token rate from core count. A server with more sockets can have more aggregate bandwidth while a poorly placed process still accesses remote memory.

Build a budget for each memory pool

Estimate before downloading a large artifact, then replace estimates with measurements after loading it. For a particular device and a particular request phase:

$$M_{\text{peak}} \approx M_{\text{weights}} + M_{\text{cache}} + M_{\text{workspace}} + M_{\text{other}}$$

The terms must refer to allocations that coexist. Do not sum unrelated peaks, count reserved allocator memory twice, or mix CPU RAM and GPU VRAM into one fictitious pool. For a split deployment, make separate host and device budgets. Unified-memory machines instead share a capacity constraint with the operating system.

Weight payload is a starting point

For $N$ uniformly stored weights at $b$ bits, the ideal payload is $Nb/8$ bytes. Real artifacts include grouping metadata and a mixture of tensor precisions. A dense array of zeros does not shrink just because someone calls it pruned; a sparse representation and compatible execution path are separate requirements.

Google's model card lists Gemma 4 12B Unified at 11.95B parameters.^{[5]Reference 5Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4} Its hypothetical uniform 16-bit payload would be 23.9 decimal GB; four-bit codes alone would take 5.975 GB. The Ollama gemma4:12b listing checked September 22 shows a 7.6 GB artifact with Q4_K_M model data and a BF16 projector.^{[6]Reference 6gemma4 tagshttps://ollama.com/library/gemma4/tags} File size alone does not reveal the loaded device footprint or explain every byte of the difference. Preserve the exact local artifact identity, because the tag can change.

Cache arithmetic: read the architecture, not just “12B”

For $L$ homogeneous full-attention layers with separate key/value buffers:

$$M_{\text{KV,raw}} = 2 L H_{\text{KV}} D T B s$$

$H_{\text{KV}}$ is the KV-head count, $D$ the configured head width, $T$ retained tokens per sequence, $B$ concurrent sequences, and $s$ bytes per scalar. Do not automatically derive $D$ as hidden width divided by query-head count; architectures can use a different projection width. Sum layer types separately when dimensions differ. This formula excludes page rounding, quantization metadata, temporary updates, and reserved-but-unused cache space.

A hypothetical 32-layer model with eight KV heads of width 128 and two-byte scalars needs $2 \times 32 \times 8 \times 128 \times 2 = 131{,}072$ bytes per retained token, or 128 KiB. At 4,096 tokens that is 0.5 GiB (536.9 decimal MB). At 32,768 tokens across four sequences it is 16 GiB (17.18 decimal GB), before workspace.

Caching avoids recomputing the old prefix's states. Full-attention decode still reads history and its attention work grows with retained length. Re-evaluating a whole full-attention prefix would instead include quadratic attention work in that prefix. Cache projection work alone is not quadratic.

Gemma 4 12B's inspected text configuration has 40 sliding layers with eight KV heads of width 256 and a 1,024-token window, plus eight full layers with one KV head of width 512.^{[7]Reference 7Gemma 4 12B Configurationhttps://huggingface.co/google/gemma-4-12B/blob/023679ed352de9bb66cc873c9009ce3482585c08/config.json} It also enables shared K/V projection in global layers. That does not automatically mean one physical cache array: the inspected Transformers implementation separately normalizes/transforms key and value states and updates both in the cache.^{[8]Reference 8Transformers Gemma 4 Unified Attention and Cache Updatehttps://github.com/huggingface/transformers/blob/2c4914fb939fe9de0d8e7a798af4684d552f18b4/src/transformers/models/gemma4_unified/modeling_gemma4_unified.py}

For a worksheet assuming separate 16-bit K/V arrays and retention capped at 1,024 tokens in sliding layers, four 16,384-token sequences require:

• Sliding payload: $2 \times 40 \times 8 \times 256 \times 1{,}024 \times 4 \times 2 = 1.342\text{ GB}$.
• Full payload: $2 \times 8 \times 1 \times 512 \times 16{,}384 \times 4 \times 2 = 1.074\text{ GB}$.
• Combined raw payload: 2.416 GB.

If the runtime stores full-length buffers for the sliding layers, the same two-array calculation gives 22.549 GB. Keeping those buffers is not, by itself, an attention-correctness bug: the attention mask can still enforce the sliding window. A runtime may reserve full buffers by design. Verify storage behavior separately from the mathematical attention pattern.

Workspace and reserve are inputs to the experiment

Memory-efficient attention avoids materializing the full quadratic score matrix. It does not provide a universal 1.5–2.5 GB prefill workspace bound. Prefill chunks, logits, graph captures, kernel workspaces, model dimensions, and concurrency all matter.

Likewise, display and CUDA context costs are not fixed constants across machines. Measure free device memory and other processes, then capture the server's peak under the intended request shape. A chosen 20% reserve is a planning assumption, not a guarantee against OOM or a substitute for those measurements.

A worksheet that makes its assumptions visible

This CPU exercise uses the Gemma layer dimensions above, a 7.6 GB weight proxy, an assumed 2 GB working allocation, and a 20% reserve on a hypothetical 24 GB device. It predicts no measured runtime allocations. evict=False changes storage retention; it does not change the model's attention semantics.

Doubling context from 16K to 32K adds 1.074 GB under capped retention. Doubling concurrency doubles this raw cache, not the shared weights or every allocation in the process. Full-length sliding storage takes the modeled total to 32.149 GB, beyond the assumed device capacity. Whether the server rejects the configuration, preempts requests, offloads, or fails allocation depends on its implementation.

Try changing the worksheet: keep four sessions, reduce the assumed working term to 1 GB, and then remove the reserve. Does the full-storage case fit? It still exceeds 24 GB. Next, keep capped retention but double sessions: identify which terms need new measurements rather than multiplying the whole total by two.

Estimate block capacity, then verify the offload flag

llama.cpp supports CPU/GPU placement through -ngl and related controls.^{[9]Reference 9llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp} A block left on CPU executes there using its supported backend; this differs from transferring its entire weight payload to GPU each token.

For a hypothetical model with equal-size blocks, assume every non-block allocation, including the output head at its intended placement, is already covered by $M_{\text{fixed}}$. Then the block-capacity estimate is:

$$n_{\text{blocks}} = \min\left(L, \left\lfloor\frac{M_{\text{budget}}-M_{\text{fixed}}}{M_{\text{block}}}\right\rfloor\right)$$

With 32 blocks totaling 9.6 GB and 3.2 GB fixed usage, a 16 GB modeled budget accommodates all 32 blocks. An 8 GB budget accommodates $(8-3.2)/0.3 = 16$ blocks, leaving the other 16 on CPU. Neither result includes a reserve unless one was included in the chosen budget.

Do not label that integer an exact, universally safe -ngl value. Hybrid blocks, MoE blocks, tensor overrides, cache placement, backend buffers, and precision mixtures can make block sizes unequal. Use actual tensor/suffix sizes and startup logs, then measure peak memory.

Flag semantics are versioned too. In the llama.cpp source inspected at commit 709fe755, the trailing offload range includes an output-layer slot. With 40 transformer blocks, -ngl 24 selects the last 23 blocks plus the output slot, before tensor/backend overrides. It does not mean 24 hidden blocks; input-layer assignment remains on CPU.^{[10]Reference 10llama.cpp Model Tensor Placementhttps://github.com/ggml-org/llama.cpp/blob/709fe755dfa810d77e2ac386292b29648b536864/src/llama-model.cpp} Check your installed version's help and allocation logs rather than copying counts from an older tutorial.

Allocation failure, offload, and paging are different events

A CUDA OOM message does not prove that weights first migrated through host RAM. Distinguish four paths:

NVIDIA's current guide distinguishes device-only allocation, managed memory, mapped host memory, HMM/ATS support, and limited-support platforms.^{[11]Reference 11CUDA Runtime API Memory Managementhttps://docs.nvidia.com/cuda/cuda-runtime-api/cuda_runtime_api/group__CUDART__MEMORY.html[12]Reference 12CUDA Programming Guide: Unified and System Memoryhttps://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/understanding-memory.html} Ordinary cudaMalloc allocations do not acquire automatic oversubscription merely because Unified Memory exists.

mmap creates a virtual file mapping, not a promise that the file is resident. File-backed weight rereads are not necessarily swap reads. Nor does GPU cache growth automatically evict ordinary CUDA weight allocations into host RAM: a server can reject, preempt, or fail instead.

A transfer calculation you can falsify

Suppose one step must read a 7.6 GB payload. Ignore compute, attention, unpacking, and launch overhead. Use 1,008 GB/s as a peak device-memory scenario, assume 25 GB/s for a host-to-device path, and assume 5 GB/s for storage. These are serial weight-stream calculations, not a Gemma benchmark or a prediction of what your runtime migrates.

The 11.35× ratio compares two modeled stream times with the same assumptions. It cannot be compared directly with an unrelated measured 50 tokens/s to claim a fivefold slowdown. CPU-executed layers have CPU compute and activation-boundary costs; overlapped transfer has another timing model; disk reads also need subsequent processing. Profile the path your server actually takes.

Choose the runner by the workload it must pass

GGUF is a tensor/metadata container, not “GPT-Generated Unified Format.” It can be sharded, and mapping it does not make startup instantaneous.^{[13]Reference 13GGUF Specificationhttps://github.com/ggml-org/ggml/blob/master/docs/gguf.md} llama.cpp can run through native binaries without PyTorch/Python as the inference runtime, but its chosen backends still have dependencies. In the inspected CLI source, cache types include f16, bf16, q8_0, and q4_0; generic FP8 is not in that accepted list. Quantized cache carries metadata, has backend restrictions, and needs task evaluation.^{[14]Reference 14llama.cpp CLI Cache and GPU-Layer Argumentshttps://github.com/ggml-org/llama.cpp/blob/709fe755dfa810d77e2ac386292b29648b536864/common/arg.cpp}

A Modelfile is configuration, not a weight serialization format. Ollama's import documentation covers supported Safetensors and GGUF inputs.^{[15]Reference 15Importing a Model - Ollamahttps://docs.ollama.com/import[16]Reference 16Ollama GitHub Repositoryhttps://github.com/ollama/ollama} Its API compatibility covers a subset of OpenAI endpoints/features. A basic client can target http://localhost:11434/v1, but model names, supported fields, context settings, and conversation behavior still need checking. The placeholder API key used by some clients does not secure the unauthenticated local server.^{[17]Reference 17OpenAI compatibility - Ollamahttps://docs.ollama.com/api/openai-compatibility[18]Reference 18Authenticationhttps://docs.ollama.com/api/authentication}

MLX is the array framework; MLX-LM supplies language-model tools. Its supported CPU/GPU operations share unified arrays, with dependencies scheduled between streams. There is no universal “MLX beats every Metal runner” result.^{[3]Reference 3MLX: An array framework for Apple siliconhttps://github.com/ml-explore/mlx[4]Reference 4MLX Unified Memoryhttps://ml-explore.github.io/mlx/build/html/usage/unified_memory.html}

vLLM's PagedAttention maps logical token blocks to cache blocks. These are not ordinary OS virtual-memory pages. Waste depends on block size, sequence lengths, sharing, and pool policy; “under 4%” is not a bound for every workload. It offers continuous batching and distributed serving, while Ollama and llama-server also support concurrent requests. Ollama documents parallel-request and queue controls, with memory implications.^{[19]Reference 19vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionhttps://github.com/vllm-project/vllm[20]Reference 20FAQ - Ollamahttps://docs.ollama.com/faq}

Draw the private-data boundary

Local inference, on-premises hosting, and air-gapping are different properties. A localhost API can still forward a request to a cloud model. An on-premises server can have Internet access. An air gap requires an actual separation from external networks, with controlled artifact/update transfer.

Local hosting alone does not establish legal compliance. HIPAA permits cloud processing under applicable agreements, safeguards, and other HIPAA requirements; it is not a blanket cloud ban.^{[21]Reference 21Guidance on HIPAA and Cloud Computinghttps://www.hhs.gov/hipaa/for-professionals/special-topics/health-information-technology/cloud-computing/index.html} GDPR international-transfer rules likewise provide transfer mechanisms, including adequacy and appropriate safeguards.^{[22]Reference 22International Transfers and International Cooperationhttps://www.edpb.europa.eu/topics/international-transfers-and-international-cooperation_en} Export controls and contractual restrictions require assessment of the particular data, recipients, and access arrangement. Do not replace that assessment with “commercial cloud is always prohibited.”

For the runbook assistant's stated internal-only policy:

1. Disable Ollama cloud features. OLLAMA_NO_CLOUD=1 disables Ollama's cloud-model and web-search features after restart. It is not documented as a firewall, registry-download ban, or blanket telemetry switch.^{[20]Reference 20FAQ - Ollamahttps://docs.ollama.com/faq}
2. Control inbound access. Ollama defaults to loopback and its local API has no built-in authentication requirement. Loopback limits direct network access, but does not authenticate local processes or protect an exposed tunnel/proxy. Multi-user access needs an authenticating gateway and caller-scoped retrieval authorization.^{[18]Reference 18Authenticationhttps://docs.ollama.com/api/authentication[20]Reference 20FAQ - Ollamahttps://docs.ollama.com/faq}
3. Verify the entire outbound path. Apply and test appropriate network policy for the server, retrieval, tools, proxies, logs, and monitoring. Generated curl text is not execution; a tool runner creates that boundary. Egress restrictions reduce permitted destinations but do not stop every leak through allowed peers, local files, or logs.
4. Exercise failure behavior. Stop the model worker in a controlled test and submit an unavailable-runbook case. Require an explicit local fallback and no external forwarding. A model's self-reported confidence is not sufficient evidence of correctness.

Inspect placement and measure what users receive

Start with a downloaded, approved artifact and a running local server. The commands below inspect the installation; their output depends on your machine.

/api/tags is inventory. /api/ps lists loaded models and reported size, VRAM size, digest, and context length.^{[23]Reference 23Ollama API: List Running Modelshttps://docs.ollama.com/api/ps} The CLI's PROCESSOR column summarizes CPU/GPU placement. 100% GPU is not a guarantee that every operation runs on Tensor Cores, that peak memory fits, or that latency is acceptable. Percentages are not an exact hidden-layer count.^{[20]Reference 20FAQ - Ollamahttps://docs.ollama.com/faq}

Separate the server's timings from client latency

This native Ollama request fixes the context and output caps. Use a locally approved model; the registry tag here is a reproducible request example, not an immutable production identity.^{[24]Reference 24Generate a Responsehttps://docs.ollama.com/api/generate}

Ollama reports timing fields in nanoseconds. Current usage documentation distinguishes total prompt tokens, cached prompt tokens, uncached-prompt evaluation duration, and output generation count/duration.^{[25]Reference 25Usagehttps://docs.ollama.com/api/usage} Dividing all prompt tokens by the uncached duration can overstate prefill performance. Reported generation counts should not be assumed to count only visible final-answer text; record thinking mode too. Earlier server versions can differ, so preserve the runtime version and actual receipt.

The helper below uses synthetic receipts, not a model run. Save it as read-generation-timing.py; passing your JSON filename prints its rates. Missing metrics and zero durations produce an unavailable rate, rather than an invented one-nanosecond duration or division by zero.

Run python3 read-generation-timing.py local-response.json after the request. JSON null means the rate cannot be computed under this contract. A missing cached-count field does not justify guessing a cache hit count.

Those rates do not measure time to first token (TTFT) or individual token gaps. For those, enable streaming and record client arrival timestamps, stating whether the first thinking/content token or first chunk defines TTFT. Transport buffering can group tokens. Also measure client completion time, queueing, cold versus warm loads, cache reuse, peak memory, and failures under the intended arrival load. Report rate distributions with the workload and measurement boundaries; one successful short request is not a concurrency benchmark.

Keep the case identities when comparing quantization

A model that fits is still unsuitable if it corrupts a rollback flag. Run baseline and candidate on the same prompts, retrieved context, templates, and declared decoding policy. Keep per-case outputs and task-specific checks, not just one aggregate score. Repeated trials help distinguish a systematic change from sampling variation; a fixed seed does not guarantee identical results across implementations.

These are invented pass/fail outcomes showing the accounting problem. The fixture executes no language model and establishes no quantization regression on Gemma.

Both dictionaries total 75%, but the cases differ. Under a release policy that rejects critical command regressions, the candidate fails. The baseline's failed abstention case also needs repair; comparison does not certify it as safe. The candidate's abstention improvement is useful evidence, not merely a lucky guess.

Diagnose with competing explanations

A restart does not itself re-pull a Docker image or an Ollama model. If a deployment step does pull mutable latest or model tags, it can install changed content. Use immutable container image digests and a verified local model artifact/manifest, with tokenizer, template, quantization, runtime, and options recorded. Do not assume Ollama accepts Docker-style model@sha256:... names.

Your deployment receipt: write down artifact identities, hardware/driver/runtime versions, placement, context and concurrency limits, cache/workspace policy, peak memory, the latency workload, per-case quality results, and the tested local fallback. Each field answers a different question. An image hash proves identity; it does not prove task quality or an enforced private-data boundary.