Local LLM Deployment

Imagine an internal runbook assistant whose quantized download is 7.6 GB. It loads on a 24 GB GPU and answers a short question. Four engineers then submit long incident logs at once, and a request fails with out-of-memory (OOM). The download fitted. The workload did not.

Measure allocations, device placement, and request phase when diagnosing a real failure. The successful short request validates that workload; concurrent long prompts can expose different memory peaks.

Our assistant retrieves authorized post-mortems and drafts rollback commands for human review. Credentials remain in the credential store rather than becoming retrieval documents. Organizational policy requires incident data to stay inside the approved network. If local inference fails, the fallback is approved local documentation or a human engineer, with no automatic public API failover.

Model Quantization explains the 7.6 GB file. KV Cache and PagedAttention explains why four long logs need more room than that file. Neither one says which device held the bytes when the allocation failed.

Follow the bytes before counting the cores

An out-of-memory failure occurs when the live model, cache, and workspace exceed the memory available to their execution path. A discrete GPU, Apple Silicon, and a CPU offer different memory pools, so follow where each allocation resides.

Single-sequence dense-model decode often has low arithmetic intensity: a matrix must supply many weight bytes for one new token, so memory bandwidth can dominate. Batching reuses weights across sequences; attention, unpacking, launches, CPU work, and long contexts can shift the bottleneck. Measure the target workload to identify its limiting resource.

Discrete GPU: capacity and bandwidth are different limits

The GeForce RTX 4090 provides a concrete capacity and bandwidth example. NVIDIA specifies 24 GB GDDR6X, a 384-bit interface, and 1,008 GB/s peak memory bandwidth.^{[1]Reference 1NVIDIA Ada GPU Architecture Whitepaperhttps://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf} Peak bandwidth describes the hardware specification; measure decode rate on the target workload. Four-bit stored weights also do not imply every multiplication uses native INT4 instructions: W4A16 kernels can unpack into floating-point arithmetic.

Host RAM does not become extra ordinary CUDA device memory just because the machine has plenty of it. A runtime can intentionally execute some layers on CPU, transfer data, or use a supported managed-memory path. An ordinary device allocation can instead fail. Establish which path your installed backend uses before explaining a slowdown.

Apple Silicon: shared storage still has a working-set limit

Unified memory lets CPU and GPU access the same physical memory pool without copying an array across a discrete GPU's PCIe link. Shared physical storage does not require identical CPU/GPU virtual addresses, eliminate synchronization, or make all installed RAM freely available to the model.

As checked September 22, 2026, Apple's Mac Studio specifications list M5 Max variants at 460 or 614 GB/s and M5 Ultra at 1.2 TB/s, with supported memory configurations reaching 128 GB and 512 GB respectively.^{[2]Reference 2Mac Studio Technical Specificationshttps://www.apple.com/mac-studio/specs/} These are advertised hardware specifications. Model fit and token rate depend on the actual model, context, and runtime, so measure them on the target system.

Include macOS, other applications, runtime caches, and the Metal working-set constraints in your budget. There is no universal rule that macOS reserves exactly 20–25%. In MLX, supported execution devices are CPU and GPU; the framework does not expose the Neural Engine as an execution device.^{[3]Reference 3MLX: An array framework for Apple siliconhttps://github.com/ml-explore/mlx[4]Reference 4MLX Unified Memoryhttps://ml-explore.github.io/mlx/build/html/usage/unified_memory.html} Evaluate MLX-LM against another compatible runner using the same artifact quality, prompt, and concurrency before claiming one is faster.

CPU: channels, locality, and workload shape

For a hypothetical dual-channel DDR5-5600 system, the transfer-rate calculation is $2 \times 5.6\text{ GT/s} \times 8\text{ bytes} = 89.6\text{ GB/s}$ before practical losses. Eight such channels give 358.4 GB/s theoretically. These are arithmetic examples, not limits shared by every EPYC or Xeon platform. Check the processor's supported channels, DIMM population, transfer rate, and NUMA placement.

More threads help until another resource dominates; they can also increase contention. Measure a thread-count sweep instead of prescribing a token rate from core count. A server with more sockets can have more aggregate bandwidth while a poorly placed process still accesses remote memory.

View larger

Build a budget for each memory pool

Estimate before downloading a large artifact, then replace estimates with measurements after loading it. For a particular device and a particular request phase:

The terms must refer to allocations that coexist. Do not sum unrelated peaks, count reserved allocator memory twice, or mix CPU RAM and GPU VRAM into one fictitious pool. For a split deployment, make separate host and device budgets. Unified-memory machines instead share a capacity constraint with the operating system.

Weight payload is a starting point

For $N$ uniformly stored weights at $b$ bits, the ideal payload is $Nb/8$ bytes. Real artifacts include grouping metadata and a mixture of tensor precisions. A dense array of zeros does not shrink just because someone calls it pruned; a sparse representation and compatible execution path are separate requirements.

Google's model card lists Gemma 4 12B Unified at 11.95B parameters.^{[5]Reference 5Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4} Its hypothetical uniform 16-bit payload would be 23.9 decimal GB; four-bit codes alone would take 5.975 GB. As checked October 10, 2026, Ollama's gemma4:12b artifact page lists an 8.0 GB native payload, with Q4_K_M model data, a BF16 projector, and a Q8_0 423M-parameter draft model.^{[6]Reference 6gemma4:12b artifact and tensor componentshttps://ollama.com/library/gemma4:12b} The draft adds another set of weights and execution/cache costs when used. Do not budget only the target model's four-bit codes. File size alone does not reveal loaded device footprint; preserve the local artifact identity because the tag can change.

Cache arithmetic starts with the architecture

For $L$ homogeneous full-attention layers with separate key/value buffers:

$H_{\text{KV}}$ is the KV-head count, $D$ the configured head width, $T$ retained tokens per sequence, $B$ concurrent sequences, and $s$ bytes per scalar. Do not automatically derive $D$ as hidden width divided by query-head count; architectures can use a different projection width. Sum layer types separately when dimensions differ. This formula excludes page rounding, quantization metadata, temporary updates, and reserved-but-unused cache space.

A hypothetical 32-layer model with eight KV heads of width 128 and two-byte scalars needs $2 \times 32 \times 8 \times 128 \times 2 = 131{,}072$ bytes per retained token, or 128 KiB. At 4,096 tokens that is 0.5 GiB (536.9 decimal MB). At 32,768 tokens across four sequences it is 16 GiB (17.18 decimal GB), before workspace.

Caching avoids recomputing the old prefix's states. Full-attention decode still reads history and its attention work grows with retained length. Re-evaluating a whole full-attention prefix would instead include quadratic attention work in that prefix. Cache projection work alone is not quadratic.

Gemma 4 12B's inspected text configuration has 40 sliding layers with eight KV heads of width 256 and a 1,024-token window, plus eight full layers with one KV head of width 512.^{[7]Reference 7Gemma 4 12B Configurationhttps://huggingface.co/google/gemma-4-12B/blob/023679ed352de9bb66cc873c9009ce3482585c08/config.json} It also enables shared K/V projection in global layers. The inspected Transformers implementation separately normalizes/transforms key and value states and updates each in the cache; inspect the cache layout to determine how storage is arranged.^{[8]Reference 8Transformers Gemma 4 Unified Attention and Cache Updatehttps://github.com/huggingface/transformers/blob/2c4914fb939fe9de0d8e7a798af4684d552f18b4/src/transformers/models/gemma4_unified/modeling_gemma4_unified.py}

For a worksheet assuming separate 16-bit K/V arrays and retention capped at 1,024 tokens in sliding layers, four 16,384-token sequences require:

• Sliding payload: $2 \times 40 \times 8 \times 256 \times 1{,}024 \times 4 \times 2 = 1.342\text{ GB}$.
• Full payload: $2 \times 8 \times 1 \times 512 \times 16{,}384 \times 4 \times 2 = 1.074\text{ GB}$.
• Combined raw payload: 2.416 GB.

If the runtime stores full-length buffers for the sliding layers, the same two-array calculation gives 22.549 GB. Keeping those buffers is not, by itself, an attention-correctness bug: the attention mask can still enforce the sliding window. A runtime may reserve full buffers by design. Verify storage behavior separately from the mathematical attention pattern.

View larger

Workspace and reserve are inputs to the experiment

Memory-efficient attention avoids materializing the full quadratic score matrix. It does not provide a universal 1.5–2.5 GB prefill workspace bound. Prefill chunks, logits, graph captures, kernel workspaces, model dimensions, and concurrency all matter.

Likewise, display and CUDA context costs vary across machines. Measure free device memory and other processes, then capture the server's peak under the intended request shape. Treat a chosen 20% reserve as a planning margin and measure OOM headroom on the target system.

A worksheet that makes its assumptions visible

This CPU exercise uses the Gemma layer dimensions above, a 7.6 GB weight proxy, an assumed 2 GB working allocation, and a 20% reserve on a hypothetical 24 GB device. It predicts no measured runtime allocations. evict=False changes storage retention; it does not change the model's attention semantics.

def positive_int(value, name):
    if type(value) is not int or value <= 0:
        raise ValueError(f"{name} must be a positive integer")
    return value

def kv_bytes(layers, heads, width, tokens, sequences, bytes_per_value=2):
    dims = (layers, heads, width, tokens, sequences, bytes_per_value)
    for name, value in zip(
        ("layers", "heads", "width", "tokens", "sequences", "bytes_per_value"), dims
    ):
        positive_int(value, name)
    return 2 * layers * heads * width * tokens * sequences * bytes_per_value

def gemma_cache_gb(tokens, sequences, evict=True):
    positive_int(tokens, "tokens")
    positive_int(sequences, "sequences")
    if type(evict) is not bool:
        raise ValueError("evict must be a boolean")
    sliding_tokens = min(tokens, 1024) if evict else tokens
    sliding = kv_bytes(40, 8, 256, sliding_tokens, sequences)
    full = kv_bytes(8, 1, 512, tokens, sequences)
    return (sliding + full) / 1e9

weight_proxy_gb = 7.6
working_gb = 2.0
capacity_gb = 24.0
reserve_fraction = 0.20
usable_gb = capacity_gb * (1 - reserve_fraction)

print(f"Assumed planning limit (80% of {capacity_gb} GB): {usable_gb:.2f} GB\n")

for tokens, sequences, evict in (
    (16384, 4, True),
    (32768, 4, True),
    (16384, 8, True),
    (16384, 4, False),
):
    cache = gemma_cache_gb(tokens, sequences, evict)
    total = weight_proxy_gb + working_gb + cache
    fits = total <= usable_gb
    print(f"Context={tokens:5d} Concurrency={sequences} Eviction={str(evict):5s} -> "
          f"KV={cache:6.3f} GB, Total={total:6.3f} GB, Fits Modeled Limit={fits}")

assert round(gemma_cache_gb(16384, 4), 3) == 2.416
assert gemma_cache_gb(16384, 8) == 2 * gemma_cache_gb(16384, 4)
assert gemma_cache_gb(32768, 4) < 2 * gemma_cache_gb(16384, 4)
try:
    gemma_cache_gb(16384, 4, evict=1)
except ValueError:
    pass
else:
    raise AssertionError("Ambiguous retention flag accepted")

Assumed planning limit (80% of 24.0 GB): 19.20 GB

Context=16384 Concurrency=4 Eviction=True  -> KV= 2.416 GB, Total=12.016 GB, Fits Modeled Limit=True
Context=32768 Concurrency=4 Eviction=True  -> KV= 3.490 GB, Total=13.090 GB, Fits Modeled Limit=True
Context=16384 Concurrency=8 Eviction=True  -> KV= 4.832 GB, Total=14.432 GB, Fits Modeled Limit=True
Context=16384 Concurrency=4 Eviction=False -> KV=22.549 GB, Total=32.149 GB, Fits Modeled Limit=False

Doubling context from 16K to 32K adds 1.074 GB under capped retention. Doubling concurrency doubles this raw cache, not the shared weights or every allocation in the process. Full-length sliding storage takes the modeled total to 32.149 GB, beyond the assumed device capacity. Whether the server rejects the configuration, preempts requests, offloads, or fails allocation depends on its implementation.

View larger

Try changing the worksheet: keep four sessions, reduce the assumed working term to 1 GB, and then remove the reserve. Does the full-storage case fit? It still exceeds 24 GB. Next, keep capped retention but double sessions: identify which terms need new measurements rather than multiplying the whole total by two.

Estimate block capacity, then verify the offload flag

llama.cpp supports CPU/GPU placement through -ngl and related controls.^{[9]Reference 9llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp} A block left on CPU executes there using its supported backend; this differs from transferring its entire weight payload to GPU each token.

For a hypothetical model with equal-size blocks, assume every non-block allocation, including the output head at its intended placement, is already covered by $M_{\text{fixed}}$. Then the block-capacity estimate is:

With 32 blocks totaling 9.6 GB and 3.2 GB fixed usage, a 16 GB modeled budget accommodates all 32 blocks. An 8 GB budget accommodates $(8-3.2)/0.3 = 16$ blocks, leaving the other 16 on CPU. Neither result includes a reserve unless one was included in the chosen budget.

Do not label that integer an exact, universally safe -ngl value. Hybrid blocks, MoE blocks, tensor overrides, cache placement, backend buffers, and precision mixtures can make block sizes unequal. Use actual tensor/suffix sizes and startup logs, then measure peak memory.

Flag semantics are versioned too. In the llama.cpp source inspected at commit 709fe755, the trailing offload range includes an output-layer slot. With 40 transformer blocks, -ngl 24 selects the last 23 blocks plus the output slot, before tensor/backend overrides. It does not mean 24 hidden blocks; input-layer assignment remains on CPU.^{[10]Reference 10llama.cpp Model Tensor Placementhttps://github.com/ggml-org/llama.cpp/blob/709fe755dfa810d77e2ac386292b29648b536864/src/llama-model.cpp} Check your installed version's help and allocation logs rather than copying counts from an older tutorial.

View larger

Allocation failure, offload, and paging are different events

A CUDA OOM message can arise through four different allocation paths. Identify the path before concluding that weights first migrated through host RAM:

NVIDIA's current guide distinguishes device-only allocation, managed memory, mapped host memory, HMM/ATS support, and limited-support platforms.^{[11]Reference 11CUDA Runtime API Memory Managementhttps://docs.nvidia.com/cuda/cuda-runtime-api/cuda_runtime_api/group__CUDART__MEMORY.html[12]Reference 12CUDA Programming Guide: Unified and System Memoryhttps://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/understanding-memory.html} Ordinary cudaMalloc allocations do not acquire automatic oversubscription merely because Unified Memory exists.

mmap maps file-backed pages into a process's virtual address space. Page residency depends on which pages the operating system keeps in memory. File-backed rereads and swap reads are separate sources; observe file I/O and swap activity to distinguish them. Ordinary CUDA weight allocations also remain device allocations as the KV cache grows, so a server may reject, preempt, or fail when VRAM runs short.

View larger

A transfer calculation you can falsify

Suppose one step must read a 7.6 GB payload. Ignore compute, attention, unpacking, and launch overhead. Use 1,008 GB/s as a peak device-memory scenario, assume 25 GB/s for a host-to-device path, and assume 5 GB/s for storage. These are serial weight-stream calculations, not a Gemma benchmark or a prediction of what your runtime migrates.

weight_gb = 7.6
vram_gbs = 1008.0
transfer_gbs = 25.0
storage_gbs = 5.0

resident_s = weight_gb / vram_gbs
mixed_s = 5.6 / vram_gbs + 2.0 / transfer_gbs
host_s = weight_gb / transfer_gbs
storage_s = weight_gb / storage_gbs

for name, seconds in (
    ("Device stream", resident_s),
    ("5.6 GB device + 2 GB transfer", mixed_s),
    ("All bytes over host link", host_s),
    ("Storage read only", storage_s),
):
    print(f"{name}: {seconds*1000:.2f} ms, {1/seconds:.2f} steps/s")
print(f"Mixed/device stream-time ratio: {mixed_s/resident_s:.2f}x")
assert mixed_s > resident_s

Device stream: 7.54 ms, 132.63 steps/s
5.6 GB device + 2 GB transfer: 85.56 ms, 11.69 steps/s
All bytes over host link: 304.00 ms, 3.29 steps/s
Storage read only: 1520.00 ms, 0.66 steps/s
Mixed/device stream-time ratio: 11.35x

The 11.35× ratio compares two modeled stream times with the same assumptions. It cannot be compared directly with an unrelated measured 50 tokens/s to claim a fivefold slowdown. CPU-executed layers have CPU compute and activation-boundary costs; overlapped transfer has another timing model; disk reads also need subsequent processing. Profile the path your server actually takes.

View larger

Choose the runner by the workload it must pass

Ruling a migration path in or out still doesn't choose the program that places the model. The assistant needs a runner that can serve several long logs and keep incident data inside the approved network.

GGUF is a tensor/metadata container, not “GPT-Generated Unified Format.” It can be sharded, and mapping it does not make startup instantaneous.^{[13]Reference 13GGUF Specificationhttps://github.com/ggml-org/ggml/blob/master/docs/gguf.md} llama.cpp can run through native binaries without PyTorch/Python as the inference runtime, but its chosen backends still have dependencies. In the inspected CLI source, cache types include f16, bf16, q8_0, and q4_0; generic FP8 is not in that accepted list. Quantized cache carries metadata, has backend restrictions, and needs task evaluation.^{[14]Reference 14llama.cpp CLI Cache and GPU-Layer Argumentshttps://github.com/ggml-org/llama.cpp/blob/709fe755dfa810d77e2ac386292b29648b536864/common/arg.cpp}

A Modelfile is configuration, not a weight serialization format. Ollama's import documentation covers supported Safetensors and GGUF inputs.^{[15]Reference 15Importing a Model - Ollamahttps://docs.ollama.com/import[16]Reference 16Ollama GitHub Repositoryhttps://github.com/ollama/ollama} Its API compatibility covers a subset of OpenAI endpoints/features. A basic client can target http://localhost:11434/v1, but model names, supported fields, context settings, and conversation behavior still need checking. The placeholder API key used by some clients does not secure the unauthenticated local server.^{[17]Reference 17OpenAI compatibility - Ollamahttps://docs.ollama.com/api/openai-compatibility[18]Reference 18Authenticationhttps://docs.ollama.com/api/authentication}

MLX is the array framework; MLX-LM supplies language-model tools. Its supported CPU/GPU operations share unified arrays, with dependencies scheduled between streams. There is no universal “MLX beats every Metal runner” result.^{[3]Reference 3MLX: An array framework for Apple siliconhttps://github.com/ml-explore/mlx[4]Reference 4MLX Unified Memoryhttps://ml-explore.github.io/mlx/build/html/usage/unified_memory.html}

vLLM's PagedAttention maps logical token blocks to cache blocks. These are not ordinary OS virtual-memory pages. Waste depends on block size, sequence lengths, sharing, and pool policy; “under 4%” is not a bound for every workload. It offers continuous batching and distributed serving, while Ollama and llama-server also support concurrent requests. Ollama documents parallel-request and queue controls, with memory implications.^{[19]Reference 19vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionhttps://github.com/vllm-project/vllm[20]Reference 20FAQ - Ollamahttps://docs.ollama.com/faq}

Draw the private-data boundary

Local inference, on-premises hosting, and air-gapping are different properties. A localhost API can still forward a request to a cloud model. An on-premises server can have Internet access. An air gap requires an actual separation from external networks, with controlled artifact/update transfer.

Authorized documents -> local retrieval -> approved local model -> reviewed draft
                                           |
                                           +-> approved local fallback on failure

Credential store: outside model/retrieval input
External model/search/tools: denied by policy and verified network controls
Logs, backups, embeddings, and monitoring: included in the same data-flow review

View larger

Legal compliance depends on the data, agreements, safeguards, recipients, and access arrangements. HIPAA permits cloud processing under applicable agreements and requirements, while GDPR international-transfer rules provide mechanisms including adequacy and appropriate safeguards.^{[21]Reference 21Guidance on HIPAA and Cloud Computinghttps://www.hhs.gov/hipaa/for-professionals/special-topics/health-information-technology/cloud-computing/index.html[22]Reference 22International Transfers and International Cooperationhttps://www.edpb.europa.eu/topics/international-transfers-and-international-cooperation_en} Assess export controls and contractual restrictions for the specific case; commercial cloud use can be permitted under the applicable rules.

For the runbook assistant's stated internal-only policy:

1. Disable Ollama cloud features. OLLAMA_NO_CLOUD=1 disables Ollama's cloud-model and web-search features after restart. Configure network, registry, and telemetry controls for those separate policies.^{[20]Reference 20FAQ - Ollamahttps://docs.ollama.com/faq}
2. Control inbound access. Ollama defaults to loopback and its local API has no built-in authentication requirement. Loopback limits direct network access, but does not authenticate local processes or protect an exposed tunnel/proxy. Multi-user access needs an authenticating gateway and caller-scoped retrieval authorization.^{[18]Reference 18Authenticationhttps://docs.ollama.com/api/authentication[20]Reference 20FAQ - Ollamahttps://docs.ollama.com/faq}
3. Verify the entire outbound path. Apply and test appropriate network policy for the server, retrieval, tools, proxies, logs, and monitoring. Generated curl text is not execution; a tool runner creates that boundary. Egress restrictions reduce permitted destinations but do not stop every leak through allowed peers, local files, or logs.
4. Exercise failure behavior. Stop the model worker in a controlled test and submit an unavailable-runbook case. Require an explicit local fallback and no external forwarding. A model's self-reported confidence is not sufficient evidence of correctness.

Inspect placement and measure what users receive

Start with a downloaded, approved artifact and a running local server. The commands below inspect the installation; their output depends on your machine.

set -euo pipefail
ollama --version
curl --fail --silent --show-error --max-time 10 \
  http://127.0.0.1:11434/api/tags > local-models.json
curl --fail --silent --show-error --max-time 10 \
  http://127.0.0.1:11434/api/ps > loaded-models.json
ollama ps

/api/tags is inventory. /api/ps lists loaded models and reported size, VRAM size, digest, and context length.^{[23]Reference 23Ollama API: List Running Modelshttps://docs.ollama.com/api/ps} The CLI's PROCESSOR column summarizes CPU/GPU placement. Treat 100% GPU as a placement summary; check Tensor Core use, peak memory fit, and latency separately. Percentages are not an exact hidden-layer count.^{[20]Reference 20FAQ - Ollamahttps://docs.ollama.com/faq}

Separate the server's timings from client latency

This native Ollama request fixes the context and output caps. Use a locally approved model; the registry tag here is a reproducible request example, not an immutable production identity.^{[24]Reference 24Generate a Responsehttps://docs.ollama.com/api/generate}

curl --fail --silent --show-error --max-time 300 \
  --write-out 'client_total_seconds=%{time_total}\n' \
  http://127.0.0.1:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gemma4:12b",
    "prompt": "Explain why model file size alone cannot predict peak inference memory.",
    "stream": false,
    "options": {"num_ctx": 4096, "num_predict": 128, "temperature": 0}
  }' \
  -o local-response.json
ollama ps

Ollama reports timing fields in nanoseconds. Current usage documentation distinguishes total prompt tokens, cached prompt tokens, uncached-prompt evaluation duration, and output generation count/duration.^{[25]Reference 25Usagehttps://docs.ollama.com/api/usage} Dividing all prompt tokens by the uncached duration can overstate prefill performance. Reported generation counts should not be assumed to count only visible final-answer text; record thinking mode too. Earlier server versions can differ, so preserve the runtime version and actual receipt.

The helper below uses synthetic receipts, not a model run. Save it as read-generation-timing.py; passing your JSON filename prints its rates. Missing metrics and zero durations produce an unavailable rate, rather than an invented one-nanosecond duration or division by zero.

import json
import math
import sys

def metric(response, key):
    value = response.get(key)
    if value is not None and (type(value) is not int or value < 0):
        raise ValueError(f"{key} must be a nonnegative integer")
    return value

def rate(count, duration_ns):
    if count is None or duration_ns is None or count == 0 or duration_ns == 0:
        return None
    try:
        result = count * 1_000_000_000 / duration_ns
    except OverflowError as exc:
        raise ValueError("Token rate exceeds finite FP64 range") from exc
    if not math.isfinite(result):
        raise ValueError("Token rate exceeds finite FP64 range")
    return result

def generation_rates(response):
    if response.get("done") is not True:
        raise ValueError("A completed response is required")
    prompt = metric(response, "prompt_eval_count")
    cached = metric(response, "prompt_eval_cached_count")
    prompt_ns = metric(response, "prompt_eval_duration")
    output = metric(response, "eval_count")
    output_ns = metric(response, "eval_duration")
    if prompt is not None and cached is not None and cached > prompt:
        raise ValueError("Cached prompt count exceeds prompt count")
    uncached = prompt - cached if prompt is not None and cached is not None else None
    return {
        "uncached_prompt_tokens_per_second": rate(uncached, prompt_ns),
        "output_tokens_per_second": rate(output, output_ns),
    }

if len(sys.argv) == 2:
    with open(sys.argv[1], encoding="utf-8") as f:
        print(json.dumps(generation_rates(json.load(f)), indent=2))
else:
    example = {
        "done": True,
        "prompt_eval_count": 1000,
        "prompt_eval_cached_count": 600,
        "prompt_eval_duration": 200_000_000,
        "eval_count": 80,
        "eval_duration": 2_000_000_000,
    }
    result = generation_rates(example)
    print(f"Uncached prompt: {result['uncached_prompt_tokens_per_second']:.1f} tok/s")
    print(f"Output: {result['output_tokens_per_second']:.1f} tok/s")
    print("Zero duration:", generation_rates({"done": True, "eval_count": 80, "eval_duration": 0})
          ["output_tokens_per_second"])
    print("Missing duration:", generation_rates({"done": True, "eval_count": 80})
          ["output_tokens_per_second"])
    assert result["uncached_prompt_tokens_per_second"] == 2000.0
    assert rate(10 ** 400, 10 ** 400) == 1_000_000_000.0
    for key, value in (
        ("eval_duration", True),
        ("prompt_eval_cached_count", 1001),
        ("eval_count", 10 ** 400),
    ):
        invalid = dict(example, **{key: value})
        try:
            generation_rates(invalid)
        except ValueError:
            pass
        else:
            raise AssertionError("Invalid timing receipt accepted")

Uncached prompt: 2000.0 tok/s
Output: 40.0 tok/s
Zero duration: None
Missing duration: None

Run python3 read-generation-timing.py local-response.json after the request. JSON null means the rate cannot be computed under this contract. A missing cached-count field does not justify guessing a cache hit count. Malformed counts, inconsistent cache counts, and rates outside finite FP64 range are rejected; integer nanosecond conversion also avoids overflowing an intermediate float when the final ratio is representable.

Those rates describe aggregate generation throughput. Measure time to first token (TTFT) and individual token gaps with streaming and client arrival timestamps, stating whether the first thinking/content token or first chunk defines TTFT. Transport buffering can group tokens. Also measure client completion time, queueing, cold versus warm loads, cache reuse, peak memory, and failures under the intended arrival load; concurrency requires representative concurrent workloads.

View larger

Keep the case identities when comparing quantization

A model that fits may still corrupt a rollback flag. Run baseline and candidate on the same prompts, retrieved context, templates, and declared decoding policy. Keep per-case outputs and task-specific checks alongside the aggregate score. Repeated trials help distinguish a systematic change from sampling variation; fixed seeds can still produce implementation-dependent results.

These are invented pass/fail outcomes showing the accounting problem. Evaluate Gemma's quantization with model runs and task-specific checks.

baseline = {
    "syntax_rollback_cmd": True,
    "identify_missing_node_id": True,
    "parse_alert_severity": True,
    "abstain_on_unknown_runbook": False,
}
quantized = {
    "syntax_rollback_cmd": False,  # Hypothetical critical flag regression
    "identify_missing_node_id": True,
    "parse_alert_severity": True,
    "abstain_on_unknown_runbook": True,  # Hypothetical abstention improvement
}
if baseline.keys() != quantized.keys():
    raise ValueError("Evaluation test fixtures do not match")
regressed = sorted(k for k in baseline if baseline[k] and not quantized[k])
improved = sorted(k for k in baseline if not baseline[k] and quantized[k])
print(f"Baseline pass rate:  {sum(baseline.values())}/{len(baseline)} (75%)")
print(f"Quantized pass rate: {sum(quantized.values())}/{len(quantized)} (75%)")
print(f"Regressed test cases: {regressed}")
print(f"Improved test cases:  {improved}")
assert regressed == ["syntax_rollback_cmd"]
assert improved == ["abstain_on_unknown_runbook"]

Baseline pass rate:  3/4 (75%)
Quantized pass rate: 3/4 (75%)
Regressed test cases: ['syntax_rollback_cmd']
Improved test cases:  ['abstain_on_unknown_runbook']

Both dictionaries total 75%, but the cases differ. Under a release policy that rejects critical command regressions, the candidate fails. The baseline's failed abstention case also needs repair; comparison does not certify it as safe. The candidate's abstention improvement is useful evidence, while the critical command regression still blocks release.

View larger

Diagnose with competing explanations

A restart does not itself re-pull a Docker image or an Ollama model. If a deployment step does pull mutable latest or model tags, it can install changed content. Use immutable container image digests and a verified local model artifact/manifest, with tokenizer, template, quantization, runtime, and options recorded. Do not assume Ollama accepts Docker-style model@sha256:... names.

Your deployment receipt: write down artifact identities, hardware/driver/runtime versions, placement, context and concurrency limits, cache/workspace policy, peak memory, the latency workload, per-case quality results, and the tested local fallback. An image hash verifies artifact identity; evaluate task quality and enforce the private-data boundary with separate checks.

Separate the download size from the bytes that have to coexist. Then name the failure: an allocation, offload, paging, or a quality regression that an average score hides. A smaller model does not retire that worksheet. The same budget and quality contract still have to hold when the device also has power, thermal, and offline limits.