SLM Specialization & Edge Deployment

Picture a technician taking an inspection handheld into a utility vault with no connectivity. The product specification forbids sending facility manuals or incident text off the device. For this lesson's hypothetical terminal, the proposed targets are time to first visible answer under 250 ms, at least 18 generated tokens/s after ten minutes of use, and enough battery for an eight-hour shift. These targets guide design and testing; measure them on the intended hardware and workload. Actual field procedures need qualified review; fluent generated instructions alone are insufficient evidence of safety.

A lookup table can already answer "which torque specification applies to flange B-12?" A lightweight classifier can route an incident report to the right department. Generating a cited, multi-sentence repair procedure is an entirely different feature with a much stricter error budget. Checkpoint fit is one hardware constraint; validate latency, battery life, data handling, and safety against the product requirements before release.

Build on workstation memory sizing, then add battery and cooling constraints, application isolation, and a product-wide data-flow policy. Offline operation means connectivity is unavailable. An air gap is an isolation boundary. A firewall rule or a local inference call alone does not establish one.

A small language model (SLM) is small relative to its comparison set. There is no universal parameter cutoff. This lesson includes sub-billion and several-billion-parameter models. Specialization adapts behavior to a defined job through data, training, retrieval, or constraints. It does not necessarily remove parameters or erase unrelated knowledge. Edge deployment concerns execution near the user or data source, including mobile devices and larger local appliances.

Those limits sit on local memory and hardware sizing and knowledge distillation. Keep the weights, the battery, and the manuals on the handheld, then see which trained behavior still meets the vault job.

View larger

Start from the physical constraint, not the model size

Broad capability can be useful, but the terminal's job is narrower: extract error codes, locate approved manual passages, and explain the retrieved evidence. Start by measuring those behaviors. A specialist can outperform a larger baseline on a particular evaluation; transfer to other tasks depends on training and workload. Specialization is not equivalent to pruning, and local execution removes an inference round-trip only when the rest of the pipeline is local too.

Before picking weights, define each offline job and quantify its failure cost. Ask whether a deterministic lookup, a classical classifier, an extractive retrieval pipeline, or a generative SLM is required. "What model fits in 4 GB of RAM?" supplies one capacity constraint; it is not a complete engineering specification.

For an industrial inspection handheld, we define candidate operational bars across four distinct device tasks:

Notice how no single row mandates a parameter count. An image classifier's total inference latency differs completely from a generative model's first-token latency. Time to first token (TTFT) must also measure the complete pipeline: application dispatch, local retrieval, prompt templating, tokenization, and prefill compute until the first visible answer token arrives. If a model emits hidden reasoning tokens first, track both first-generated-token latency and first-visible-token latency.

Here is a filter over invented scorecard rows. A separate review supplies the privacy_local decision; this filter uses that prior input to screen candidates:

jobs = [
    {
        "name": "intent",
        "privacy_local": True,
        "p95_ms": 42,
        "max_p95_ms": 60,
        "quality": 0.98,
        "min_quality": 0.97,
    },
    {
        "name": "policy_qa",
        "privacy_local": True,
        "p95_ms": 228,
        "max_p95_ms": 250,
        "quality": 0.93,
        "min_quality": 0.92,
    },
    {
        "name": "photo_support",
        "privacy_local": True,
        "p95_ms": 470,
        "max_p95_ms": 400,
        "quality": 0.91,
        "min_quality": 0.92,
    },
]

approved = [
    job["name"]
    for job in jobs
    if job["privacy_local"] is True
    and job["p95_ms"] < job["max_p95_ms"]
    and job["quality"] >= job["min_quality"]
]
print(f"routes passing these fixture checks: {approved}")

routes passing these fixture checks: ['intent', 'policy_qa']

Photo support misses both its latency ceiling and quality threshold. Intent routing and policy Q&A pass these preliminary checks, yet hardware qualification remains to be done.

These approved rows also separate distinct model workloads. Intent routing only selects an execution lane. Policy Q&A must retrieve rules, ground its response in exact citations, and abstain when documentation is missing.

A bandwidth worksheet, with its assumptions visible

For a dense model whose matrix weights exceed cache capacity, batch-1 decode often spends much of its time moving weights. Compute, attention, small kernel launches, format conversion, and runtime scheduling can also limit it. Embedding lookups, inactive experts, cached weights, and immediate reuse invalidate an assumption that every stored parameter is read from DRAM on every token.

Let $P$ count matrix weights used in this step. Assuming one complete packed-weight read, two floating-point operations per multiply-add, and no other traffic gives:

This model gives $1$ FLOP/byte for $16$-bit weights and $4$ for four-bit weights. At an assumed $50$ GB/s, the latter corresponds to $50\text{ GB/s} \times 4\text{ FLOP/byte} = 200\text{ GFLOP/s}$ of modeled matrix work. Integer TOPS, sparse operation counts, and floating-point FLOPs are different metrics; dividing this work rate by an NPU's advertised TOPS does not establish utilization or idle time. Quantization scales and unpacking also cost bytes and work.

We calculate the theoretical decoding ceiling using the weight-streaming bound:

Assume a dense model step reads 3.8 billion active matrix weights once over a hypothetical 50 GB/s bus. The raw payload is 1.9 GB at four bits or 3.8 GB at eight bits. This excludes quantization metadata, mixed-precision tensors, KV/activation traffic, and invocation overhead; it is not an artifact-size prediction for every 3.8B model:

def weight_gb(params_billion: float, bits: int) -> float:
    return params_billion * bits / 8

def weight_streaming_ceiling(bandwidth_gbs: float, active_weight_gb: float) -> float:
    return bandwidth_gbs / active_weight_gb

params_b = 3.8
bandwidth_gbs = 50.0  # illustrative handheld budget, not a device datasheet
floor_tps = 18.0

for bits in (8, 4):
    artifact_gb = weight_gb(params_b, bits)
    ceiling = weight_streaming_ceiling(bandwidth_gbs, artifact_gb)
    print(
        f"{bits}-bit: {artifact_gb:.1f} GB weights, "
        f"ceiling {ceiling:.1f} t/s, "
        f"not ruled out by this bound: {ceiling >= floor_tps}"
    )

8-bit: 3.8 GB weights, ceiling 13.2 t/s, not ruled out by this bound: False
4-bit: 1.9 GB weights, ceiling 26.3 t/s, not ruled out by this bound: True

Under these assumptions, the eight-bit candidate misses the $18$ steps/s target: $50/3.8 \approx 13.2$ steps/s. The four-bit estimate clears this throughput gate at $26.3$ steps/s, pending a working kernel and device measurement. Changing the model's active weights, cache reuse, or decoding algorithm changes the worksheet; measure token rates on the target device.

View larger

Total memory footprint requires its own accounting:

Flash storage, system RAM, and accelerator-accessible allocations are different budgets. A download can fit while serving allocations fail or the OS terminates the app under memory pressure. Include tokenizer and retrieval storage, multimodal encoders, runtime pools, and other applications. An individual allocation limit and a system low-memory termination are different failure mechanisms.

MobileLLM introduced dedicated architectural optimizations for sub-billion models deployed to mobile chips.^{[1]Reference 1MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Caseshttps://arxiv.org/abs/2402.14905} Use its empirical findings to select model topologies, but always measure actual energy, memory residency, and throughput on your specific target SKU.

Distillation: transferring dark knowledge to compact silicon

A bandwidth ceiling can rule a checkpoint out. It cannot put the cited repair procedure into a smaller model. Knowledge distillation trains a student against a teacher's predictions or other selected signals. The teacher need not be a frontier language model. The student can use a different architecture, and deployment need not include the teacher's weights. Whether this improves the device task depends on the training setup and evaluation.^{[2]Reference 2Distilling the Knowledge in a Neural Network.https://arxiv.org/abs/1503.02531}

In classification, hard labels tell the student which class won, but hide how the teacher ranked the alternatives. In language modeling, teacher logits provide a dense probability distribution over the entire vocabulary. This distribution contains "dark knowledge": the subtle semantic relationships between runner-up tokens that were plausible versus those that were absurd.

The distillation objective

When distilling token by token, we combine standard cross-entropy on ground-truth tokens with Kullback-Leibler (KL) divergence over temperature-scaled teacher and student distributions:

Here $z_T$ and $z_S$ are aligned teacher and student logits, positive $\tau$ is the temperature, $\sigma$ is softmax, and $0\leq\alpha\leq1$ balances supervision. The hard-label CE uses unscaled student logits; the soft term uses the same temperature for teacher and student.

Temperature changes how strongly logit differences affect probabilities. For a peaked distribution, increasing positive $\tau$ gives alternatives more mass without changing their rank:

Notice the ratio between two candidate tokens $B$ and $C$:

For a fixed teacher, the exact gradient of the unscaled forward KL with respect to a student logit is:^{[2]Reference 2Distilling the Knowledge in a Neural Network.https://arxiv.org/abs/1503.02531}

There are two temperature effects: the chain rule contributes $1/\tau$, and at high temperature the probability difference is approximately proportional to $1/\tau$. Center each logit vector by subtracting its mean. For a vocabulary of size $V$:

Multiplying by $\tau^2$ approximately compensates for this high-temperature shrinkage. Gradient magnitudes can still vary across temperatures, and successful optimization depends on the temperature and loss weights selected during validation.

Check the exact derivative against a finite difference before relying on the approximation:

import math

teacher = [4.0, 2.0, 1.0]
student = [3.2, 2.4, 1.1]

def probabilities(logits, tau):
    peak = max(logits)
    values = [math.exp((v - peak) / tau) for v in logits]
    return [v / sum(values) for v in values]

def loss(logits, tau):
    p, q = probabilities(teacher, tau), probabilities(logits, tau)
    return sum(a * math.log(a / b) for a, b in zip(p, q))

for tau in (1.0, 3.0, 100.0):
    p, q = probabilities(teacher, tau), probabilities(student, tau)
    exact = (q[0] - p[0]) / tau
    step = 1e-4
    plus, minus = student.copy(), student.copy()
    plus[0] += step
    minus[0] -= step
    numeric = (loss(plus, tau) - loss(minus, tau)) / (2 * step)
    assert math.isclose(numeric, exact, rel_tol=1e-6, abs_tol=1e-10)
    print(f"tau={tau:.0f}: gradient={exact:.6f}, scaled={tau*tau*exact:.6f}")

tau=1: gradient=-0.207576, scaled=-0.207576
tau=3: gradient=-0.029854, scaled=-0.268686
tau=100: gradient=-0.000024, scaled=-0.235045

The scaled values remain of comparable order, but are not equal. For these centered logits, the high-temperature limit is about -0.233333 for the first coordinate. The example checks one derivative; a trained model's behavior requires an evaluation on its outputs.

View larger

The derivative used centered logits. A three-way teacher target is the case that matters for a token. Start from logits [4.0, 2.0, 1.0] and see which alternatives keep probability when the temperature rises:

import math

def softmax(values, temperature):
    shifted = [value / temperature for value in values]
    peak = max(shifted)
    weights = [math.exp(value - peak) for value in shifted]
    total = sum(weights)
    return [weight / total for weight in weights]

teacher_logits = [4.0, 2.0, 1.0]
for temperature in (1.0, 3.0):
    probs = softmax(teacher_logits, temperature)
    print(f"temperature={temperature:.0f}: {[round(p, 3) for p in probs]}")

temperature=1: [0.844, 0.114, 0.042]
temperature=3: [0.532, 0.273, 0.196]

View larger

At $\tau=1$, A receives 84.4% and C 4.2%. At $\tau=3$, A remains preferred, while B and C receive more mass. A one-hot label discards their relative preferences. If the teacher is wrong, matching those preferences can transfer that error too.

Distillation with execution-based verifiers

Direct token-level KL needs distributions over aligned token events at aligned prefixes. Matching token IDs alone is insufficient if their meanings differ. Different tokenizers need an alignment method, or sequence-level supervision using teacher text retokenized by the student. Closed APIs may expose only sampled text or a limited set of log probabilities; full-vocabulary distillation requires scores for every token at each aligned prefix.

Phi-3 combined filtered public data with synthetic examples.^{[3]Reference 3Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonehttps://arxiv.org/abs/2404.14219} For domain specialization, include approved documents, human examples, unanswerable questions, and verified synthetic cases. Split evaluation by document, equipment family, or site before generating variants so near-duplicates cannot leak across the split.

Check each synthetic example against the contract it is supposed to teach:

1. Code and tool-use verification: Run bounded tests in an environment with appropriate isolation, resource limits, and no secrets. A container alone is not a sufficient boundary for hostile code. A passing test covers its assertions, not every behavior.
2. Schema validation: Check required fields, types, and allowed values. Well-formed JSON can still request an unauthorized action or contain a wrong fact.
3. Retrieval grounding: Check source identity, version, applicable equipment and conditions, and whether the passage supports the claim. An existing paragraph number or matching numeric substring alone is insufficient.

Here is an invented maintenance fixture, not a real torque recommendation. Both outputs pass a naive citation-and-number check:

manual = {
    "M-7": {"equipment": "pump-A", "torque_nm": 12},
    "M-8": {"equipment": "pump-B", "torque_nm": 30},
}
question_equipment = "pump-A"
answers = [
    {"citation": "M-7", "torque_nm": 12},
    {"citation": "M-8", "torque_nm": 30},
]
for answer in answers:
    source = manual[answer["citation"]]
    naive_pass = source["torque_nm"] == answer["torque_nm"]
    applicable_pass = naive_pass and source["equipment"] == question_equipment
    print(f'{answer["citation"]}: naive={naive_pass}, applicable={applicable_pass}')

M-7: naive=True, applicable=True
M-8: naive=True, applicable=False

The second answer quotes a real value for the wrong equipment. The check above covers only this tiny structured fixture. Review ambiguous cases and measure held-out correctness and abstention; curation reduces selected error modes while those evaluations test the resulting student's reliability.

View larger

MiniLLM uses on-policy optimization of reverse KL, $\text{KL}(p_S \parallel p_T)$, with variance reduction, teacher-mixed sampling, and length normalization.^{[4]Reference 4MiniLLM: Knowledge Distillation of Large Language Modelshttps://arxiv.org/abs/2306.08543} Reverse KL tends to favor teacher modes over broad coverage of alternatives. This objective shapes how the student matches the teacher; factual correctness still requires task evaluation.

This CPU calculation combines three invented signals. In real training, freeze teacher gradients, align token prefixes, mask padding, and define reductions explicitly. Hidden-state matching also needs chosen layers and compatible dimensions, often a learned projection; unrelated hidden coordinates are not interchangeable:

import math

def log_softmax(logits, temperature=1.0):
    if not logits or not math.isfinite(temperature) or temperature <= 0:
        raise ValueError("nonempty logits and positive finite temperature required")
    if not all(math.isfinite(value) for value in logits):
        raise ValueError("finite logits required")
    shifted = [value / temperature for value in logits]
    if not all(math.isfinite(value) for value in shifted):
        raise ValueError("temperature-scaled logits must remain finite")
    peak = max(shifted)
    centered = [value - peak for value in shifted]
    if not all(math.isfinite(value) for value in centered):
        raise ValueError("centered logits must remain finite")
    normalizer = math.log(sum(math.exp(value) for value in centered))
    return [value - normalizer for value in centered]

def cross_entropy(logits, label):
    return -log_softmax(logits)[label]

def kl_divergence(target_logs, candidate_logs):
    if len(target_logs) != len(candidate_logs):
        raise ValueError("aligned vocabulary sizes required")
    return sum(
        math.exp(p_log) * (p_log - q_log)
        for p_log, q_log in zip(target_logs, candidate_logs)
    )

teacher_logits = [4.0, 2.0, 1.0]
student_logits = [3.2, 2.4, 1.1]
label = 0
temperature = 2.0
assert cross_entropy([0.0, -1000.0], 1) == 1000.0
assert kl_divergence(log_softmax([0.0, -1000.0]),
                     log_softmax([-1000.0, 0.0])) == 1000.0

hard_loss = cross_entropy(student_logits, label)
soft_loss = kl_divergence(
    log_softmax(teacher_logits, temperature),
    log_softmax(student_logits, temperature),
) * (temperature * temperature)

teacher_hidden = [0.4, -0.1, 0.2]
student_hidden = [0.3, 0.0, 0.1]
assert len(teacher_hidden) == len(student_hidden) > 0
hidden_loss = sum(
    (s - t) ** 2 for s, t in zip(student_hidden, teacher_hidden)
) / len(teacher_hidden)

alpha, beta = 0.8, 0.2
total_loss = (1 - alpha) * hard_loss + alpha * soft_loss + beta * hidden_loss
print(f"hard_loss={hard_loss:.4f}")
print(f"soft_loss={soft_loss:.4f}")
print(f"hidden_loss={hidden_loss:.4f}")
print(f"total_loss={total_loss:.4f}")

hard_loss=0.4522
soft_loss=0.1480
hidden_loss=0.0100
total_loss=0.2109

Once the student finishes training, evaluate its exported and quantized artifact again. Sensitivity depends on the model, format, calibration data, kernels, and workload. There is no universal sub-2B cutoff requiring one quantization method.

These synthetic rows illustrate a quality-versus-latency decision. Use measured INT8/INT4 results with a defined task metric and uncertainty before applying this filter to a release:

baseline = {"quality": 0.94, "p95_ms": 132}
exports = [
    {"name": "int8", "quality": 0.938, "p95_ms": 104},
    {"name": "int4", "quality": 0.887, "p95_ms": 71},
]
max_quality_drop = 0.02
latency_gate_ms = 120

approved = [
    item["name"]
    for item in exports
    if baseline["quality"] - item["quality"] <= max_quality_drop
    and item["p95_ms"] <= latency_gate_ms
]
print(f"approved exports: {approved}")

approved exports: ['int8']

In this fixture, INT4 loses 0.053 quality and fails the 0.02 tolerance; INT8 loses 0.002 and passes both listed thresholds. Neither row establishes hardware qualification. Compare paired held-out outcomes, source support, abstention, latency, and memory under the actual decode settings; perplexity alone is insufficient.

View larger

Architecture choices that matter at small scale

The distillation loss changes the training signal. It does not choose how many KV heads the student carries onto the device.

Efficient execution depends on layer structure and attention heads, which change the work, memory traffic, and cache behavior as parameter count changes.

Meta's MobileLLM research revealed key architectural principles for sub-billion models:^{[1]Reference 1MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Caseshttps://arxiv.org/abs/2402.14905}

1. Depth versus width: The paper's approximately 125M/350M experiments favored deeper, thinner designs. Scaling laws do not require equal parameter allocation to depth and width; test the balance for the target training recipe and device.
2. Embedding tying: Sharing input and output weights reduces storage. One ablation reduced 135M to 119M parameters, with a small accuracy drop; reallocating some savings to depth recovered it.
3. Immediate block reuse: MobileLLM-LS repeats each block with shared weights. Cache locality can reduce repeat weight traffic, while compute and attention state still cost resources. Measure cache residency and DRAM traffic. Its iPhone 13 FP16/MPS experiment measured 15.6 versus 16.0 ms execution, averaged over 50 iterations; measure sustained token rate on the target terminal.
4. GQA: Fewer KV heads reduce cache payload for otherwise equal dimensions. Choose a trained architecture; replacing an existing model's attention heads is not an inference-time switch.

The same paper found its tested distillation from Llama-v2 7B slower to train and comparable or inferior in accuracy to hard-label training. Architectural improvement and successful distillation are separate hypotheses.^{[1]Reference 1MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Caseshttps://arxiv.org/abs/2402.14905}

KV cache growth can exhaust RAM on a small device. For 24 layers, head dimension 64, and a 4,096-token window, how many mebibytes do 4 KV heads save against 16?

def kv_mib(layers, kv_heads, head_dim, tokens, bytes_per_value=2):
    return 2 * layers * kv_heads * head_dim * tokens * bytes_per_value / 2**20

mha = kv_mib(layers=24, kv_heads=16, head_dim=64, tokens=4096)
gqa = kv_mib(layers=24, kv_heads=4, head_dim=64, tokens=4096)
print(f"MHA KV cache: {mha:.1f} MiB")
print(f"GQA KV cache: {gqa:.1f} MiB")
print(f"cache reduction for this architecture: {mha / gqa:.1f}x")

MHA KV cache: 384.0 MiB
GQA KV cache: 96.0 MiB
cache reduction for this architecture: 4.0x

The hypothetical homogeneous cache shrinks from $384$ to $96$ MiB, a $4\times$ reduction. This excludes alignment, pools, quantization metadata, and temporary state. It does not establish total memory fit or fourfold speed. Hybrid attention and recurrent layers require their own per-layer formulas.

View larger

These documented model examples illustrate different tradeoffs as checked September 22, 2026. Refresh model availability and compare candidates on the target task before selecting one:

• Phi-4-mini-instruct (3.8B): Employs GQA, shared input-output embeddings, a 128K context window, and high-density synthetic data tuning for tool calling.^{[5]Reference 5Phi-4-mini-instruct Model Cardhttps://huggingface.co/microsoft/Phi-4-mini-instruct}
• Gemma 4 E2B and E4B: The card lists 2.3B/4.5B effective parameters and 5.1B/8B including embeddings. Per-layer embeddings use token lookups. Effective is not the same label as an MoE active-expert count. Include the selected artifact's stored tensors, encoder components, cache, and measured serving residency; the effective count cannot establish fit.^{[6]Reference 6Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4}
• Qwen3-1.7B: Its card documents 28 layers, 16 query/8 KV heads, a 32,768-token context, and thinking/non-thinking chat-template modes.^{[7]Reference 7Qwen3-1.7B Model Cardhttps://huggingface.co/Qwen/Qwen3-1.7B} Thinking tokens consume generation work before the visible answer. Budget and evaluate both modes under the installed runtime; an output cap can truncate the answer rather than improve it.

On-device runtimes: mapping graphs to silicon

A thinner stack can fit the worksheet and still fail as a graph the handheld cannot run. A runtime may use supported accelerator kernels, native CPU kernels, or both. CPU execution is not necessarily emulation.

Mobile NPUs and the danger of CPU fallback

An NPU may improve energy efficiency for supported work. Advertised TOPS/W figures across formats, sparsity modes, and chips do not establish energy per generated token for your application.

Check the chosen accelerator's restrictions:

1. Shape support: Shapes, ranges, and cache updates may impose restrictions. Buckets can help some backends, but padding can waste work and change state if masking is wrong.
2. Numerical support: Weight storage, multiply format, and accumulator precision are separate choices. Check this backend's exact support rather than assuming all NPUs use INT4/INT8 with FP16 accumulation.
3. Operator support: An unsupported operation can remain on CPU, cause export failure, or require a different lowering. Check placement and partition boundaries, not only a percentage of delegated operators.

Partial delegation can add synchronization, layout conversion, intermediate materialization, or copies. Unified physical memory does not remove all those costs, and it does not imply a mandatory copy between two private memories either. Some partitions are beneficial. Profile their frequency and critical-path time, then compare a supported fused graph and an optimized CPU path.^{[8]Reference 8ExecuTorch Backendshttps://docs.pytorch.org/executorch/stable/backends-overview.html[11]Reference 11Deploy ONNX Runtime Mobilehttps://onnxruntime.ai/docs/tutorials/mobile/}

View larger

Apple's preparation guide explicitly distinguishes compression methods and device strengths, including GPU-oriented four-bit weight quantization and stateful KV-cache updates. After Core ML conversion, inspect graph partitioning and confirm which operations run on the Neural Engine.^{[9]Reference 9Bring Your Machine Learning and AI Models to Apple Siliconhttps://developer.apple.com/videos/play/wwdc2024/10159/}

Testing graph export on CPU

First check capture, serialization, and declared shape constraints. torch.export records a graph that downstream tools can lower; it is not an ExecuTorch executable or an NPU compiler.^{[14]Reference 14torch.export API Referencehttps://docs.pytorch.org/docs/stable/user_guide/torch_compiler/export/api_reference.html} This CPU round-trip uses pinned PyTorch 2.14.0, the version checked for this lesson:

from pathlib import Path
from tempfile import TemporaryDirectory
import torch
from torch import nn
torch.set_num_threads(1)

class NumericRouter(nn.Module):
    def __init__(self):
        super().__init__()
        self.projection = nn.Linear(3, 2)
        with torch.no_grad():
            self.projection.weight.copy_(torch.tensor([
                [-1.0, -1.0, -1.0], [1.0, 1.0, 1.0],
            ]))
            self.projection.bias.zero_()

    def forward(self, features):
        return self.projection(features.clamp(-1.0, 1.0))

model = NumericRouter().eval()
example = torch.tensor([[-0.5, -0.2, 0.1], [0.8, 0.7, 0.6]])
batch = torch.export.Dim("batch", min=1, max=8)
program = torch.export.export(
    model, (example,), dynamic_shapes=({0: batch},),
)

with TemporaryDirectory() as directory:
    artifact = Path(directory) / "router.pt2"
    torch.export.save(program, artifact)
    restored = torch.export.load(artifact).module()
    with torch.inference_mode():
        for size in (1, 2, 5, 8):
            inputs = torch.linspace(-2, 2, size * 3).reshape(size, 3)
            expected, actual = model(inputs), restored(inputs)
            torch.testing.assert_close(actual, expected)
            assert torch.equal(actual.argmax(-1), expected.argmax(-1))
        classes = restored(example).argmax(-1).tolist()
    assert classes == [0, 1]
    for shape in ((2, 4), (9, 3)):
        try:
            restored(torch.zeros(shape))
        except (AssertionError, RuntimeError):
            pass
        else:
            raise AssertionError(f"exported graph accepted invalid shape {shape}")

print(f"restored class choices: {classes}")
print("logits match for batches 1, 2, 5, 8; feature width 4 and batch 9 rejected")
print("CPU graph round-trip only: no mobile runtime or accelerator measured")

restored class choices: [0, 1]
logits match for batches 1, 2, 5, 8; feature width 4 and batch 9 rejected
CPU graph round-trip only: no mobile runtime or accelerator measured

The example checks four permitted batches and rejects both a wrong feature dimension and a batch above the declared maximum. This exported artifact accepts batches 1 through 8, while eager execution also accepts batch 9. It uses PyTorch on CPU after reload. Backend lowering, mobile packaging, delegate precision, and device performance still need their own checks.

View larger

Heat, battery, and physical edge limits

Cooling depends on the enclosure, contact with the user, ambient temperature, workload, and any active cooling. A fanless device transfers heat through conduction, convection, and radiation. There is no universal 3-to-5 W sustained limit for all handhelds. Our 3.5 W allowance is an assumed product budget, not a chip TDP or a regulatory threshold.

Dynamic voltage and frequency scaling (DVFS)

Dynamic voltage and frequency scaling (DVFS) changes operating points to manage performance and power. Thermal control can also limit available performance. Junction, battery, and chassis temperatures are different measurements; throttling thresholds and touch-temperature requirements depend on the device and applicable conditions. Neither 85°C nor 42°C is a universal rule.

Android's Thermal API exposes thermal status and headroom. Its documentation notes that mappings and support vary by device; even THERMAL_STATUS_NONE does not always establish an absence of throttling.^{[15]Reference 15Android Thermal APIhttps://developer.android.com/games/optimize/adpf/thermal} Record the actual device, OS/runtime versions, ambient conditions, charging state, workload, output length, and measurement method.

View larger

This code checks three synthetic samples. Evaluate hardware soak, tail latency, and performance between samples with separate measurements:

samples_tps = [23.0, 19.4, 16.8]
minimum_sustained_tps = 18.0
burst_tps = samples_tps[0]
floor_tps = min(samples_tps)

print(f"burst throughput: {burst_tps:.1f} t/s")
print(f"lowest sampled throughput: {floor_tps:.1f} t/s")
print(f"sampled throughput gate passes: {floor_tps >= minimum_sustained_tps}")

burst throughput: 23.0 t/s
lowest sampled throughput: 16.8 t/s
sampled throughput gate passes: False

The lowest supplied sample fails 18 tokens/s. A repair could involve a smaller candidate, a supported kernel, better workload scheduling, or fewer generated tokens, but each changes tradeoffs and needs measurement. Quantization may add conversion overhead; output caps can damage answer quality. Test continuous load and a realistic shift duty cycle, including warm starts and hot ambient conditions.

Battery energy accounting

Assume a $15$ Wh battery; $4{,}000$ mAh at a nominal $3.7$ V corresponds to $14.8$ Wh before usable-capacity and aging considerations. With constant measured power $P$ at a defined boundary and generation rate $r$:

At $18$ tokens per second, generating $1000$ tokens takes $1000/18 \approx 55.6$ seconds. If the system draws $3.5$ W while decoding, it consumes:

Do those watts include display, retrieval, prefill, idle time, and radios, or only incremental decode power? That boundary changes the interpretation. A lower-power candidate can consume more energy per token if it is sufficiently slower. This synthetic shift includes decode, prefill/retrieval, and remaining idle time with mutually exclusive whole-system power assumptions:

shift_seconds = 8 * 3600
generated_tokens = 30_000
battery_wh = 15
retrieval_prefill_seconds = 600
retrieval_prefill_watts = 4
idle_watts = 1.2

for name, decode_watts, tokens_per_second in [("candidate-A", 3.5, 18), ("candidate-B", 6, 10)]:
    decode_seconds = generated_tokens / tokens_per_second
    idle_seconds = shift_seconds - decode_seconds - retrieval_prefill_seconds
    assert idle_seconds >= 0
    decode_wh = decode_watts * decode_seconds / 3600
    total_wh = (decode_watts * decode_seconds
                + retrieval_prefill_watts * retrieval_prefill_seconds
                + idle_watts * idle_seconds) / 3600
    print(f"{name}: decode={decode_wh:.2f} Wh, shift={total_wh:.2f} Wh, below nominal={total_wh < battery_wh}")

candidate-A: decode=1.62 Wh, shift=11.13 Wh, below nominal=True
candidate-B: decode=5.00 Wh, shift=14.07 Wh, below nominal=True

View larger

Battery qualification must account for reserve, usable capacity, aging, and longer prompts, which can erase a margin below nominal capacity. For varying power, integrate $P(t)$ over time; dividing average power by average token rate requires a consistent measurement interval. Measure total energy for a complete task when comparing classifiers, extractive answers, and generators of different lengths.

Hybrid edge-cloud routing with fail-closed privacy

Some products permit a cloud route for selected inputs; our restricted facility task does not. Complexity alone supplies no authority to transmit data. On an isolated terminal, an unsupported question needs local abstention or review. The optional router below illustrates a different product mode with an explicitly permitted public-data route, not an exception to the terminal's no-egress requirement.

Mathematical confidence routing

Neither self-reported confidence nor token probabilities prove an answer is correct. Some useful diagnostic features are:

1. Predictive entropy: Calculate the entropy of the next-token probability distribution: $$H(p) = -\sum_{i=1}^{V} p_i \log_2 p_i$$ Low next-token entropy means a concentrated distribution over the tokens being considered. A confidently wrong number can have low entropy; several equally correct phrasings can have high entropy. Temperature, logits filtering, vocabulary, and prefix affect this statistic.
2. Top-2 softmax margin: Measure the gap between the most likely token and the runner-up: $$\Delta p = p_{(1)} - p_{(2)}$$ A small margin means the two token probabilities are close. Calibrate statistical significance and escalation thresholds against held-out task outcomes.
3. Evidence checks: Verify that cited passages apply and support the answer, then confirm their identifiers resolve.

If you combine these into a routing score, calibrate it against held-out answer correctness for the task and recheck after model, quantization, prompt, or sampling changes. Report coverage and error rates for answered versus abstained cases. The router's 0.80 threshold is an invented policy value, not the result of such calibration.

A fail-closed transmission policy

When uncertainty triggers an escalation request, application security policies must govern whether data can actually leave the device.

A permitted hybrid product needs a transmission policy enforced outside generated text:

• Data classification: Trusted classification must cover the assembled payload: user input, retrieved documents, history, attachments, and tool results. A model or user-provided public label is not authority. Unknown or mixed sensitive inputs retain the restrictive policy.
• Fail-closed rule: If a query touches confidential or restricted data, or if the device is currently offline, cloud escalation is strictly forbidden. The system must fail closed: it uses a supported local answer, routes to local human review, or returns a safe local abstention.
• Surrounding paths: Review telemetry, crash reports, sync, backups, tools, downloads, and fallback paths. Use minimal allowed metrics, access controls, retention, and egress restrictions. A locally running model does not establish privacy for the surrounding app.

View larger

The following is a pure decision function over assumed trusted inputs. It does not perform redaction, classify a payload, check a permission receipt, send a request, or enforce a network boundary. Bind real authorization to the actual payload and destination immediately before transmission:

import math

def route_query(score, *, supported=False, data_class="unknown",
                cloud_approved=False, redaction_reviewed=False, online=False):
    valid_score = type(score) in (int, float) and 0 <= score <= 1 and math.isfinite(score)
    if not valid_score:
        return "human_review_local_only"
    if supported is True and score >= 0.80:
        return "local_answer"
    if (data_class == "public" and cloud_approved is True
            and redaction_reviewed is True and online is True):
        return "cloud_allowed_after_redaction"
    return "human_review_local_only"

public_approval = dict(data_class="public", cloud_approved=True,
                       redaction_reviewed=True, online=True)
cases = [
    ("routine_public", route_query(0.94, supported=True), "local_answer"),
    ("hard_public", route_query(0.41, **public_approval), "cloud_allowed_after_redaction"),
    ("hard_private", route_query(0.38, data_class="restricted"), "human_review_local_only"),
    ("unknown", route_query(0.41), "human_review_local_only"),
    ("offline", route_query(0.41, **(public_approval | {"online": False})), "human_review_local_only"),
]
for name, actual, expected in cases:
    assert actual == expected
    print(f"{name}: {actual}")
for invalid in (float("nan"), float("inf"), -0.1, 1.1, 10**1000, True, None):
    assert route_query(invalid, supported=True, **public_approval) == "human_review_local_only"
for permission in ("cloud_approved", "redaction_reviewed", "online"):
    assert route_query(0.41, **(public_approval | {permission: False})) == "human_review_local_only"

routine_public: local_answer
hard_public: cloud_allowed_after_redaction
hard_private: human_review_local_only
unknown: human_review_local_only
offline: human_review_local_only

hard_private, unknown, and offline select local review. The tests establish decisions for those supplied inputs, not protection against mislabeled content, mutable payloads, unauthorized callers, or an independent network path.

Candidate selection and over-the-air deployment gates

With constraints defined and routing established, benchmark candidate model families against your specific hardware SKU:

Here are synthetic candidates checked against two thresholds. Replace these rows with measurements from the named model families on the target hardware before making a selection:

candidates = [
    {"name": "tiny_policy", "quality": 0.88, "sample_min_tps": 33},
    {"name": "policy_student", "quality": 0.93, "sample_min_tps": 19},
    {"name": "hot_policy", "quality": 0.95, "sample_min_tps": 11},
]
min_quality, min_sample_tps = 0.92, 18

approved = [
    item["name"]
    for item in candidates
    if item["quality"] >= min_quality
    and item["sample_min_tps"] >= min_sample_tps
]
assert approved == ["policy_student"]
print(f"candidates passing the two fixture thresholds: {approved}")

candidates passing the two fixture thresholds: ['policy_student']

Only policy_student passes these two fixture thresholds. The row names do not identify actual models; diagnose thermal throttling with device telemetry. Quality uncertainty, memory, energy, privacy, packaging, and sustained workload checks remain outstanding.

Over-the-air (OTA) lifecycle and dual-slot rollback

An interrupted in-place update can leave the model feature unusable. Preserve the last approved complete bundle while staging a candidate. A/B slots are one design; protection depends on explicit activation and recovery logic. Android's system-update documentation illustrates active, bootable, and successful states.^{[16]Reference 16A/B Seamless System Updateshttps://source.android.com/docs/core/ota/ab}

For this terminal, design the model update around four questions:

1. What is authorized? Authenticate a manifest against trusted keys. Bind hashes, lengths, model configuration, tokenizer, templates, modality assets, runtime requirements, device compatibility, version, and expiry to that signed metadata. A valid signature authenticates the signed bytes under the trust policy; separate checks must establish compilation, quality, and safety. Plan key rotation and revocation, version checks against replay, and freshness under the offline update policy. TUF specifies separate signature, version, expiry, and artifact checks.^{[17]Reference 17The Update Framework Specificationhttps://theupdateframework.github.io/specification/latest/}
2. What survives interruption? Download to inactive storage, with bounded size and safe paths. Check all artifacts against authenticated metadata before activation. Test truncated downloads, disk exhaustion, and power loss. Crash durability also depends on the platform's durable-write protocol; use it with a recovery journal. A strictly isolated terminal needs an approved offline transfer rather than online OTA.
3. What is compatible? Load the candidate on each supported device/runtime family. Check tokenizer/template behavior, reference outputs within declared tolerances, citations, abstention, memory peaks, and a representative offline workload. Five smoke cases cover only those examples; evaluate broader task quality and ten-minute thermal behavior separately. Different weights need not have identical logits to the previous model.
4. Who decides recovery? Activate only after required checks. Start a fresh session or use an explicitly supported transition; cached states from different weights/tokenizers are generally incompatible. Track health and restore a compatible approved bundle on failure. Avoid running two full models simultaneously if peak memory cannot support it. A rollback must respect anti-replay policy and schema compatibility; it is a controlled recovery action, not permission to install any old signed version.

View larger

Diagnosing edge deployment breakdowns

The following authored failure scenarios demonstrate a diagnostic process. On a target device, collect observations that distinguish possible causes:

1. The model draws 6W and runs at 5 tokens per second

Symptom: A 1B candidate runs at 35 tokens/s on the developer laptop but only 5 on the terminal, where whole-system power is 6 W and the enclosure feels hot.

Investigate: Check delegate placement, partition timing, synchronization, formats, clocks, and competing load. Compare supported accelerator and CPU paths with the same workload. Laptop throughput is not a mobile baseline.

Change after diagnosis: Choose a supported lowering or a different candidate. Replacing activations changes the model's computation and may require retraining; verify GELU or SiLU support in the target runtime. Recheck quality and energy as well as speed.

2. INT4 quantization destroys citation accuracy

Symptom: The unquantized FP16 model cites maintenance manual paragraphs with 96% accuracy, but the INT4 quantized model hallucinates non-existent section numbers.

Investigate: Compare tokenization, retrieval, template, sampling, conversion correctness, and paired reference outputs before attributing the regression to particular layers. A text-only decoder need not have cross-attention.

Change after diagnosis: Try a supported calibrated format, greater precision for a demonstrably sensitive tensor, or a different model. AWQ/GPTQ may help on suitable models; FP16 embeddings may also break the memory budget. Repeat held-out source-support and abstention evaluation on the exact artifact.

3. Application crashes after fifteen minutes of continuous conversation

Symptom: The handheld app operates smoothly during brief queries, but terminates abruptly without an error log during prolonged diagnostic sessions.

Investigate: Inspect OS termination records, allocation peaks, retained sessions, cache reservations, and crash reports. Linear cache growth is one possible cause for full attention; a watchdog, unsupported operation, or leak points to a different repair.

Change after diagnosis: Bound sessions and context, reduce a measured allocation, or select a suitable trained GQA/hybrid architecture. Cache quantization requires runtime support and quality checks. Arbitrarily evicting full-attention KV changes what the model can attend to; trimming history and rebuilding a prompt is different from native sliding-window attention.

4. Throughput collapses halfway through a work shift

Symptom: The model performs well in morning testing, but field technicians complain that answers slow to an unusable crawl during afternoon inspections.

Investigate: Correlate workload and rate with thermal state, ambient temperature, charging, background jobs, and power. A decline alone cannot identify the thermal governor as the cause.

Change after diagnosis: Evaluate smaller candidates, supported kernels, shorter outputs, or workload duty-cycle changes. Size alone does not ensure power below 3 W. A generated-token cap must leave room for the required answer and needs quality evaluation.

5. Over-the-air update bricks offline functionality

Symptom: Technicians update their handhelds before heading into the field, only to find the app crashes on launch when cell connectivity drops.

Investigate: Reproduce with networking disabled. Check missing files, lazy downloads, tokenizer/template mismatches, modality assets, authorization refresh, and incompatible runtime versions. A missing-token exception is only one possibility.

Change after diagnosis: Stage and authenticate a complete compatible bundle, test first launch offline, and exercise interrupted activation and recovery. Retain an approved offline recovery path.

Core takeaways for edge engineering

Use the vault job as a rejection test before you compare download sizes. These are the checks that still have to pass on the device:

1. Write the task contract: Compare lookup, classification, extraction, and generation against the same real job. Hypothetical sizing can reject candidates; it cannot qualify them.
2. Separate training signals from truth: Soft targets expose teacher preferences. Use verifiers to check correctness against their contracts. Exact forward-KL gradients have a 1/tau factor; the familiar squared scaling is a high-temperature approximation.
3. Budget the actual architecture: Stored bytes, active weight traffic, KV/recurrent state, modality components, pools, and temporary allocations are different quantities. Effective parameter labels cannot establish fit.
4. Inspect the installed runtime: Test graph lowering on the target mobile runtime. Partition boundaries can help or hurt; measure placement, output quality, and critical-path time.
5. Measure a shift, not only a burst: Track visible-answer latency, generation work, memory peaks, thermal behavior, and energy for the whole product under representative conditions.
6. Bind release and privacy decisions to real inputs: Enforce complete-payload transmission policy and authenticated, compatible, recoverable updates. Verify the actual network path, installed weights, signature, and recovery flow against those requirements.

Follow-up architectural questions

• Assume $34$ GB/s and one active-weight stream per step. A $15$-step/s target permits at most about $34/15 \approx 2.27$ GB for that traffic before other costs. What evidence would invalidate this assumption, and why does it not select a quantization method by itself?
• An NPU delegate reports 98% operator coverage but misses latency. Which partition boundaries occur per generated token, and how will you separate synchronization, conversion, copies, and compute?
• Derive the exact forward-KL gradient and its high-temperature approximation. What changes when teacher and student token events cannot be directly aligned?
• A validly signed update changes the tokenizer and prompt template. What cache/session transition, offline test, and recovery evidence must precede activation?

Evaluation rubric

• Differentiate parameter count from memory bandwidth and thermal dissipation constraints.
• Derive the Hinton knowledge distillation loss and explain the mathematical role of temperature scaling and gradient compensation.
• Calculate KV cache memory savings from Grouped-Query Attention.
• Diagnose and resolve mobile NPU operator fallbacks and CPU memory thrashing.
• Implement fail-closed routing and cryptographic verification for edge model artifacts.