SLM Specialization & Edge Deployment

Picture a technician taking an inspection handheld into a utility vault with no connectivity. The product specification forbids sending facility manuals or incident text off the device. For this lesson's hypothetical terminal, the proposed targets are time to first visible answer under 250 ms, at least 18 generated tokens/s after ten minutes of use, and enough battery for an eight-hour shift. These are design requirements, not measured device results or industry standards. Actual field procedures need qualified review; fluent generated instructions alone are insufficient evidence of safety.

A lookup table can already answer "which torque specification applies to flange B-12?" A lightweight classifier can route an incident report to the right department. Generating a cited, multi-sentence repair procedure is an entirely different feature with a much stricter error budget. Just because a 4-billion-parameter checkpoint fits on the handheld's flash storage doesn't mean the feature is ready to ship.

Build on workstation memory sizing, then add battery and cooling constraints, application isolation, and a product-wide data-flow policy. Offline operation means connectivity is unavailable. An air gap is an isolation boundary; a firewall rule or a local inference call alone does not establish one.

A small language model (SLM) is small relative to its comparison set; there is no universal parameter cutoff. This lesson includes sub-billion and several-billion-parameter models. Specialization adapts behavior to a defined job through data, training, retrieval, or constraints. It does not necessarily remove parameters or erase unrelated knowledge. Edge deployment concerns execution near the user or data source, including mobile devices and larger local appliances.

Build on local memory and hardware sizing and knowledge distillation. Here we focus on what survives the journey from server-side training to physical hardware. Our executable examples demonstrate numerical routing, CPU graph export, and dispatch decisions.

Start from the physical constraint, not the model size

Broad capability can be useful, but the terminal's job is narrower: extract error codes, locate approved manual passages, and explain the retrieved evidence. Start by measuring those behaviors. A specialist can outperform a larger baseline on a particular evaluation; neither its size nor a teacher relationship guarantees that outcome. Specialization is not equivalent to pruning, and local execution removes an inference round-trip only when the rest of the pipeline is local too.

Before picking weights, define each offline job and quantify its failure cost. Ask whether a deterministic lookup, a classical classifier, an extractive retrieval pipeline, or a generative SLM is required. "What model fits in 4 GB of RAM?" supplies one capacity constraint; it is not a complete engineering specification.

For an industrial inspection handheld, we define candidate operational bars across four distinct device tasks:

Notice how no single row mandates a parameter count. An image classifier's total inference latency differs completely from a generative model's first-token latency. Time to first token (TTFT) must also measure the complete pipeline: application dispatch, local retrieval, prompt templating, tokenization, and prefill compute until the first visible answer token arrives. If a model emits hidden reasoning tokens first, track both first-generated-token latency and first-visible-token latency.

Here is a filter over invented scorecard rows. The privacy_local field is an assumed prior review result, not something this code establishes:

Photo support misses both its latency ceiling and quality threshold. Intent routing and policy Q&A pass these preliminary checks, yet hardware qualification remains to be done.

These approved rows also separate distinct model workloads. Intent routing simply selects an execution lane. Policy Q&A must retrieve rules, ground its response in exact citations, and abstain when documentation is missing.

A bandwidth worksheet, with its assumptions visible

For a dense model whose matrix weights exceed cache capacity, batch-1 decode often spends much of its time moving weights. Compute, attention, small kernel launches, format conversion, and runtime scheduling can also limit it. Embedding lookups, inactive experts, cached weights, and immediate reuse invalidate an assumption that every stored parameter is read from DRAM on every token.

Let $P$ count matrix weights used in this step. Assuming one complete packed-weight read, two floating-point operations per multiply-add, and no other traffic gives:

$\text{Arithmetic Intensity} = \frac{\text{FLOPs per token}}{\text{Bytes moved per token}} \approx \frac{2 \times P}{P \times \frac{\text{bits}}{8}} = \frac{16}{\text{bits}} \text{ FLOPs/byte}$

This model gives 1 FLOP/byte for 16-bit weights and 4 for four-bit weights. At an assumed 50 GB/s, the latter corresponds to 200 GFLOP/s of modeled matrix work. Integer TOPS, sparse operation counts, and floating-point FLOPs are different metrics; dividing this work rate by an NPU's advertised TOPS does not establish utilization or idle time. Quantization scales and unpacking also cost bytes and work.

We calculate the theoretical decoding ceiling using the weight-streaming bound:

$\text{Weight-streaming ceiling (tokens/s)} = \frac{\text{Memory bandwidth (GB/s)}}{\text{Active weight footprint (GB)}}$

Assume a dense model step reads 3.8 billion active matrix weights once over a hypothetical 50 GB/s bus. The raw payload is 1.9 GB at four bits or 3.8 GB at eight bits. This excludes quantization metadata, mixed-precision tensors, KV/activation traffic, and invocation overhead; it is not an artifact-size prediction for every 3.8B model:

Under these assumptions, the eight-bit candidate is ruled out: 50 / 3.8 is only 13.2 steps/s. Four-bit storage is not ruled out by the 26.3 steps/s estimate. It still needs a working kernel and device measurement. Changing the model's active weights, cache reuse, or decoding algorithm changes this worksheet's applicability; these numbers are not guaranteed token rates.

Total memory footprint requires its own accounting:

$\text{Memory}_{\text{total}} = \text{Weights} + \text{KV Cache} + \text{Activation Buffers} + \text{Runtime Overhead}$

Flash storage, system RAM, and accelerator-accessible allocations are different budgets. A download can fit while serving allocations fail or the OS terminates the app under memory pressure. Include tokenizer and retrieval storage, multimodal encoders, runtime pools, and other applications. An individual allocation limit and a system low-memory termination are different failure mechanisms.

MobileLLM introduced dedicated architectural optimizations for sub-billion models deployed to mobile chips.^{[1]Reference 1MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Caseshttps://arxiv.org/abs/2402.14905} Use its empirical findings to select model topologies, but always measure actual energy, memory residency, and throughput on your specific target SKU.

Distillation: transferring dark knowledge to compact silicon

Knowledge distillation trains a student against a teacher's predictions or other selected signals. The teacher need not be a frontier language model. The student can use a different architecture, and deployment need not include the teacher's weights. Whether this improves the device task depends on the training setup and evaluation.^{[2]Reference 2Distilling the Knowledge in a Neural Network.https://arxiv.org/abs/1503.02531}

In classification, hard labels tell the student which class won, but hide how the teacher ranked the alternatives. In language modeling, teacher logits provide a dense probability distribution over the entire vocabulary. This distribution contains "dark knowledge": the subtle semantic relationships between runner-up tokens that were plausible versus those that were absurd.

The distillation objective

When distilling token by token, we combine standard cross-entropy on ground-truth tokens with Kullback-Leibler (KL) divergence over temperature-scaled teacher and student distributions:

$$\begin{aligned} \mathcal{L}_{\text{distill}} =&\;\alpha \tau^2 \cdot \text{KL}\left( \sigma\left(\frac{z_T}{\tau}\right) \parallel \sigma\left(\frac{z_S}{\tau}\right) \right) \\ &+ (1-\alpha) \cdot \text{CE}(y, z_S) \end{aligned}$$

Here $z_T$ and $z_S$ are aligned teacher and student logits, positive $\tau$ is the temperature, $\sigma$ is softmax, and $0\leq\alpha\leq1$ balances supervision. The hard-label CE uses unscaled student logits; the soft term uses the same temperature for teacher and student.

Temperature changes how strongly logit differences affect probabilities. For a peaked distribution, increasing positive $\tau$ gives alternatives more mass without changing their rank:

$p_i(\tau) = \frac{\exp(z_i / \tau)}{\sum_j \exp(z_j / \tau)}$

Notice the ratio between two candidate tokens $B$ and $C$:

$\frac{p(B)}{p(C)} = \exp\left(\frac{z_B - z_C}{\tau}\right)$

For a fixed teacher, the exact gradient of the unscaled forward KL with respect to a student logit is:^{[2]Reference 2Distilling the Knowledge in a Neural Network.https://arxiv.org/abs/1503.02531}

$\frac{\partial \mathcal{L}_{\text{KL}}}{\partial z_{S,i}} = \frac{p_{S,i}-p_{T,i}}{\tau}$

There are two temperature effects: the chain rule contributes $1/\tau$, and at high temperature the probability difference is approximately proportional to $1/\tau$. Center each logit vector by subtracting its mean. For a vocabulary of size $V$:

$$\frac{\partial \mathcal{L}_{\text{KL}}}{\partial z_{S,i}} \approx \frac{\widetilde z_{S,i}-\widetilde z_{T,i}}{V\tau^2}$$

Multiplying by $\tau^2$ approximately compensates for this high-temperature shrinkage. It does not guarantee equal gradients at every temperature, stable optimization, or successful training. Temperature and loss weights remain validation choices.

Check the exact derivative against a finite difference before relying on the approximation:

The scaled values remain of comparable order, but are not equal. For these centered logits, the high-temperature limit is about -0.233333 for the first coordinate. This example checks a derivative, not a trained language model.

Let's test this behavior with teacher logits [4.0, 2.0, 1.0]:

At $\tau=1$, A receives 84.4% and C 4.2%. At $\tau=3$, A remains preferred, while B and C receive more mass. A one-hot label discards their relative preferences. If the teacher is wrong, matching those preferences can transfer that error too.

Distillation with execution-based verifiers

Direct token-level KL needs distributions over aligned token events at aligned prefixes. Matching token IDs alone is insufficient if their meanings differ. Different tokenizers need an alignment method, or sequence-level supervision using teacher text retokenized by the student. Closed APIs may expose only sampled text or a limited set of log probabilities; these are not automatically a full-vocabulary distribution.

Phi-3 combined filtered public data with synthetic examples; it is a data-curation result, not a universal requirement to use only teacher-generated text.^{[3]Reference 3Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonehttps://arxiv.org/abs/2404.14219} For domain specialization, include approved documents, human examples, unanswerable questions, and verified synthetic cases. Split evaluation by document, equipment family, or site before generating variants so near-duplicates cannot leak across the split.

Check each synthetic example against the contract it is supposed to teach:

1. Code and tool-use verification: Run bounded tests in an environment with appropriate isolation, resource limits, and no secrets. A container alone is not a sufficient boundary for hostile code. A passing test covers its assertions, not every behavior.
2. Schema validation: Check required fields, types, and allowed values. Well-formed JSON can still request an unauthorized action or contain a wrong fact.
3. Retrieval grounding: Check source identity, version, applicable equipment and conditions, and whether the passage supports the claim. An existing paragraph number or matching numeric substring alone is insufficient.

Here is an invented maintenance fixture, not a real torque recommendation. Both outputs pass a naive citation-and-number check:

The second answer quotes a real value for the wrong equipment. Even the better check above covers only this tiny structured fixture. Review ambiguous cases and measure held-out correctness and abstention; curation cannot guarantee a flawless student.

MiniLLM uses on-policy optimization of reverse KL, $\text{KL}(p_S \parallel p_T)$, with variance reduction, teacher-mixed sampling, and length normalization.^{[4]Reference 4MiniLLM: Knowledge Distillation of Large Language Modelshttps://arxiv.org/abs/2306.08543} Reverse KL tends to favor teacher modes rather than covering every alternative. That is an objective tradeoff, not a guarantee of factual correctness or a replacement for task evaluation.

This CPU calculation combines three invented signals. In real training, freeze teacher gradients, align token prefixes, mask padding, and define reductions explicitly. Hidden-state matching also needs chosen layers and compatible dimensions, often a learned projection; unrelated hidden coordinates are not interchangeable:

Once the student finishes training, evaluate its exported and quantized artifact again. Sensitivity depends on the model, format, calibration data, kernels, and workload. There is no universal sub-2B cutoff requiring one quantization method.

These synthetic rows illustrate a quality-versus-latency decision, not measured INT8/INT4 results. quality must have a defined task metric and uncertainty before using this filter in a release:

In this fixture, INT4 loses 0.053 quality and fails the 0.02 tolerance; INT8 loses 0.002 and passes both listed thresholds. Neither row establishes hardware qualification. Compare paired held-out outcomes, source support, abstention, latency, and memory under the actual decode settings; perplexity alone is insufficient.

Architecture choices that matter at small scale

A smaller parameter count doesn't guarantee efficient execution. Layer structure and attention heads change the work, memory traffic, and cache behavior.

Meta's MobileLLM research revealed key architectural principles for sub-billion models:^{[1]Reference 1MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Caseshttps://arxiv.org/abs/2402.14905}

1. Depth versus width: The paper's approximately 125M/350M experiments favored deeper, thinner designs. Scaling laws do not require equal parameter allocation to depth and width, and this result is not universal across training recipes or devices.
2. Embedding tying: Sharing input and output weights reduces storage. One ablation reduced 135M to 119M parameters, with a small accuracy drop; reallocating some savings to depth recovered it.
3. Immediate block reuse: MobileLLM-LS repeats each block with shared weights. Cache locality can reduce repeat weight traffic, but compute and attention state still cost resources. Cache residency and zero extra DRAM traffic are not guaranteed. Its iPhone 13 FP16/MPS experiment measured 15.6 versus 16.0 ms execution, averaged over 50 iterations, not our terminal's sustained token rate.
4. GQA: Fewer KV heads reduce cache payload for otherwise equal dimensions. Choose a trained architecture; replacing an existing model's attention heads is not an inference-time switch.

The same paper found its tested distillation from Llama-v2 7B slower to train and comparable or inferior in accuracy to hard-label training. Architectural improvement and successful distillation are separate hypotheses.^{[1]Reference 1MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Caseshttps://arxiv.org/abs/2402.14905}

KV cache growth can rapidly trigger out-of-memory crashes on devices with tight RAM. Let's calculate the memory savings of 4 KV heads compared to 16 KV heads across a 4,096-token context window:

The hypothetical homogeneous cache shrinks from 384 to 96 MiB. This excludes alignment, pools, quantization metadata, and temporary state. It does not establish total memory fit or fourfold speed. Hybrid attention and recurrent layers require their own per-layer formulas.

These documented model examples illustrate different tradeoffs, checked September 22, 2026. They are not a ranking or an exhaustive list of current candidates:

• Phi-4-mini-instruct (3.8B): Employs GQA, shared input-output embeddings, a 128K context window, and high-density synthetic data tuning for tool calling.^{[5]Reference 5Phi-4-mini-instruct Model Cardhttps://huggingface.co/microsoft/Phi-4-mini-instruct}
• Gemma 4 E2B and E4B: The card lists 2.3B/4.5B effective parameters and 5.1B/8B including embeddings. Per-layer embeddings use token lookups. Effective is not the same label as an MoE active-expert count. Include the selected artifact's stored tensors, encoder components, cache, and measured serving residency; the effective count cannot establish fit.^{[6]Reference 6Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4}
• Qwen3-1.7B: Its card documents 28 layers, 16 query/8 KV heads, a 32,768-token context, and thinking/non-thinking chat-template modes.^{[7]Reference 7Qwen3-1.7B Model Cardhttps://huggingface.co/Qwen/Qwen3-1.7B} Thinking tokens consume generation work before the visible answer. Budget and evaluate both modes under the installed runtime; an output cap can truncate the answer rather than improve it.

On-device runtimes: mapping graphs to silicon

Next inspect how the selected model and format execute. A runtime may use supported accelerator kernels, native CPU kernels, or both; CPU execution is not necessarily emulation.

Mobile NPUs and the danger of CPU fallback

An NPU may improve energy efficiency for supported work. Advertised TOPS/W figures across formats, sparsity modes, and chips do not establish energy per generated token for your application.

Check the chosen accelerator's restrictions:

1. Shape support: Shapes, ranges, and cache updates may impose restrictions. Buckets can help some backends, but padding can waste work and change state if masking is wrong.
2. Numerical support: Weight storage, multiply format, and accumulator precision are separate choices. Check this backend's exact support rather than assuming all NPUs use INT4/INT8 with FP16 accumulation.
3. Operator support: An unsupported operation can remain on CPU, cause export failure, or require a different lowering. Check placement and partition boundaries, not only a percentage of delegated operators.

Partial delegation can add synchronization, layout conversion, intermediate materialization, or copies. Unified physical memory does not remove all those costs, and it does not imply a mandatory copy between two private memories either. Some partitions are beneficial. Profile their frequency and critical-path time, then compare a supported fused graph and an optimized CPU path.^{[8]Reference 8ExecuTorch Backendshttps://docs.pytorch.org/executorch/stable/backends-overview.html[11]Reference 11Deploy ONNX Runtime Mobilehttps://onnxruntime.ai/docs/tutorials/mobile/}

Apple's preparation guide explicitly distinguishes compression methods and device strengths, including GPU-oriented four-bit weight quantization and stateful KV-cache updates. Core ML conversion is not a promise that the whole LLM runs on the Neural Engine.^{[9]Reference 9Bring Your Machine Learning and AI Models to Apple Siliconhttps://developer.apple.com/videos/play/wwdc2024/10159/}

Testing graph export on CPU

First check capture, serialization, and declared shape constraints. torch.export records a graph that downstream tools can lower; it is not an ExecuTorch executable or an NPU compiler.^{[14]Reference 14torch.export API Referencehttps://docs.pytorch.org/docs/stable/user_guide/torch_compiler/export/api_reference.html} This CPU round-trip uses pinned PyTorch 2.14.0, the version checked for this lesson:

The example checks four permitted batches and rejects a wrong feature dimension. It uses PyTorch on CPU after reload. Backend lowering, mobile packaging, delegate precision, and device performance still need their own checks.

Heat, battery, and physical edge limits

Cooling depends on the enclosure, contact with the user, ambient temperature, workload, and any active cooling. A fanless device transfers heat through conduction, convection, and radiation. There is no universal 3-to-5 W sustained limit for all handhelds. Our 3.5 W allowance is an assumed product budget, not a chip TDP or a regulatory threshold.

Dynamic voltage and frequency scaling (DVFS)

Dynamic voltage and frequency scaling (DVFS) changes operating points to manage performance and power. Thermal control can also limit available performance. Junction, battery, and chassis temperatures are different measurements; throttling thresholds and touch-temperature requirements depend on the device and applicable conditions. Neither 85°C nor 42°C is a universal rule.

Android's Thermal API exposes thermal status and headroom. Its documentation notes that mappings and support vary by device; even THERMAL_STATUS_NONE does not always establish an absence of throttling.^{[15]Reference 15Android Thermal APIhttps://developer.android.com/games/optimize/adpf/thermal} Record the actual device, OS/runtime versions, ambient conditions, charging state, workload, output length, and measurement method.

This code checks three synthetic samples. It is not a ten-minute hardware soak test, a tail-latency check, or proof of performance between samples:

The lowest supplied sample fails 18 tokens/s. A repair could involve a smaller candidate, a supported kernel, better workload scheduling, or fewer generated tokens, but each changes tradeoffs and needs measurement. Quantization may add conversion overhead; output caps can damage answer quality. Test continuous load and a realistic shift duty cycle, including warm starts and hot ambient conditions.

Battery energy accounting

Assume a 15 Wh battery; 4,000 mAh at a nominal 3.7 V corresponds to 14.8 Wh before usable-capacity and aging considerations. With constant measured power $P$ at a defined boundary and generation rate $r$:

$E_{\text{1K tokens}} = \frac{P_{\text{avg}} \times 1000}{\text{Tokens per second} \times 3600} \text{ Wh}$

If the system draws 3.5W while decoding at 18 tokens per second, generating 1,000 tokens takes 55.6 seconds and consumes:

$E_{\text{1K tokens}} = \frac{3.5 \times 1000}{18 \times 3600} \approx 0.054 \text{ Wh} = 54 \text{ mWh}$

Do those watts include display, retrieval, prefill, idle time, and radios, or only incremental decode power? That boundary changes the interpretation. A lower-power candidate can consume more energy per token if it is sufficiently slower. This synthetic shift includes decode, prefill/retrieval, and remaining idle time with mutually exclusive whole-system power assumptions:

Being below nominal capacity is not a battery qualification. Reserve, usable capacity, aging, and longer prompts can erase the margin. For varying power, integrate $P(t)$ over time; dividing average power by average token rate requires a consistent measurement interval. Measure total energy for a complete task when comparing classifiers, extractive answers, and generators of different lengths.

Hybrid edge-cloud routing with fail-closed privacy

Some products permit a cloud route for selected inputs; our restricted facility task does not. Complexity alone supplies no authority to transmit data. On an isolated terminal, an unsupported question needs local abstention or review. The optional router below illustrates a different product mode with an explicitly permitted public-data route, not an exception to the terminal's no-egress requirement.

Mathematical confidence routing

Neither self-reported confidence nor token probabilities prove an answer is correct. Some useful diagnostic features are:

1. Predictive entropy: Calculate the entropy of the next-token probability distribution: $H(p) = -\sum_{i=1}^{V} p_i \log_2 p_i$ Low next-token entropy means a concentrated distribution over the tokens being considered. A confidently wrong number can have low entropy; several equally correct phrasings can have high entropy. Temperature, logits filtering, vocabulary, and prefix affect this statistic.
2. Top-2 softmax margin: Measure the gap between the most likely token and the runner-up: $\Delta p = p_{(1)} - p_{(2)}$ A small margin means the two token probabilities are close. It is not a statistical significance test or a universal escalation threshold.
3. Evidence checks: Verify that cited passages apply and support the answer, not merely that their identifiers exist.

If you combine these into a routing score, calibrate it against held-out answer correctness for the task and recheck after model, quantization, prompt, or sampling changes. Report coverage and error rates for answered versus abstained cases. The router's 0.80 threshold is an invented policy value, not the result of such calibration.

A fail-closed transmission policy

When uncertainty triggers an escalation request, application security policies must govern whether data can actually leave the device.

A permitted hybrid product needs a transmission policy enforced outside generated text:

• Data classification: Trusted classification must cover the assembled payload: user input, retrieved documents, history, attachments, and tool results. A model or user-provided public label is not authority. Unknown or mixed sensitive inputs retain the restrictive policy.
• Fail-closed rule: If a query touches confidential or restricted data, or if the device is currently offline, cloud escalation is strictly forbidden. The system must fail closed: it either routes to local human review or returns a safe local abstention.
• Surrounding paths: Review telemetry, crash reports, sync, backups, tools, downloads, and fallback paths. Use minimal allowed metrics, access controls, retention, and egress restrictions. A locally running model does not establish privacy for the surrounding app.

The following is a pure decision function over assumed trusted inputs. It does not perform redaction, classify a payload, check a permission receipt, send a request, or enforce a network boundary. Bind real authorization to the actual payload and destination immediately before transmission:

hard_private, unknown, and offline select local review. The tests establish decisions for those supplied inputs, not protection against mislabeled content, mutable payloads, unauthorized callers, or an independent network path.

Candidate selection and over-the-air deployment gates

With constraints defined and routing established, benchmark candidate model families against your specific hardware SKU:

Here are synthetic candidates checked against two thresholds. They are not benchmark results for the named model families above:

Only policy_student passes these two fixture thresholds. The row names do not identify actual models; a low rate does not establish thermal throttling. Quality uncertainty, memory, energy, privacy, packaging, and sustained workload checks remain outstanding.

Over-the-air (OTA) lifecycle and dual-slot rollback

An interrupted in-place update can leave the model feature unusable. Preserve the last approved complete bundle while staging a candidate. A/B slots are one design, not a universal requirement or automatic protection. Android's system-update documentation illustrates explicit active, bootable, and successful states; an application model updater needs its own activation and recovery logic.^{[16]Reference 16A/B Seamless System Updateshttps://source.android.com/docs/core/ota/ab}

For this terminal, design the model update around four questions:

1. What is authorized? Authenticate a manifest against trusted keys. Bind hashes, lengths, model configuration, tokenizer, templates, modality assets, runtime requirements, device compatibility, version, and expiry to that signed metadata. A valid signature authenticates the signed bytes under the trust policy; it does not prove compilation, quality, or safety. Plan key rotation and revocation, version checks against replay, and freshness under the offline update policy. TUF specifies separate signature, version, expiry, and artifact checks.^{[17]Reference 17The Update Framework Specificationhttps://theupdateframework.github.io/specification/latest/}
2. What survives interruption? Download to inactive storage, with bounded size and safe paths. Check all artifacts against authenticated metadata before activation. Test truncated downloads, disk exhaustion, and power loss. Atomic pointer replacement alone does not guarantee crash durability; use the platform's durable-write protocol and recovery journal. A strictly isolated terminal needs an approved offline transfer rather than online OTA.
3. What is compatible? Load the candidate on each supported device/runtime family. Check tokenizer/template behavior, reference outputs within declared tolerances, citations, abstention, memory peaks, and a representative offline workload. Five smoke cases cannot establish full task quality or ten-minute thermal behavior. Different weights need not have identical logits to the previous model.
4. Who decides recovery? Activate only after required checks. Start a fresh session or use an explicitly supported transition; cached states from different weights/tokenizers are generally incompatible. Track health and restore a compatible approved bundle on failure. Avoid running two full models simultaneously if peak memory cannot support it. A rollback must respect anti-replay policy and schema compatibility; it is a controlled recovery action, not permission to install any old signed version.

Diagnosing edge deployment breakdowns

These are illustrative failure scenarios, not measured devices. Treat symptoms as observations, then collect evidence that distinguishes causes:

1. The model draws 6W and runs at 5 tokens per second

Symptom: A 1B candidate runs at 35 tokens/s on the developer laptop but only 5 on the terminal, where whole-system power is 6 W and the enclosure feels hot.

Investigate: Check delegate placement, partition timing, synchronization, formats, clocks, and competing load. Compare supported accelerator and CPU paths with the same workload. Laptop throughput is not a mobile baseline.

Change after diagnosis: Choose a supported lowering or a different candidate. Replacing activations changes the model's computation and may require retraining; GELU or SiLU support is not universal. Recheck quality and energy as well as speed.

2. INT4 quantization destroys citation accuracy

Symptom: The unquantized FP16 model cites maintenance manual paragraphs with 96% accuracy, but the INT4 quantized model hallucinates non-existent section numbers.

Investigate: Compare tokenization, retrieval, template, sampling, conversion correctness, and paired reference outputs before attributing the regression to particular layers. A text-only decoder need not have cross-attention.

Change after diagnosis: Try a supported calibrated format, greater precision for a demonstrably sensitive tensor, or a different model. AWQ/GPTQ are candidates, not guaranteed repairs, and FP16 embeddings may break the memory budget. Repeat held-out source-support and abstention evaluation on the exact artifact.

3. Application crashes after fifteen minutes of continuous conversation

Symptom: The handheld app operates smoothly during brief queries, but terminates abruptly without an error log during prolonged diagnostic sessions.

Investigate: Inspect OS termination records, allocation peaks, retained sessions, cache reservations, and crash reports. Linear cache growth is a possibility for full attention, not proof of this failure. A watchdog, unsupported operation, or leak needs a different repair.

Change after diagnosis: Bound sessions and context, reduce a measured allocation, or select a suitable trained GQA/hybrid architecture. Cache quantization requires runtime support and quality checks. Arbitrarily evicting full-attention KV changes what the model can attend to; trimming history and rebuilding a prompt is different from native sliding-window attention.

4. Throughput collapses halfway through a work shift

Symptom: The model performs well in morning testing, but field technicians complain that answers slow to an unusable crawl during afternoon inspections.

Investigate: Correlate workload and rate with thermal state, ambient temperature, charging, background jobs, and power. A decline alone cannot identify the thermal governor as the cause.

Change after diagnosis: Evaluate smaller candidates, supported kernels, shorter outputs, or workload duty-cycle changes. Size alone does not ensure power below 3 W. A generated-token cap must leave room for the required answer and needs quality evaluation.

5. Over-the-air update bricks offline functionality

Symptom: Technicians update their handhelds before heading into the field, only to find the app crashes on launch when cell connectivity drops.

Investigate: Reproduce with networking disabled. Check missing files, lazy downloads, tokenizer/template mismatches, modality assets, authorization refresh, and incompatible runtime versions. A missing-token exception is only one possibility.

Change after diagnosis: Stage and authenticate a complete compatible bundle, test first launch offline, and exercise interrupted activation and recovery. Retain an approved offline recovery path.

Core takeaways for edge engineering

Deploying language models to the edge requires balancing physical hardware limits, architectural compactness, and software safety gates:

1. Write the task contract: Compare lookup, classification, extraction, and generation against the same real job. Hypothetical sizing can reject candidates; it cannot qualify them.
2. Separate training signals from truth: Soft targets expose teacher preferences, not guaranteed correctness. Exact forward-KL gradients have a 1/tau factor; the familiar squared scaling is a high-temperature approximation. Verifiers cover their contracts.
3. Budget the actual architecture: Stored bytes, active weight traffic, KV/recurrent state, modality components, pools, and temporary allocations are different quantities. Effective parameter labels cannot establish fit.
4. Inspect the installed runtime: A CPU graph round-trip does not prove mobile lowering. Partition boundaries can help or hurt; measure placement, output quality, and critical-path time.
5. Measure a shift, not only a burst: Track visible-answer latency, generation work, memory peaks, thermal behavior, and energy for the whole product under representative conditions.
6. Bind release and privacy decisions to real inputs: Enforce complete-payload transmission policy and authenticated, compatible, recoverable updates. A route string, local weights, or a valid signature alone cannot establish those properties.

Follow-up architectural questions

• Assume 34 GB/s and one active-weight stream per step. A 15-step/s target permits at most about 2.27 GB for that traffic before other costs. What evidence would invalidate this assumption, and why does it not select a quantization method by itself?
• An NPU delegate reports 98% operator coverage but misses latency. Which partition boundaries occur per generated token, and how will you separate synchronization, conversion, copies, and compute?
• Derive the exact forward-KL gradient and its high-temperature approximation. What changes when teacher and student token events cannot be directly aligned?
• A validly signed update changes the tokenizer and prompt template. What cache/session transition, offline test, and recovery evidence must precede activation?

Evaluation rubric

• Differentiate parameter count from memory bandwidth and thermal dissipation constraints.
• Derive the Hinton knowledge distillation loss and explain the mathematical role of temperature scaling and gradient compensation.
• Calculate KV cache memory savings from Grouped-Query Attention.
• Diagnose and resolve mobile NPU operator fallbacks and CPU memory thrashing.
• Implement fail-closed routing and cryptographic verification for edge model artifacts.