A local Qwen3.6 box should read a repo, explain a failing test, and suggest a patch without sending source to a hosted model. The first attempt often fails before it answers anything: the file fills the GPU, a projector is downloaded by surprise, or the context setting reserves more memory than expected.
That failure is useful if we make it legible. Use llama.cpp as the runtime and Unsloth GGUF as the model source. You'll download a prebuilt GGUF, not install the Unsloth Python package or export weights yourself. Pick one artifact, make one short text request, and keep that run as the control while you change one variable at a time.
Qwen positions Qwen3.6 around stability, agentic coding, repository-level reasoning, and thinking preservation.[1] Ollama exposes convenient tags, while the Unsloth GGUF path keeps the exact file, quant, llama.cpp flags, and multi-token prediction (MTP) variant visible.[2][3][4] That extra visibility matters when a run fails: we can identify which bytes and which runtime produced the result.
Pick the right first model
Two open-weight Qwen3.6 models make useful starting points. Qwen3.6-27B is dense, so every parameter path runs for every token. Qwen3.6-35B-A3B is a Mixture-of-Experts (MoE) model, so a router activates only part of its expert capacity for each token. Qwen released 35B-A3B on April 16, 2026 and 27B on April 22, 2026. Both official model repositories use Apache 2.0.[1][5][6]
Before choosing, ask which experiment you want. A dense 27B run gives us a simpler first debug loop. The 35B-A3B run tests sparse routing and agentic coding, but its 35B total parameters still occupy memory even though only 3B are active per token. The model card lists 256 experts, with 8 routed experts and 1 shared expert active during inference.[5] Mixture of Experts Architecture explains that active compute and weight residency are different budgets.
Both choices also need memory for weights, KV cache, runtime buffers, and any vision projector file. Inference: TTFT, TPS & KV Cache goes deeper on cache sizing, but we can make the first decision with Unsloth's planning ladder below.
| Starting lane | Choose it for | Keep fixed during first test |
|---|---|---|
| 27B dense Q3 | Lowest-memory dense control; Unsloth plans 15 GB total memory | Exact GGUF pin, 8K context, one request |
| 27B dense Q4 | Dense quality comparison; Unsloth plans 18 GB total memory | Same prompt pack and server build |
| 35B-A3B MoE Q4 | Deliberate sparse-model test; Unsloth plans 23 GB total memory | Same context, sampling, and acceptance tests |
| Ollama Qwen3.6 tag | The shortest setup rather than exact artifact control | Explicit tag and observed context allocation |
First-run rule: Change one variable at a time. Prove the plain text endpoint before adding longer context, MTP, vision, or parallel slots.
Unsloth defines those estimates as total available memory across RAM and VRAM, or unified memory. They aren't guaranteed GPU-fit numbers.[7] MTP uses a separate ladder: 27B needs about 16 GB at 3-bit and 19 GB at 4-bit; 35B-A3B needs about 18 GB at 3-bit and 24 GB at 4-bit. Those rounded plans include neither a promise that one GPU holds the whole file nor a fixed context budget. KV cache and runtime buffers still sit on top.
A 14.47 GB Q3 file on a 16 GB card leaves little room for cache and buffers, so expect CPU offload or choose a smaller quant if the measured run doesn't fit. The same warning applies to a 23 GB Q4 plan on a 24 GB card. The file size is the first subtraction, not the final capacity test.
If you only want the shortest setup, Ollama lists qwen3.6, qwen3.6:27b, and qwen3.6:35b tags with 256K context and text plus image input. Its current library page shows qwen3.6:27b at 18 GB and qwen3.6:35b at 23 GB, so those tags aren't the Unsloth 15 GB Q3 pin.[2] Run ollama run qwen3.6:27b for that path. This tutorial continues with llama.cpp because a repository revision, filename, and checksum make the artifact easier to replay.
Read the quant name
GGUF is the model-file format used by llama.cpp and many local apps. These Unsloth repositories already contain exported and quantized files. You're selecting an artifact, not converting the original BF16 checkpoint during setup.
Suppose the file is Qwen3.6-35B-A3B-UD-Q4_K_M.gguf. Read it left to right: model family, parameter shape, Unsloth Dynamic label, rough bit width, llama.cpp quant tier, and file format. 35B-A3B tells you which weights you need. Q4 narrows the size and quality trade-off. The complete filename tells the downloader exactly which artifact to fetch.
Unsloth's Dynamic 2.0 docs explain that newer GGUFs use model-specific layer choices and calibration data instead of applying one uniform quantization recipe everywhere.[8] Model Quantization: GPTQ, AWQ & GGUF covers how GGUF differs from GPU-oriented GPTQ and AWQ workflows.
Quantization stores weights with fewer bits. Smaller quants need less memory, but task quality can change. Q4 is a quant family, not a promise that every weight uses exactly four bits or that every Q4 file has the same size. Compare exact filenames and byte sizes before downloading:
Q3: first 16 GB dense testQ4: default useful local coding targetQ5orQ6: try when you have headroomQ8or BF16 (bfloat16): quality comparison or server-class memory
Unsloth's Qwen3.6 guide gives this total-memory ladder:[7]
| Model | 3-bit | 4-bit |
|---|---|---|
| 27B | 15 GB | 18 GB |
| 27B MTP | 16 GB | 19 GB |
| 35B-A3B | 17 GB | 23 GB |
| 35B-A3B MTP | 18 GB | 24 GB |
Those are planning numbers. Runtime allocations can still push a nominal fit over capacity.
Run one local endpoint
llama.cpp is the local inference engine behind many GGUF workflows. Qwen lists it as a supported path for Qwen3.6 text and vision models, and the Unsloth model cards show direct llama-server -hf ... commands.[1][3]
There are two ways to get a model into the server. -hf resolves a repository and quant for you, which is convenient for a quick trial. It also follows the repository's current state and, on current llama.cpp builds, downloads an mmproj when one is available unless you pass --no-mmproj. The command below uses --model instead: the repository revision, filename, and SHA-256 are explicit, and the first run stays text-only.
The Linux/NVIDIA setup below pins a llama.cpp commit, an Unsloth repository revision, a GGUF filename, and a SHA-256 verified on August 25, 2026.[9][4] The pin is intentional. Reproduce this control first, then try a newer llama.cpp commit as a separate variable if support or performance changes.
1git clone https://github.com/ggml-org/llama.cpp
2git -C llama.cpp checkout 8e7f22b67ef4667b4ddd50230771287f328cfb3f
3
4cmake -S llama.cpp -B llama.cpp/build \
5 -DBUILD_SHARED_LIBS=OFF \
6 -DGGML_CUDA=ON
7
8cmake --build llama.cpp/build \
9 --config Release \
10 -j \
11 --target llama-server llama-cli
12
13./llama.cpp/build/bin/llama-server --version
14
15MODEL_REPO="unsloth/Qwen3.6-27B-GGUF"
16MODEL_REV="82d411acf4a06cfb8d9b073a5211bf410bfc29bf"
17MODEL_FILE="Qwen3.6-27B-UD-Q3_K_XL.gguf"
18MODEL_SHA256="cff4a2da6b5350a53f57dc6f798516e0195816f90c196c3f7bd3fd55556632dd"
19MODEL_DIR="models/qwen36-27b-q3"
20
21mkdir -p "$MODEL_DIR"
22curl -fL --retry 5 --retry-delay 2 -C - \
23 -o "$MODEL_DIR/$MODEL_FILE" \
24 "https://huggingface.co/$MODEL_REPO/resolve/$MODEL_REV/$MODEL_FILE?download=true"
25
26printf '%s %s\n' "$MODEL_SHA256" "$MODEL_DIR/$MODEL_FILE" \
27 | sha256sum -c -
28
29./llama.cpp/build/bin/llama-server \
30 --model "$MODEL_DIR/$MODEL_FILE" \
31 --no-mmproj \
32 --alias local-qwen36 \
33 --ctx-size 8192 \
34 --n-gpu-layers all \
35 --host 127.0.0.1 \
36 --port 8080The download is about 14.47 decimal GB. curl -C - resumes a partial file, while sha256sum -c rejects a corrupt or different artifact. --no-mmproj keeps this control run text-only. The alias matches the smoke-test "model" field below.
On Apple Silicon, build the same pinned commit with -DGGML_CUDA=OFF; Metal is enabled by default. Use shasum -a 256 in place of sha256sum. For CPU-only inference, also set --n-gpu-layers 0. Qwen lists MLX as another supported Apple Silicon path, but switching runtimes creates a different benchmark.[1]
The plain Q3 command gives us a baseline. If it loads and answers the smoke request, vary one server setting at a time:
- Lowest-memory dense control:
Qwen3.6-27B-UD-Q3_K_XL.ggufat 8192 context. - 27B quality test:
unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XLat 8192 context. The matching Q4 file is about 17.61 decimal GB, so it doesn't fit a 16 GB card without offload. - 35B-A3B MoE test:
unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_Mat 8192 first, then 32768.
If output is repeated symbols or broken formatting, check the selected file, checksum, and llama.cpp version before blaming the model. Qwen3.6 support has moved quickly, so an older binary is one possible variable, not a reason to change three settings at once.
Verify artifact and host state
The pinned workflow downloads before server startup, so model-transfer time can't be mistaken for load time. Preserve the startup log until it shows the selected file, embedded chat template, memory placement, context allocation, and listening address.
Run the server command a second time before measuring cold model load. Did it reuse the file? If another large transfer starts, inspect the repository selector, cache location, and partial download before changing the model. A repeatable load is part of the control, not a minor convenience.
For strict artifact control, resolve the repository revision and filename before the benchmark, then store both in the run receipt. A quant label describes a file family; the repository revision records which published state supplied it.
Keep separate disk and runtime checks:
1df -h .
2du -sh models/qwen36-27b-q3
3git -C llama.cpp rev-parse HEAD
4./llama.cpp/build/bin/llama-server --version
5sha256sum models/qwen36-27b-q3/Qwen3.6-27B-UD-Q3_K_XL.ggufFree disk space answers whether the artifact can be stored. It doesn't answer whether weights, KV cache, and buffers fit during inference. Keep those questions separate when a download succeeds but the first request fails.
Point tools at http://127.0.0.1:8080/v1 and test one completion from your normal SDK or client. OpenAI-compatible SDKs usually expect an API key even for local servers. Use a placeholder such as local, set the base URL to the local endpoint, and keep the first prompt short enough to separate runtime problems from task complexity.
Verify the endpoint contract
Start with a health request, then send one non-streaming chat completion:
1curl -fsS http://127.0.0.1:8080/health
2
3curl -fsS http://127.0.0.1:8080/v1/chat/completions \
4 -H 'Content-Type: application/json' \
5 -H 'Authorization: Bearer local' \
6 -d '{
7 "model": "local-qwen36",
8 "messages": [
9 {"role": "user", "content": "Reply with exactly: endpoint-ready"}
10 ],
11 "temperature": 0,
12 "max_tokens": 16,
13 "stream": false
14 }' > qwen36-smoke.jsonThat smoke request answers one narrow question: "is the server up, and can it satisfy a tiny output contract?" Qwen3.6 generates thinking content by default and doesn't support the older /think or /nothink soft switch.[6] For a non-thinking client, restart the server with --chat-template-kwargs '{"enable_thinking":false}' and use the non-thinking sampling row below. To keep prior reasoning traces in later turns, Unsloth documents a separate preserve_thinking kwarg.[7]
The smoke request keeps temperature at zero only to make a tiny endpoint assertion. It isn't the coding baseline. First prove transport and response shape; then choose a thinking mode and sampling policy for the task.
The exact response ID and usage counts vary. Validate stable structure and requested content:
1jq -e '
2 (.choices | length) == 1
3 and .choices[0].message.role == "assistant"
4 and (.choices[0].message.content | contains("endpoint-ready"))
5' qwen36-smoke.jsonIf health passes but completion fails, read the server log before changing the model. If completion returns malformed content, repeat with the same prompt after confirming the llama.cpp build, selected GGUF, and chat template.
Add MTP after the plain run works
Multi-token prediction (MTP) lets a draft path propose more than one future token while the main model verifies them.[10] Qwen trained MTP into Qwen3.6, and Unsloth publishes dedicated MTP GGUF repositories for both 27B and 35B-A3B.[5][6][11][12]
The useful prediction is simple: if the draft tokens match what the main model would have produced, decode needs fewer full passes. Prompt ingestion, model weights, and context memory don't disappear. Unsloth reports roughly 1.4x to 2.2x faster generation without an accuracy change in its own benchmarks, while also warning that the best draft count depends on hardware.[7][10]
Use --spec-draft-n-max 2 as a starting point, not a guarantee. Unsloth's current guide says to try values from 1 through 6, and the pinned llama.cpp build defaults this option to 3. A wider draft can use more memory or produce fewer accepted tokens, so the only useful speedup is the one that survives your output checks.
Stop the plain server, download the matching MTP artifact, verify it, then launch with the MTP flags:
1MTP_REPO="unsloth/Qwen3.6-27B-MTP-GGUF"
2MTP_REV="5cb35eb3dcbf52dbce5f87dbc64df6aaffadcace"
3MTP_FILE="Qwen3.6-27B-UD-Q3_K_XL.gguf"
4MTP_SHA256="661a031ced3e048eeb6d831f8dccaa2020dee480fdd15d9fc9f0fc588e0851f6"
5MTP_DIR="models/qwen36-27b-q3-mtp"
6
7mkdir -p "$MTP_DIR"
8curl -fL --retry 5 --retry-delay 2 -C - \
9 -o "$MTP_DIR/$MTP_FILE" \
10 "https://huggingface.co/$MTP_REPO/resolve/$MTP_REV/$MTP_FILE?download=true"
11
12printf '%s %s\n' "$MTP_SHA256" "$MTP_DIR/$MTP_FILE" \
13 | sha256sum -c -
14
15./llama.cpp/build/bin/llama-server \
16 --model "$MTP_DIR/$MTP_FILE" \
17 --no-mmproj \
18 --alias local-qwen36 \
19 --ctx-size 8192 \
20 --n-gpu-layers all \
21 --flash-attn on \
22 --parallel 1 \
23 --spec-type draft-mtp \
24 --spec-draft-n-max 2 \
25 --host 127.0.0.1 \
26 --port 8080The MTP file is about 14.79 decimal GB, roughly 315 MB larger than the plain Q3 artifact. The current MTP cards say -np > 1 and --mmproj aren't supported with MTP.[11][12] Keep --parallel 1 and --no-mmproj. Unsloth's general how-to-run examples include --mmproj, so follow the more specific MTP card for this combination and recheck it after the article's refresh date.
The same cards' -hf snippets can still auto-pull a projector. The pinned --model path plus --no-mmproj avoids that extra file.
Compare MTP against a control
Measure plain decoding first, then change only the model repository and MTP flags. Keep these fixed:
- llama.cpp commit and build flags
- base model and quant tier
- prompt tokens and output cap
- context length and parallel slots
- sampling settings and warm-up policy
- hardware, power mode, and background load
Run a prompt pack with short and sustained generations. Record time to first token separately from generated-token throughput because speculation can help decode without helping prompt ingestion. Also record accepted draft tokens when the runtime reports them. Low acceptance can erase the expected speedup, especially on the MoE path.
MTP passes only when output still clears the same task checks. Compare tests, JSON validity, and required facts before comparing speed. A faster run that breaks the answer contract is a regression.
Store medians from repeated warm runs, plus every failed run. Don't delete out-of-memory or malformed-output trials: they define the usable operating envelope.
Keep context small at first
What changes after the plain 8K run? The context setting changes allocation even when the first prompt is short. Qwen3.6 model cards list a native context length of 262,144 tokens and an extended path up to 1,010,000 tokens with RoPE scaling (YaRN).[5][6] That's a model capability, not a laptop startup setting. The cards also warn that static YaRN can hurt shorter texts, so don't turn on the 1M path for an 8K coding loop.
Qwen3.6 uses a hybrid stack. The 27B card lists 16 Gated Attention layers and 48 Gated DeltaNet layers; the 35B-A3B card lists 10 Gated Attention layers and 30 Gated DeltaNet layers.[5][6] For a first estimate, count conventional attention KV entries from those Gated Attention layers. The recurrent state, runtime buffers, and backend choices still make total memory machine-specific, so a simple full-attention formula can overstate or understate the running process.
A model can load weights successfully, then fail on the first long request because context allocation pushes it over the edge. Treat context as measured capacity, not a number you can infer from download size.
Context ladder
Start at 8K, then double to 16K and 32K while recording peak memory and the same prompt result. Move to 64K, 128K, or 262K only when a measured workload needs it and each earlier step fits. Task labels such as "agentic coding" don't determine a safe context allocation.
The Qwen model cards advise keeping at least 128K when possible to preserve thinking capabilities, but also recommend reducing context if you hit out-of-memory errors.[5][6] For a beginner setup, read 128K as a target to grow toward, not a first command.
Add vision only after text works
The official Qwen3.6 model cards describe both 27B and 35B-A3B as causal language models with vision encoders.[5][6] The Unsloth GGUF repositories include mmproj (multimodal projector) files that connect image features to the language model.[3][4] Keep the first run text-only:
- Run text-only
llama-server - Test
/v1/chat/completions - Increase context or switch model pins only after it stays stable
- Download
mmproj-F16.gguffrom the same repository revision, verify its checksum, and pass its local path with--mmproj
Vision adds another file, more memory pressure, and more syntax variation across runtimes.
For the pinned 27B repository revision above, mmproj-F16.gguf is 927.61 MB and its SHA-256 is eacf610d1ee4bd5ed0197a0777dd8f4fceb8eefa27009067c7d496cb68fbde45. A projector from another model or repository revision isn't an interchangeable vision adapter.
Modality rule: Keep text-only as the control run. Add the matching vision projector only to the plain GGUF path after the same server build, GGUF pin, and context length are stable. Current Qwen3.6 MTP cards don't support combining MTP with
--mmproj.
Measure memory on the running process
Planning numbers tell you which download to try. Measure the running process before increasing context or concurrency.
On NVIDIA, sample device memory while the server loads and while a request generates:
1nvidia-smi \
2 --query-compute-apps=pid,process_name,used_memory \
3 --format=csv \
4 --loop=1In another terminal, send the fixed smoke request, then a longer benchmark request. Record idle loaded-model memory, peak prompt-processing memory, and peak generation memory. Use ps -o pid,rss,command -C llama-server to capture host resident memory when CPU offload or unified memory matters.
Repeat after each context increase. Change parallel slots only after the single-request curve is stable. This produces a machine-specific capacity table:
| Run | Context | Parallel slots | Peak device memory | Peak host memory | Result |
|---|---|---|---|---|---|
| Control | 8K | 1 | measured | measured | pass or fail |
| Context step | 32K | 1 | measured | measured | pass or fail |
| Plain concurrency step | 32K | 2 | measured | measured | pass or fail |
| MTP control | 8K | 1 | measured | measured | pass or fail |
Don't copy memory values from another machine into this table. Backend, quant, offload, context, and runtime build all change the result.
Plan the memory footprint
Before launching the server, turn the published ladder into a rough budget. Unsloth's numbers are rounded total-memory plans, not loader guarantees.[7] Qwen3.6 is hybrid, so the cache estimate below counts only conventional FP16 attention K/V entries; recurrent state and backend buffers still need measurement. conservative_total_gb adds explicit cache and projector terms to the published plan as a guardrail, not as a promise that the process will use exactly that amount.
The projector entries use rounded current F16 file sizes from the Unsloth repositories: 0.93 GB for 27B and 0.90 GB for 35B-A3B.[4][3]
1def qwen36_memory_plan_gb(
2 model: str,
3 quant: str,
4 context_tokens: int = 8192,
5 slots: int = 1,
6 with_projector: bool = False,
7 with_mtp: bool = False,
8) -> dict[str, float | str | bool]:
9 published_plans = {
10 ("27B", "Q3", False): 15.0,
11 ("27B", "Q4", False): 18.0,
12 ("35B-A3B", "Q3", False): 17.0,
13 ("35B-A3B", "Q4", False): 23.0,
14 ("27B", "Q3", True): 16.0,
15 ("27B", "Q4", True): 19.0,
16 ("35B-A3B", "Q3", True): 18.0,
17 ("35B-A3B", "Q4", True): 24.0,
18 }
19 architecture = {
20 "27B": {"attention_layers": 16, "kv_heads": 4, "head_dim": 256},
21 "35B-A3B": {"attention_layers": 10, "kv_heads": 2, "head_dim": 256},
22 }
23 projector_gb = {"27B": 0.93, "35B-A3B": 0.90}[model] if with_projector else 0.0
24 published_total_gb = published_plans[(model, quant, with_mtp)]
25
26 # Conventional FP16 attention KV cache: K and V use 2 bytes each.
27 attention = architecture[model]
28 bytes_per_token = (
29 attention["attention_layers"]
30 * attention["kv_heads"]
31 * attention["head_dim"]
32 * 4
33 )
34 kv_cache_bytes = context_tokens * bytes_per_token * slots
35 kv_cache_gb = round(kv_cache_bytes / (1000**3), 2)
36 conservative_total_gb = round(published_total_gb + projector_gb + kv_cache_gb, 2)
37
38 return {
39 "config": f"Qwen3.6-{model}-{quant}{'+MTP' if with_mtp else ''}{'+Vision' if with_projector else ''}",
40 "published_total_gb": published_total_gb,
41 "projector_gb": projector_gb,
42 "attention_kv_cache_gb": kv_cache_gb,
43 "conservative_total_gb": conservative_total_gb,
44 "fits_16gb_gpu": conservative_total_gb <= 16.0,
45 "fits_24gb_gpu": conservative_total_gb <= 24.0,
46 }
47
48q3_control = qwen36_memory_plan_gb("27B", "Q3", context_tokens=8192)
49q3_vision = qwen36_memory_plan_gb("27B", "Q3", context_tokens=8192, with_projector=True)
50q4_dense = qwen36_memory_plan_gb("27B", "Q4", context_tokens=8192)
51moe_q4 = qwen36_memory_plan_gb("35B-A3B", "Q4", context_tokens=8192)
52
53for plan in [q3_control, q3_vision, q4_dense, moe_q4]:
54 print(
55 f"{plan['config']}: {plan['conservative_total_gb']} GB conservative "
56 f"(published plan: {plan['published_total_gb']} GB, "
57 f"projector: {plan['projector_gb']} GB, attention KV: {plan['attention_kv_cache_gb']} GB) | "
58 f"16 GB fit: {plan['fits_16gb_gpu']} | 24 GB fit: {plan['fits_24gb_gpu']}"
59 )
60
61assert q3_control["published_total_gb"] == 15.0
62assert q3_vision["projector_gb"] == 0.93
63assert q3_control["attention_kv_cache_gb"] == 0.54
64assert q4_dense["fits_16gb_gpu"] is False
65assert moe_q4["fits_24gb_gpu"] is True1Qwen3.6-27B-Q3: 15.54 GB conservative (published plan: 15.0 GB, projector: 0.0 GB, attention KV: 0.54 GB) | 16 GB fit: True | 24 GB fit: True
2Qwen3.6-27B-Q3+Vision: 16.47 GB conservative (published plan: 15.0 GB, projector: 0.93 GB, attention KV: 0.54 GB) | 16 GB fit: False | 24 GB fit: True
3Qwen3.6-27B-Q4: 18.54 GB conservative (published plan: 18.0 GB, projector: 0.0 GB, attention KV: 0.54 GB) | 16 GB fit: False | 24 GB fit: True
4Qwen3.6-35B-A3B-Q4: 23.17 GB conservative (published plan: 23.0 GB, projector: 0.0 GB, attention KV: 0.17 GB) | 16 GB fit: False | 24 GB fit: TrueTune for coding
For coding, use narrow prompts with visible acceptance criteria. Name the function or file, list allowed inputs, state the output shape, and ask for code plus one short explanation. Then run tests.
Sampling baseline
Qwen and Unsloth publish separate sampling settings for thinking and non-thinking modes.[6][5][7]
| Mode | Temperature | top_p | top_k | min_p | Presence penalty | Repeat penalty |
|---|---|---|---|---|---|---|
| Thinking, general | 1.0 | 0.95 | 20 | 0.0 | 0.0 | 1.0 or disabled |
| Thinking, precise coding | 0.6 | 0.95 | 20 | 0.0 | 0.0 | 1.0 or disabled |
| Non-thinking | 0.7 | 0.8 | 20 | 0.0 | 1.5 | 1.0 or disabled |
Set the output cap from the task rather than copying a large default. Keep one baseline prompt pack, raw responses, and task checks so quant, context, and MTP comparisons use the same work.
Local iteration is cheap, but code still needs a verifier: your test suite.
When things go wrong
| Symptom | Inspect first | Safe next action |
|---|---|---|
| Out of memory during load or first prompt | Device and host peaks, context, GPU placement | Return to 8K, one slot, and text-only; then choose a smaller quant or measured CPU offload |
| Slow generation | CPU offload, device utilization, prompt versus decode timing | Keep context fixed, reduce offload with a smaller artifact, and compare generated tokens per second |
| Broken or gibberish output | Quant family, CUDA toolkit, pinned llama.cpp commit, exact checksum, context, cache types | If the file is IQ3_S, IQ3_XXS, or IQ2_M on CUDA 13.2, move to a supported toolkit; otherwise retest the verified file and build before changing context |
| MTP has no speedup | Same base/quant, accepted draft tokens, one-slot control | Return to plain GGUF when median decode doesn't improve; expect a smaller gain on 35B-A3B than on 27B |
| Image input fails | Plain text control, projector repo and revision | Remove --mmproj, recover text, then re-add the matching projector on the non-MTP path |
| Almost-valid tool JSON | Embedded chat template, client schema, raw response | Test one tool call, validate externally, and don't treat parseable-looking text as a successful call |
⚠️ CUDA 13.2: Unsloth's tracked issue names IQ3_S, IQ3_XXS, and IQ2_M quants as affected by gibberish on CUDA 13.2; the issue is now marked fixed. If one of those files fails, try CUDA 12.8 or 13.0, or a current build with the fix.[13] Unsloth's guide still advises avoiding 13.2, so verify current guidance for other quant families before changing toolkits.[7]
Keep a reproducible receipt
Start with one known-good text run. Record the llama.cpp commit, build flags, model repository revision and filename, context, sampling, GPU placement, prompt-pack revision, raw response, and memory measurements. That receipt makes dense-versus-MoE, Q3-versus-Q4, and plain-versus-MTP comparisons reproducible.
Once the control run is stable, change one dimension at a time: MTP, context, vision, then concurrency. If a step fails, return to the control and compare logs before changing the model.
That discipline generalizes beyond this model. Local LLM Deployment connects fit, quantization, and serving placement; Speculative Decoding explains why an MTP draft path can help decode without making prefill or memory pressure disappear.