Model Quantization: GPTQ, AWQ & GGUF

The previous chapter treated every AI-generated kernel as an untrusted candidate until correctness, memory safety, and benchmark gates agreed. Quantized model artifacts need the same discipline: fewer bytes help only when encoding, calibration, metadata, and runtime kernel agree.

Before adding another GPU, ask whether fewer bytes would solve the fit problem. Three names will recur: GPTQ is Hessian-aware post-training quantization, AWQ is activation-aware weight quantization, and GGUF is a portable local inference container format.

Use dense sibling Qwen3.6-27B as the running example. Every parameter is resident, so the memory math stays honest. Its BF16 weights need about 54 GB, which can fit in raw H100 capacity but not on a typical 24 GB workstation before KV cache or runtime buffers.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}

Model parallelism can shard those bytes. Quantization tries to pack them before sharding.

Quantization maps selected tensors to a smaller set of values plus scale and, sometimes, zero-point metadata. The storage cut is predictable. The quality hit depends on which values the coarser grid distorts.

For speed, ask what moves through the bottleneck. Decode is often limited by memory bandwidth, so 4-bit weights reduce raw weight traffic to about one quarter of FP16. That isn't a promise of 4x end-to-end throughput: kernels still unpack values, accumulate in higher precision, and do non-weight work.

A 4-bit label can leave other tensors and operations at higher precision. W4A16 stores 4-bit weights with 16-bit activations, W8A8 quantizes both sides of a matmul, and KV-cache precision is a separate runtime choice. Start with this contract, then compare GPTQ, AWQ, and GGUF by mechanism and hardware path.

GPTQ and AWQ compress a model after training. Quantization-aware training (QAT) takes a different path: it simulates low-precision rounding during training so the weights can adapt before the final artifact is exported.

What problem does quantization solve first: speed, memory, or accuracy?

Answer

Memory and bandwidth come first. Quantization stores weights with fewer bits, which reduces weight memory and memory traffic. Speed may improve when the runtime is bandwidth-bound and has good low-bit kernels, but it's not guaranteed.

---

Coarser numbers, smaller tensors

Quantization replaces many floating-point values with a finite grid. A precise ruler can record 3.7 cm, 4.2 cm, and 12.1 cm; a whole-centimeter ruler records 4 cm, 4 cm, and 12 cm. The values need fewer bits to represent, but nearby measurements collapse onto the same mark. A model's quality depends on which weights share those marks.

A tiny worked example

Predict before calculating: with $w = 0.73$ and scale $s = 0.1$, which integer code is stored, and what value comes back? Each integer step represents $0.1$ in the original space.

1. Divide: $0.73 / 0.1 = 7.3$
2. Round to the nearest integer: $7$
3. Store the integer $q = 7$

During inference you reverse the process:

$\hat{w} = s \cdot q = 0.1 \cdot 7 = 0.7$

The fine grid stores $0.7$, not $0.73$, for an absolute error of $0.03$. With the coarser scale $s = 0.5$, the same weight becomes $q = \text{round}(0.73 / 0.5) = 1$ and rebuilds as $0.5$. Scale is grid spacing, not a quality score by itself: smaller spacing helps only when the range and metadata still represent the values that drive the output.

In the tiny example, why is scale choice the main quality decision?

Answer

The scale decides how large each integer step is in the original value space. A small scale reconstructs 0.73 as 0.7, while a coarse scale reconstructs it as 0.5, creating much larger error.

The quantization formula

Take the original weight $w$, divide by the scale $s$ to express it in integer-sized steps, shift by the zero point $z$, round to the nearest integer, and clamp to the representable range. That's how a high-precision weight becomes an INT8 (8-bit integer) or INT4 value.

$q = \text{clip}\left(\text{round}\left(\frac{w}{s}\right) + z,\ q_{\min},\ q_{\max}\right)$

• $w$ is the original floating-point weight
• $q$ is the stored integer
• $s$ is the scale factor
• $z$ is the zero point
• $q_{\min}, q_{\max}$ are the integer limits (for example, 0 to 15 for unsigned 4-bit)

Dequantization reverses the process during inference:

$\hat{w} = s \cdot (q - z)$

Reading the formula

Subtract the zero point from the stored integer, then multiply by the scale. The result $\hat{w}$ is only an approximation of the original weight because rounding already threw away information.

What information is lost during quantization, and what do scale and zero point preserve?

Answer

Rounding loses exact floating-point values. The scale preserves approximate spacing between values, and the zero point lets an integer value represent real zero or shift the represented range.

Run the same five weights through a signed INT4 round-trip. Predict which value will use the largest code, then check whether 0.73 comes back as 0.686. This is a toy error measurement, not a model-quality score.

The largest magnitude, 2.40, sets the group scale, so 0.73 maps to code 2 and returns as $2 \times 0.34286 = 0.686$. The group shares one grid; a different grouping or scale would change every reconstruction.

Symmetric vs. asymmetric

Suppose an activation buffer has 16 markings. Centering those markings around zero is convenient for values with positive and negative signs. If the buffer only sees values from 0.0 to 3.0, spending half the markings below zero leaves fewer positive levels.

• Symmetric: center the 16 markings around zero, such as -8 to +7. This is simple and hardware-friendly, but it wastes range if the values are skewed.
• Asymmetric: move the markings to the actual range you observed. This uses the integer range more efficiently, but it requires the extra zero-point offset.

More formally:

1. Symmetric quantization: uses $z = 0$ and maps weights around zero. This is common for weight-only LLM kernels because the math is simpler.
2. Asymmetric quantization: uses a non-zero $z$ to better cover distributions that aren't centered at zero. This is common for activations and some weight formats.

Most LLM weight tensors are roughly zero-centered, so symmetric or near-symmetric per-group quantization often works well for weights. Activations are harder because a few channels can contain very large outliers. Techniques like SmoothQuant^{[2]Reference 2SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.https://arxiv.org/abs/2211.10438} make activation quantization easier by shifting some of that difficulty into the weights.

Why are activations usually harder to quantize than weights?

Answer

Weights are often roughly zero-centered and stable after training. Activations depend on input data and can contain large channel outliers, so the same low-bit range can distort important values more easily.

Predict the step size before running the example: which scheme gives a finer grid for activations from 0.0 to 3.0, signed symmetric or unsigned asymmetric?

The output makes the trade concrete: asymmetric scaling uses a 0.200 step, while symmetric scaling uses about 0.429 because its signed range reserves levels for negative values. The zero point costs metadata, but it lets the grid follow a shifted range.

The pipeline figure uses the same five-weight group as the round-trip above: snap to the nearest 4-bit level, pack two codes per byte, then rebuild with the stored scale.

Memory savings

Memory is the first measurable gate. Smaller weights can make a model fit, leave room for KV cache, or buy a larger batch. Cost and throughput still depend on the runtime and workload.

The memory table counts weights only. Actual runtime memory is higher because you still pay for activations, the KV cache, scale metadata, and framework overhead.

Gemma 4 12B is the dense contrast: its BF16 weights already sit near a 24 GB card, while Qwen3.6-27B still needs a lower-bit artifact on that same workstation.^{[3]Reference 3Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}

Those numbers are idealized weight math in decimal GB (1e9 bytes). The same INT4 ideal is about 12.6 GiB (1024^3 bytes). Pick one base per worksheet and stick to it; 13.5 decimal GB ≈ 12.6 GiB is the same storage, not two competing answers. Real packed formats are larger when they also store per-group scales, alignment, or tensors kept at higher precision. For a GGUF artifact, inspect the selected quantization type and actual file/runtime footprint rather than assuming the ideal 13.5 GB number.^{[4]Reference 4llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}

Weights are only one lever. KV cache and activations can bind capacity after weights fit:

Because decode is often weight-bandwidth bound, shrinking weights can also speed up inference. Raw weight traffic falls with bit width, but observed throughput depends on the kernel, batch size, sequence lengths, concurrency, and unpacking work.

Treat every speedup as an experiment. Record accelerator and count, runtime and kernel versions, model and workload, batch and prompt/decode lengths, concurrency, warmup and steady-state timing, precision and algorithm contract, exact FP16 baseline, and task-quality checks. Without that packet, "4-bit is 4x faster" is not reproducible.

Why is ideal INT4 storage for Qwen3.6-27B about 13.5 GB, but real files can be larger?

Answer

INT4 means 0.5 bytes per weight, so 27B weights are about 13.5 GB in ideal math. Real formats also store scales, group metadata, alignment padding, and sometimes mixed tensor types.

A group size of 128 with one FP16 scale per group adds about 0.42 GB on this 27B model. That's why a real Q4 file isn't the ideal 13.5 GB.

---

Sizing exercise

Try this before moving on. You have a workstation with one NVIDIA RTX 4060 (8 GB VRAM) and you want to run Qwen3.6-27B entirely on the GPU. The model has roughly 27 billion dense parameters.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}

1. How much VRAM would the model need in FP16?
2. How much would it need at INT4?
3. Can you run it on the 4060?

Use this tiny calculator to check the arithmetic without any framework overhead:

Solution

1. FP16/BF16: $27 \times 2 = 54$ GB. The model doesn't fit.
2. INT4: $27 \times 0.5 = 13.5$ GB. The model still doesn't fit on an 8 GB card.
3. You can't run the full model on the GPU alone. One local option is a GGUF artifact loaded by a runtime with heavy CPU offload. The other options are a larger GPU or a smaller model. The INT4 size is a huge improvement, but it doesn't remove the memory budget.

This worksheet answers the fit question, not the serving question. Parameters $\times$ bytes per parameter gives the ideal weight footprint; activations, KV cache, metadata, and runtime buffers still need space. If INT4 is larger than usable VRAM, a faster kernel won't make the artifact GPU-resident.

Why can't an 8 GB RTX 4060 run Qwen3.6-27B entirely on GPU even at INT4?

Answer

INT4 weights alone are about 13.5 GB, before KV cache, activations, metadata, and runtime overhead. An 8 GB GPU can only run it with heavy CPU offload, a smaller model, or different hardware.

---

Weight-only vs. weight-activation quantization

Bits on the weights and bits on the activations answer different memory and kernel questions. Name both sides before comparing artifacts.

Weight-only quantization is what GPTQ and AWQ target. The stored weights are low precision, while optimized kernels unpack or dequantize them during computation and accumulate with higher-precision activations or accumulators. W4A16 means 4-bit stored weights with 16-bit activations.

Weight-activation quantization pushes both sides of the matmul down, for example W8A8 or W4A8. This can reduce activation traffic and use integer kernels, but this path is harder because activation distributions are input-dependent and can contain outliers. SmoothQuant moves part of that difficulty into the weights with an offline, mathematically equivalent rescaling step, making W8A8 easier to calibrate.^{[2]Reference 2SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.https://arxiv.org/abs/2211.10438}

GPTQ and AWQ are weight-only methods, so activations and the KV cache remain higher precision. Lower-bit activation paths such as W4A4 need separate hardware, kernel, and quality validation. KV-cache dtype is another independent runtime setting. FP8 is an 8-bit floating-point path; it pays off only when the accelerator and kernel execute it natively, not because its file label is smaller.^{[5]Reference 5FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}

What does W4A16 mean, and why doesn't it imply that every computation is 4-bit?

Answer

W4A16 means 4-bit stored weights and 16-bit activations. A fused kernel can unpack weights while computing with higher-precision activations and accumulation, so it isn't the same contract as W4A4.

Common mistake: W4A16 doesn't guarantee a 4x end-to-end speedup. Low-bit weights reduce raw weight traffic, but kernel overhead and non-weight work remain, and the workload may not be bandwidth-bound. The guaranteed first-order gain is smaller stored weights, not a fixed tokens-per-second multiplier.

This next calculation is deliberately illustrative, not a benchmark. Plug in the 27B weight sizes and hold other work at 3 ms. The traffic drop is 4x; the step speedup is not.

The toy model spends 54 ms moving FP16 weights and 13.5 ms moving INT4 weights at the chosen 1,000 GB/s rate. The fixed 3 ms leaves a 3.45x step ratio instead of 4x. Real kernels add their own unpacking, cache, scheduling, and non-weight costs.

---

GPTQ (post-training quantization)

Weight errors don't cost the same

Take a layer with only two weights: $w_1 = 1.2$ and $w_2 = 0.8$. A naive quantizer might round both toward the nearest integer, turning them into $1.0$ and $1.0$. The first weight lost $0.2$, the second gained $0.2$. Before deciding whether that is acceptable, ask which input direction matters more.

If the calibration data shows that $w_1$ is multiplied by large activations and $w_2$ by small ones, the $-0.2$ error on $w_1$ hurts the output far more than the $+0.2$ error on $w_2$. GPTQ uses that activation geometry to compensate: it can quantize $w_1$ more carefully or adjust $w_2$ to cancel some damage. It preserves layer output on representative inputs, not equal numeric distance for every weight.

Algorithm: minimize output error with curvature information

After GPTQ rounds one set of weights, it updates remaining floating-point weights to compensate for the induced layer-output error. Plain independent rounding skips that correction. Sequential compensation makes processing order and approximate curvature information part of the method.

GPTQ^{[6]Reference 6GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323} is a one-shot post-training quantization method based on approximate second-order information. It builds on the layer-wise Optimal Brain Quantization (OBQ) solver, which quantizes weights one at a time and updates remaining weights to minimize layer output error. GPTQ makes that idea practical for billion-parameter models with a fixed order and lazy batched updates.

For a weight row $w$ and calibration activations $X$, the teaching objective is output fidelity rather than raw weight distance:

$$\begin{aligned} \hat{w} &= \arg\min_{\tilde{w} \in \mathcal{Q}} \|wX - \tilde{w}X\|_2^2 \\ &\approx \arg\min_{\tilde{w} \in \mathcal{Q}} (w - \tilde{w}) H (w - \tilde{w})^T \\ H &\approx XX^T \end{aligned}$$

Here $H \approx XX^T$ is a teaching shorthand for the curvature signal. The original derivation uses a regularized inverse-Hessian update, which is what lets GPTQ decide how to redistribute a rounding residual.^{[6]Reference 6GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323}

Reading the formula

The Hessian approximation $H$ tells GPTQ which input directions matter most on the calibration set. A small error on an unimportant direction is cheap; the same numeric error on a frequently used direction is expensive. In the GPTQ paper's comparisons, this output-aware correction improved over independent rounding at the same bit width. The gap depends on model, calibration data, group size, and runtime.^{[6]Reference 6GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323}

The GPTQ workflow looks like this:

1. Collect representative activations from a calibration set such as C4^{[7]Reference 7Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683}.
2. Approximate $H \approx XX^T$ for each linear layer.
3. Quantize the weights sequentially while using an approximate inverse Hessian to compensate the remaining floating-point weights.
4. Pack the result into a low-bit format that an inference kernel can consume efficiently.

The original GPTQ paper reports quantizing 175B-class models, including OPT-175B and BLOOM-176B, in about four GPU-hours while preserving strong accuracy at 3-bit and 4-bit settings. Its calibration used 128 random 2048-token segments from C4.^{[6]Reference 6GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323[7]Reference 7Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683}

Today's software stack still needs its own check. Current Hugging Face Transformers routes GPTQ through GPT-QModel (gptqmodel), while AutoGPTQ is no longer the supported backend. Marlin is a 4-bit CUDA kernel path for Ampere-class GPUs such as A100. Check current docs before pinning a backend or kernel name.^{[8]Reference 8GPTQhttps://huggingface.co/docs/transformers/main/en/quantization/gptq}

That reconstruction step is also a risk. The AWQ paper discusses how calibration repair can overfit its calibration set and distort behavior on other domains or modalities.^{[9]Reference 9AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978} If your serving stack has a fast GPTQ kernel, evaluate target tasks, not only calibration-distribution loss.

Take the two-weight layer from the earlier example. After rounding $w_1$, GPTQ updates $w_2$ so the layer output on the calibration activations stays closer to the original.

The output shows what compensation buys in this toy layer: naive rounding misses by 0.8, while updating the second weight reduces absolute error to 0.2. GPTQ still has to quantize the repaired weights, so this is an intuition for the update, not a full implementation.

What makes GPTQ different from plain round-to-nearest quantization?

Answer

GPTQ uses representative activations and approximate curvature information to minimize layer output error. It cares about which weight errors change real model outputs, not which individual weights stay numerically close.

The same 0.2 rounding error is cheap on a quiet input direction and expensive on a busy one. The Hessian proxy is how GPTQ scores that difference.

Both directions have the same absolute error, but the first contributes 25 times as much curvature-weighted cost. GPTQ therefore spends effort protecting or compensating the first direction. Calibration examples need to resemble the inputs the layer will see.

Granularity: per-tensor, per-channel, per-group

A naive quantizer uses one scale for the entire tensor. This is per-tensor quantization. It's cheap, but one large outlier can ruin the precision of everything else.

Many LLM quantizers use finer granularity:

Per-group quantization is the common compromise for 4-bit LLM inference. Smaller groups usually improve fidelity, but they also require storing more scale metadata and may reduce kernel efficiency.

Why is per-group quantization a common 4-bit compromise?

Answer

One global scale is too sensitive to outliers, while one scale per tiny slice adds overhead. Per-group scales give local accuracy while keeping metadata and kernel complexity manageable.

---

AWQ (activation-aware weight quantization)

High-activation channels need more protection

GPTQ repairs layer outputs after rounding. AWQ takes a lighter path: find the channels that would amplify rounding error, then spend more of the integer range there before quantization. Predict the important channel before reading the example: the one with the larger activation, or the one with the larger weight?

Equal numeric weight errors don't produce equal output errors. If one input channel carries activations 1,000 times larger than another, the same rounding error on its weight column can contribute roughly 1,000 times more error to the matrix product. AWQ uses activation statistics to identify and protect those salient channels.

AWQ^{[9]Reference 9AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978} starts from the observation that activation magnitudes aren't evenly distributed. A small fraction of channels carry disproportionately large activations. If their weight columns are quantized poorly, downstream matmul error is amplified. The AWQ paper reports that protecting only about 1% of salient weights can greatly reduce quantization error.^{[9]Reference 9AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}

What does AWQ mean by a salient weight channel?

Answer

It's a weight channel connected to unusually large or important activations. Errors in those weights get amplified during the matmul, so AWQ protects them with activation-aware rescaling before quantization.

The same 0.1 weight error on a channel with activation 100 dominates a channel with activation 0.1. Protecting the busy channel cuts the output-error proxy from 10.01 to 2.01.

The proxy drops from 10.01 to 2.01 because the larger activation dominates the sum. This isolates AWQ's selection signal rather than measuring perplexity or latency.

Why protecting a few weights matters

The same two-weight layer makes the mechanism visible: $w_1$ is multiplied by $100$, while $w_2$ is multiplied by $0.1$. A $0.1$ rounding error becomes a $10$ unit output error on the first channel but only $0.01$ on the second. AWQ identifies these high-traffic channels and rescales their columns so limited integer range goes where it matters.

Algorithm

AWQ doesn't rebuild the entire weight matrix the way GPTQ does. Instead, it uses an equivalent rescaling trick:

$W x = W \cdot \mathrm{diag}(s) \cdot \mathrm{diag}(s)^{-1} x \approx Q\!\left(W \cdot \mathrm{diag}(s)\right) \cdot \mathrm{diag}(s)^{-1} x$

Reading the formula

Multiply important weight columns by a scaling vector $s$ before quantization so they occupy more of the available integer range. Then divide the corresponding activation channels by the same factor. The floating-point computation stays equivalent, but the quantizer now spends more precision on the columns that matter most.

The practical workflow is:

1. Run representative inputs through the model and collect activation statistics.
2. Identify salient channels with unusually large activation magnitude.
3. Search for scaling factors that reduce the quantization error on those channels.
4. Quantize the rescaled weights into a hardware-friendly 4-bit format.

AWQ artifacts are typically produced offline and loaded by a serving runtime that understands the artifact's quantization metadata, such as group size and zero-point policy. Loader and kernel support varies by runtime version, so verify the chosen artifact/runtime pair before benchmarking.^{[10]Reference 10AWQhttps://huggingface.co/docs/transformers/main/en/quantization/awq}

Compared with GPTQ, AWQ is lighter-weight because it avoids GPTQ's reconstruction step. The AWQ paper reports better generalization on instruction-tuned checkpoints and other modalities in its evaluations, where calibration repair can overfit.^{[9]Reference 9AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}

That is an algorithm result, not a universal speed claim. Latency still depends on the runtime, kernel, workload, and artifact.^{[10]Reference 10AWQhttps://huggingface.co/docs/transformers/main/en/quantization/awq}

How should you choose between GPTQ and AWQ when both are available?

Answer

Start from runtime support and target workload. GPTQ is strong when your serving stack has optimized GPTQ kernels. AWQ is often attractive for 4-bit instruction-tuned models because it protects activation-sensitive channels with a lighter offline workflow.

---

GGUF (llama.cpp format)

GGUF is a container, not a quantizer

The 8 GB RTX 4060 couldn't hold Qwen3.6-27B even at INT4. GPU-oriented GPTQ and AWQ paths usually target a GPU-resident packed artifact. GGUF answers a different question: how do you package tensors so a local runtime can spill layers into system RAM and still run?

GGUF is a file format, not a quantization algorithm. It's the container used by the ggml / llama.cpp ecosystem for local inference.^{[4]Reference 4llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp} Published Qwen3.6-27B GGUF trees exist; you still have to name the tensor type inside the file, not only "GGUF."^{[11]Reference 11Qwen3.6-27B-GGUFhttps://huggingface.co/unsloth/Qwen3.6-27B-GGUF}

The file bundles the tensors with the metadata a local runtime needs:

• tokenizer and vocabulary information
• architecture metadata and tensor shapes
• tensor-by-tensor quantization types inside one portable file
• enough information to run on CPU or to choose partial GPU offload

The file doesn't place layers by itself. If the full model doesn't fit in VRAM, a llama.cpp-style runtime can keep some layers on the GPU and spill the rest to system memory.

Why is GGUF not the same kind of thing as GPTQ or AWQ?

Answer

GPTQ and AWQ are quantization algorithms. GGUF is a container format that stores tensors, metadata, tokenizer information, and chosen low-bit tensor types for local runtimes.

The file's tensor type determines its stored bytes; GGUF itself leaves layer placement to the runtime. Qwen3.6-27B has 64 layers.

For an even-size placement sketch, give an 8 GB card a 20% reserve and a 0.5 GB runtime reserve. That leaves about 5.9 GB for weights. Ideal INT4 weights are 13.5 GB, so only some layers fit on GPU.

Quantization families inside GGUF

GGUF can store several quantization families. The format doesn't force one specific quantizer.

Q4_K_M is a commonly encountered local-inference candidate, but the right choice depends on target model, quality check, and hardware. If the full model fits in VRAM, benchmark GPU-oriented GPTQ or AWQ artifacts against the local runtime; if partial offload is required, GGUF is a useful packaging option.

iMatrix quantization

Importance-matrix quantization uses representative text to estimate which directions are expensive to distort. That extra signal lets IQ formats spend precision where it buys the most quality. Conceptually, it fills the same role as calibration in GPTQ and AWQ: representative data tells the quantizer what errors matter most.

The commands below illustrate a llama.cpp-style conversion and quantization path. Binary names and supported quant types can change, so check the installed revision's documentation before running it.^{[4]Reference 4llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}

When does iMatrix-style GGUF quantization help most?

Answer

It helps when you're squeezing below comfortable 4-bit settings or using IQ formats. Representative text tells the quantizer which directions matter, similar to calibration data in GPTQ and AWQ.

---

Beyond weight-only: FP8 and KV cache quantization

The low-bit artifacts above shrink static model weights. Once those weights fit, check whether the next bottleneck is the KV cache.

FP8 sits adjacent to GPTQ, AWQ, and GGUF rather than replacing them. It's an 8-bit floating-point format that becomes attractive when the serving hardware has native FP8 kernels.^{[5]Reference 5FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}

FP8 has two common encodings:^{[5]Reference 5FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}

• E4M3: more mantissa precision, less dynamic range
• E5M2: less mantissa precision, more dynamic range

Unlike 4-bit weight-only methods, FP8 is often a milder first test when you want a smaller accuracy-memory tradeoff and the accelerator is built to run FP8 directly.

Quantizing the KV cache is a separate lever. Weight quantization shrinks static model weights. KV-cache quantization shrinks attention state that grows with sequence length. vLLM exposes this as a runtime kv_cache_dtype choice, including FP8 KV variants, after you measure whether long-context concurrency is the resource that binds.^{[12]Reference 12Quantized KV Cachehttps://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/}

Qwen3.6-27B is a hybrid: 64 layers, but only 16 of them are full gated-attention layers (4 KV heads, head dim 256). The sketch below sizes FP16 KV for those 16 layers. Gated DeltaNet state is extra and isn't in this KV formula.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}

Why can KV-cache quantization matter after weight quantization succeeds?

Answer

Weight quantization shrinks static model weights. KV cache grows with active sequence length and concurrency, so after weights fit, long-context serving may be limited by cached attention state instead.

The next sketch uses those 16 full-attention layers at batch 32 and 8K context. Ideal INT4 weights are about 12.6 GiB; that KV cache is already 16 GiB.

The sketch reverses the bottleneck: INT4 weights take about 12.6 GiB, while the selected FP16 KV state takes 16.0 GiB. Once weights fit, reducing KV bytes or limiting context and concurrency can matter more than changing GPTQ to AWQ.

---

Comparison

Compare strategies by the constraint that failed first, not by a universal "best" label. Fit, task quality, kernel support, and placement can point to different artifacts.

The FP8 row is a candidate, not a promise. Native support and measured quality and memory still need to beat an INT4 path for the target workload.

AWQ's paper reports strong 4-bit results by protecting activation-sensitive channels.^{[9]Reference 9AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978} GPTQ remains relevant when a runtime or kernel stack supports its packed artifacts efficiently.^{[6]Reference 6GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323} Those papers describe quantization behavior, not your serving latency. On GPU servers, benchmark the exact low-bit kernel path. For local or partial-offload deployments, benchmark the GGUF runtime and placement plan rather than assuming a format name decides performance.

What is the fastest decision rule for GPTQ, AWQ, and GGUF?

Answer

If the model fits fully on GPU, start with AWQ or GPTQ based on runtime kernel support. If a local deployment needs CPU/GPU split placement or a portable artifact, start with GGUF and a compatible local runtime.

---

Diagnose before changing bit width

The fit math tells you whether an artifact can load. Calibration, kernel efficiency, and task quality need separate checks. Use the failure signature to choose the next experiment.

Treating all quantizers as equivalent

• Symptom: You choose "4-bit" from a model hub without checking whether it's GPTQ, AWQ, GGUF, or a runtime quantization path.

• Cause: Bit width describes storage size, not calibration method, tensor layout, kernel support, offload behavior, or quality profile.

• Fix: Name the artifact and the runtime together: "AWQ on vLLM," "GPTQ on Transformers," or "Q4_K_M GGUF on llama.cpp." Then test that exact pair.

The calibration trap

• Symptom: Your quantized French incident-assistant model speaks gibberish, even though the English version quantized fine.

• Cause: GPTQ and AWQ both rely on calibration data to understand which weights matter. If you use English Wikipedia to quantize a model trained on French incident runbooks, the activation statistics are wrong and the quantizer throws away precision in the wrong places.

• Fix: Use calibration text that matches the target domain and language, then verify quality on held-out target tasks.

Confusing weight-only and full quantization

• Symptom: A design doc claims GPTQ or AWQ makes the entire model 4-bit.

• Cause: GPTQ and AWQ are weight-only methods. Activations usually stay at FP16 or BF16, and accumulation happens in higher precision.

• Fix: Write the precision contract explicitly. W4A16 means 4-bit stored weights and 16-bit activations, not full W4A4 inference.

The speed fallacy

• Symptom: You quantize to 4-bit expecting a 4x speedup, but tokens per second barely improve.

• Cause: 4-bit weights save memory bandwidth, but the kernel still has to dequantize them into higher precision before the matrix multiply. If the dequantization code path is slow or the GPU isn't memory-bound to begin with, the speedup shrinks.

• Fix: Measure end-to-end tokens per second on your exact hardware and batch size. Bandwidth savings are real, but they only translate to speed when the runtime is optimized for your GPU.

Treating perplexity as sufficient

• Symptom: The quantized model still chats politely, but it hallucinates deploy status or generates invalid JSON for your incident API.

• Cause: Perplexity on held-out text is a fast sanity check, but a model can show only a small perplexity increase while regressing sharply on structured tasks like code generation or multi-step reasoning. Relying only on perplexity (not ignoring it) is the failure mode.

• Fix: Always pair perplexity with task-specific benchmarks. For an incident-assistant model, run your own production eval set that includes the exact output formats the model must produce.

Confusing GGUF with the quantizer

• Symptom: Someone says "we used GGUF quantization" as if that fully specifies the quality and runtime behavior.

• Cause: GGUF is the container. Q4_K_M, IQ4_XS, Q5_K_M, and related tensor types describe the actual low-bit encoding inside the file.

• Fix: Report both: "GGUF Q4_K_M with 20 GPU layers," "GGUF IQ4_XS with iMatrix," or another concrete artifact/runtime pairing.

---

How to evaluate a quantized model

Perplexity is a useful first sanity check, but it isn't enough to approve a serving artifact. A quantized developer assistant can chat politely while regressing on code patches, JSON tool calls, or incident-status answers. Test the formats the route must emit, not general text alone.

GPTQ and AWQ both report that 4-bit weight-only quantization can preserve language-modeling quality on large models, while more aggressive bit widths degrade more sharply.^{[6]Reference 6GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323[9]Reference 9AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}

The next figure is an illustrative 24 GB workstation gate for Qwen3.6-27B, not a measured leaderboard. Before reading it, predict which candidates clear both gates. FP16 misses fit, 3-bit misses a 90% task floor, and AWQ-4bit is the only path that clears both in this sketch.

Three practical rules:

• GPTQ and AWQ results support testing 4-bit weight-only artifacts before pushing to more aggressive bit widths.^{[6]Reference 6GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323[9]Reference 9AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}
• Task-specific regressions can't be inferred from a generic quality score; reasoning, structured output, and tool-use tasks need their own gates.
• Group size, calibration data, and the serving kernel can matter as much as the headline format name.

Why isn't perplexity enough to approve a quantized model?

Answer

Perplexity checks broad next-token drift, but structured outputs, code, math, tool calls, and domain-specific decisions can regress without a dramatic perplexity change. Pair it with task evals and serving metrics.

Apply the same two-gate rule to the 24 GB workstation: illustrative task scores, not a measured Qwen leaderboard. FP16 misses the weight budget. 3-bit misses the 90% floor.

The approval code applies two explicit gates: at least 90% task accuracy and at most 24 GB of weights. AWQ-4bit is the only illustrative candidate that clears both; its 49 ms value is still a placeholder until the performance evidence packet is measured on target hardware.

---

Start from the measured bottleneck

Match the artifact to the pressure

Once fit and task quality are gates, ask which resource failed first. A VRAM miss, a free FP8 path, a growing KV cache, and CPU/RAM spill are different experiments, even when each option is called quantization.

Practical rule: if the whole model fits on GPU, benchmark an AWQ or GPTQ path supported by your runtime. If a local deployment needs CPU/GPU split placement, benchmark a GGUF artifact with a compatible runtime. If native FP8 kernels or KV growth are the measured limit, test those levers directly.

For a local workstation GPU and a model too large for full VRAM residency, why is GGUF a reasonable first artifact?

Answer

GGUF works well with local runtimes that can split layers between GPU and system RAM. It may be slower than full-GPU kernels, but it lets the model run when a GPU-only AWQ or GPTQ deployment doesn't fit.

---

Keep the artifact contract explicit

The decision can fit in five lines:

• Quantization stores weights with fewer bits using scales and, sometimes, zero points. The first win is lower memory and raw weight traffic.
• GPTQ uses representative activations and approximate second-order information to minimize output error during post-training quantization.^{[6]Reference 6GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323}
• AWQ identifies activation-sensitive channels and rescales them so the quantizer spends precision where it matters most.^{[9]Reference 9AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}
• GGUF is a portable container for local inference that can store many ggml quantization types; a compatible runtime such as llama.cpp chooses CPU/GPU-offload placement.^{[4]Reference 4llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}
• Benchmark the full artifact/runtime/workload contract. AWQ or GPTQ fit GPU-resident paths; GGUF fits portability or partial-offload paths.