Model Quantization: GPTQ, AWQ & GGUF

Model parallelism splits a large model across several accelerators. Quantization starts earlier: what if each weight used fewer bytes before you split anything?

Model quantization stores selected tensors with fewer bits so large models can require less memory and potentially less bandwidth. Imagine serving Qwen3.6-27B for a developer-assistant route. Its dense weights need about 54 GB in FP16 or BF16, which can fit raw H100 capacity but not a typical 24 GB workstation budget before KV cache or runtime buffers.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Model parallelism can split those bytes across accelerators. Quantization asks whether you can shrink the bytes first.

The technique works like replacing a precise ruler with a coarser grid: storage reduction is predictable, while error depends on which values the grid distorts. A weight-only method packs approximate low-bit values plus metadata. LLM generation is often memory-bandwidth bound, so 4-bit weights also cut raw weight traffic to about one quarter of FP16. Real speedups are smaller than 4x because kernels still have to unpack values and accumulate in higher precision. Compare GPTQ, a Hessian-aware post-training quantization method; AWQ, activation-aware weight quantization; and GGUF, a portable local inference container format. For each one, focus on the serving tradeoff: what changes, which runtime can use the artifact, and what quality you must measure before deployment.

What problem does quantization solve first: speed, memory, or accuracy?

Answer

Memory and bandwidth come first. Quantization stores weights with fewer bits, which reduces weight memory and memory traffic. Speed may improve when the runtime is bandwidth-bound and has good low-bit kernels, but it's not guaranteed.

---

What is quantization?

Quantization resembles rounding measurements to coarser marks. A precise ruler has millimeter markings, so you can measure 3.7 mm, 4.2 mm, 12.1 mm. A coarse ruler only has centimeter markings, so those same values become 4 cm, 4 cm, 12 cm. You lose some precision, but the ruler is simpler and cheaper. Quantization does this to every number in a neural network.

A tiny worked example

Before the general formula, walk through one weight by hand. Suppose a weight has the value $w = 0.73$ and you choose a scale of $s = 0.1$. That means each integer step represents $0.1$ in the original space.

1. Divide: $0.73 / 0.1 = 7.3$
2. Round to the nearest integer: $7$
3. Store the integer $q = 7$

During inference you reverse the process:

$\hat{w} = s \cdot q = 0.1 \cdot 7 = 0.7$

The stored value is $0.7$, not $0.73$. The error is tiny ($0.03$), and when this happens across billions of weights the model usually stays useful. If you had chosen a coarser scale of $s = 0.5$, the same weight would become $q = \text{round}(0.73 / 0.5) = 1$ and the reconstructed value would be $0.5$, which is a much larger error. The art of quantization is choosing the right scale so the errors stay small where they matter most.

In the tiny example, why is scale choice the main quality decision?

Answer

The scale decides how large each integer step is in the original value space. A small scale reconstructs 0.73 as 0.7, while a coarse scale reconstructs it as 0.5, creating much larger error.

At the simplest level, quantization maps a floating-point weight to a smaller integer range using a scale and, in some schemes, a zero point:

$q = \text{clip}\left(\text{round}\left(\frac{w}{s}\right) + z,\ q_{\min},\ q_{\max}\right)$

The quantization formula

Take the original weight $w$, divide by the scale $s$ to express it in "integer-sized steps," shift by the zero point $z$, round to the nearest integer, and clamp to the representable range. That's how a high-precision weight becomes an INT8 or INT4 value.

• $w$ is the original floating-point weight
• $q$ is the stored integer
• $s$ is the scale factor
• $z$ is the zero point
• $q_{\min}, q_{\max}$ are the integer limits (for example, 0 to 15 for unsigned 4-bit)

Dequantization reverses the process during inference:

$\hat{w} = s \cdot (q - z)$

Reading the formula

Subtract the zero point from the stored integer, then multiply by the scale. The result $\hat{w}$ is only an approximation of the original weight because rounding already threw away information.

What information is lost during quantization, and what do scale and zero point preserve?

Answer

Rounding loses exact floating-point values. The scale preserves approximate spacing between values, and the zero point lets an integer value represent real zero or shift the represented range.

Symmetric vs. asymmetric

A numeric sensor display that stores activation readings with only 16 possible markings has two strategies available:

• Symmetric: center the 16 markings around zero, such as -8 to +7. This is simple and hardware-friendly, but it wastes range if the values are skewed.
• Asymmetric: move the markings to the actual range you observed. This uses the integer range more efficiently, but it requires the extra zero-point offset.

More formally:

1. Symmetric quantization: uses $z = 0$ and maps weights around zero. This is common for weight-only LLM kernels because the math is simpler.
2. Asymmetric quantization: uses a non-zero $z$ to better cover distributions that aren't centered at zero. This is common for activations and some weight formats.

Most LLM weight tensors are roughly zero-centered, so symmetric or near-symmetric per-group quantization often works well for weights. Activations are harder because a few channels can contain very large outliers. Techniques like SmoothQuant^{[2]Reference 2SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.https://arxiv.org/abs/2211.10438} make activation quantization easier by shifting some of that difficulty into the weights.

Why are activations usually harder to quantize than weights?

Answer

Weights are often roughly zero-centered and stable after training. Activations depend on input data and can contain large channel outliers, so the same low-bit range can distort important values more easily.

The pipeline figure walks through the basic quantization flow: start with high-precision weights, estimate the statistics needed to compute scales, then pack the results into a low-bit representation plus metadata.

Memory savings

The most obvious benefit of quantization is memory reduction. That directly translates to lower serving cost, larger batch sizes, and the ability to fit bigger models on smaller devices.

The memory table counts weights only. Actual runtime memory is higher because you still pay for activations, the KV cache, scale metadata, and framework overhead.

Those numbers are idealized weight math in decimal GB (1e9 bytes). The same INT4 ideal is about 12.6 GiB (1024^3 bytes). Pick one base per worksheet and stick to it; 13.5 decimal GB ≈ 12.6 GiB is the same storage, not two competing answers. Real packed formats are larger when they also store per-group scales, alignment, or tensors kept at higher precision. For a GGUF artifact, inspect the selected quantization type and actual file/runtime footprint rather than assuming the ideal 13.5 GB number.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}

Weights are only one lever. KV cache and activations can bind capacity after weights fit:

Because LLM generation is often weight-bandwidth bound, shrinking the weights can also speed up inference. The raw bandwidth demand drops almost linearly with bit width, although the observed throughput gain depends on the kernel, batch size, and how much extra work the runtime does to unpack the weights.

Why is ideal INT4 storage for Qwen3.6-27B about 13.5 GB, but real files can be larger?

Answer

INT4 means 0.5 bytes per weight, so 27B weights are about 13.5 GB in ideal math. Real formats also store scales, group metadata, alignment padding, and sometimes mixed tensor types.

---

Sizing exercise

Try this before moving on. You have a workstation with one NVIDIA RTX 4060 (8 GB VRAM) and you want to run Qwen3.6-27B entirely on the GPU. The model has roughly 27 billion dense parameters.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}

1. How much VRAM would the model need in FP16?
2. How much would it need at INT4?
3. Can you run it on the 4060?

Use this tiny calculator to check the arithmetic without any framework overhead:

Solution

1. FP16/BF16: $27 \times 2 = 54$ GB. The model doesn't fit.
2. INT4: $27 \times 0.5 = 13.5$ GB. The model still doesn't fit on an 8 GB card.
3. You can't run the full model on the GPU alone. One local option is a GGUF artifact loaded by a runtime with heavy CPU offload. The other options are a larger GPU or a smaller model. The INT4 size is a huge improvement, but it doesn't remove the memory budget.

This is the calculation you should do before choosing a quantization strategy. The formula is simple: parameters $\times$ bytes per parameter = weight footprint. Runtime memory is higher because of activations, KV cache, and metadata.

Why can't an 8 GB RTX 4060 run Qwen3.6-27B entirely on GPU even at INT4?

Answer

INT4 weights alone are about 13.5 GB, before KV cache, activations, metadata, and runtime overhead. An 8 GB GPU can only run it with heavy CPU offload, a smaller model, or different hardware.

---

Weight-only vs. weight-activation quantization

Two broad approaches to quantization exist, and confusing them causes a lot of interview mistakes.

Weight-only quantization is what GPTQ and AWQ target. The stored weights are low precision, while optimized kernels unpack or dequantize low-bit values during computation and accumulate with higher-precision activations or accumulators. Notation like W4A16 means 4-bit stored weights with 16-bit activations.

Weight-activation quantization pushes both sides of the matmul down, for example W8A8 or W4A8. This can be faster and more memory-efficient, but it's harder because activation distributions are spikier and less stable than weight distributions. Techniques like SmoothQuant^{[2]Reference 2SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.https://arxiv.org/abs/2211.10438} exist specifically to make weight-activation quantization practical.

GPTQ and AWQ are weight-only methods. They capture large weight-memory savings without requiring equally low-bit activations. Lower-bit activation paths such as W4A4 need separate hardware, kernel, and quality validation.

What does W4A16 mean, and why doesn't it imply that every computation is 4-bit?

Answer

W4A16 means 4-bit stored weights and 16-bit activations. A fused kernel can unpack weights while computing with higher-precision activations and accumulation, so it is not the same contract as W4A4.

Common mistake: Candidates often claim W4A16 quantization gives a 4x end-to-end speedup. It doesn't guarantee that. Low-bit weights reduce raw weight traffic, but kernel overhead and non-weight work remain, and the workload may not be bandwidth-bound. The guaranteed first-order gain is smaller stored weights, not a fixed tokens-per-second multiplier.

---

GPTQ (post-training quantization)

The intuition: weight errors have unequal cost

For simplicity, take a layer with only two weights: $w_1 = 1.2$ and $w_2 = 0.8$. A naive quantizer might round both toward the nearest integer, turning them into $1.0$ and $1.0$. The first weight lost $0.2$, the second gained $0.2$. The total output change depends on how the layer uses those weights.

If the calibration data shows that $w_1$ is multiplied by large activations and $w_2$ by small ones, the $-0.2$ error on $w_1$ hurts the output far more than the $+0.2$ error on $w_2$. GPTQ notices this through the Hessian approximation and compensates: it might quantize $w_1$ more carefully, or adjust $w_2$ in the opposite direction to cancel some of the damage. GPTQ isn't trying to make every weight close to its original value. It tries to keep the layer's output on real data as close as possible to the original output.

Algorithm: minimize output error with curvature information

After GPTQ rounds one set of weights, it updates the remaining floating-point weights to compensate for the induced layer-output error. Plain independent rounding never makes that correction. Sequential compensation is why processing order and approximate curvature information matter.

GPTQ^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323} is a one-shot post-training quantization method based on approximate second-order information. It builds on the layer-wise Optimal Brain Quantization (OBQ) solver, which quantizes weights one at a time and updates the remaining weights to minimize the layer's output error. GPTQ makes that idea fast enough for billion-parameter models by quantizing weights in a fixed order and using lazy batched updates. The key objective isn't "make the quantized weights numerically close to the originals." It's "make the layer output on real activations stay close to the original output." For a weight row $w$ and calibration activations $X$, GPTQ approximates:

$$\begin{aligned} \hat{w} &= \arg\min_{\tilde{w} \in \mathcal{Q}} \|wX - \tilde{w}X\|_2^2 \\ &\approx \arg\min_{\tilde{w} \in \mathcal{Q}} (w - \tilde{w})^T H (w - \tilde{w}) \\ H &\approx XX^T \end{aligned}$$

Reading the formula

The Hessian approximation $H$ tells GPTQ which input directions matter most on the calibration set. A small error on an unimportant direction is cheap. The same numeric error on a frequently used direction is expensive. That's why GPTQ usually beats plain round-to-nearest quantization at the same bit width.

In practice, GPTQ looks like this:

1. Collect representative activations from a calibration set such as C4^{[5]Reference 5Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683}.
2. Approximate $H \approx XX^T$ for each linear layer.
3. Quantize the weights sequentially while using an approximate inverse Hessian to compensate the remaining floating-point weights.
4. Pack the result into a low-bit format that an inference kernel can consume efficiently.

The original paper reports quantizing 175B-class models (OPT-175B and BLOOM-176B) in about four GPU-hours while preserving strong accuracy at 3-bit and 4-bit settings.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323}

What makes GPTQ different from plain round-to-nearest quantization?

Answer

GPTQ uses representative activations and approximate curvature information to minimize layer output error. It cares about which weight errors change real model outputs, not which individual weights stay numerically close.

The configuration sketch below shows the shape of a transformers GPTQ workflow. Backend packages and supported arguments change over time, so verify current library documentation and evaluate the resulting artifact before deployment.^{[6]Reference 6GPTQhttps://huggingface.co/docs/transformers/main/en/quantization/gptq}

Granularity: per-tensor, per-channel, per-group

A naive quantizer uses one scale for the entire tensor. This is per-tensor quantization. It's cheap, but one large outlier can ruin the precision of everything else.

Modern LLM quantizers almost always use finer granularity:

Per-group quantization is the common compromise for 4-bit LLM inference. Smaller groups usually improve fidelity, but they also require storing more scale metadata and may reduce kernel efficiency.

Why is per-group quantization a common 4-bit compromise?

Answer

One global scale is too sensitive to outliers, while one scale per tiny slice adds overhead. Per-group scales give local accuracy while keeping metadata and kernel complexity manageable.

---

AWQ (activation-aware weight quantization)

High-activation channels need more protection

Equal numeric weight errors don't produce equal output errors. If one input channel carries activations 1,000 times larger than another, the same rounding error on its weight column can contribute roughly 1,000 times more error to the matrix product. AWQ uses activation statistics to identify and protect those salient channels.

AWQ^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978} starts from the observation that activation magnitudes aren't evenly distributed. A small fraction of channels carry disproportionately large activations. If the corresponding weight columns are quantized poorly, the downstream matmul error gets amplified. The AWQ paper reports that protecting only about 1% of salient weights can greatly reduce quantization error.^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}

What does AWQ mean by a salient weight channel?

Answer

It's a weight channel connected to unusually large or important activations. Errors in those weights get amplified during the matmul, so AWQ protects them with activation-aware rescaling before quantization.

Why protecting a few weights matters

Now make the two-weight layer's activation pattern extreme: $w_1$ is almost always multiplied by $100$, while $w_2$ is multiplied by $0.1$. A $0.1$ rounding error on $w_1$ becomes a $10$ unit output error, while the same error on $w_2$ becomes only $0.01$ units. AWQ identifies these "high-traffic" channels and rescales them so the quantizer spends more of its limited integer range on the weights that matter most.

Algorithm

AWQ doesn't rebuild the entire weight matrix the way GPTQ does. Instead, it uses an equivalent rescaling trick:

$W x = W \cdot \mathrm{diag}(s) \cdot \mathrm{diag}(s)^{-1} x \approx Q\!\left(W \cdot \mathrm{diag}(s)\right) \cdot \mathrm{diag}(s)^{-1} x$

Reading the formula

Multiply important weight columns by a scaling vector $s$ before quantization so they occupy more of the available integer range. Then divide the corresponding activation channels by the same factor. The floating-point computation stays equivalent, but the quantizer now spends more precision on the columns that matter most.

The practical workflow is:

1. Run representative inputs through the model and collect activation statistics.
2. Identify salient channels with unusually large activation magnitude.
3. Search for scaling factors that reduce the quantization error on those channels.
4. Quantize the rescaled weights into a hardware-friendly 4-bit format.

AWQ artifacts are typically produced offline and loaded by a serving runtime that understands the artifact's quantization metadata, such as group size and zero-point policy. Loader and kernel support varies by runtime version, so verify the chosen artifact/runtime pair before benchmarking.^{[8]Reference 8AWQhttps://huggingface.co/docs/transformers/main/en/quantization/awq}

Compared with GPTQ, AWQ is lighter-weight because it avoids GPTQ's reconstruction step. It often works especially well on instruction-tuned checkpoints, but the final speed and latency picture still depends on the runtime and kernel implementation.^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978[8]Reference 8AWQhttps://huggingface.co/docs/transformers/main/en/quantization/awq}

How should you choose between GPTQ and AWQ when both are available?

Answer

Start from runtime support and target workload. GPTQ is strong when your serving stack has optimized GPTQ kernels. AWQ is often attractive for 4-bit instruction-tuned models because it protects activation-sensitive channels with a lighter offline workflow.

---

GGUF (llama.cpp format)

What is GGUF?

GGUF is a file format, not a quantization algorithm. It's the container format used by the ggml / llama.cpp ecosystem for local inference.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}

GPTQ and AWQ mainly answer the question "How should I quantize the weights?" GGUF answers a different question: "How should I package model tensors and metadata so local runtimes can load and run them efficiently?"

GGUF matters because it bundles the tensors with the metadata needed to run them:

• tokenizer and vocabulary information
• architecture metadata and tensor shapes
• tensor-by-tensor quantization types inside one portable file
• enough information for compatible local runtimes to execute on CPU or choose partial GPU offload

That last point is why GGUF is so important for local LLMs. The file doesn't place layers by itself. If the full model doesn't fit in VRAM, a llama.cpp-style runtime can keep some layers on the GPU and spill the rest to system memory.

Why is GGUF not the same kind of thing as GPTQ or AWQ?

Answer

GPTQ and AWQ are quantization algorithms. GGUF is a container format that stores tensors, metadata, tokenizer information, and chosen low-bit tensor types for local runtimes.

Quantization families inside GGUF

GGUF can store several quantization families. The format doesn't force one specific quantizer.

Q4_K_M is a commonly encountered local-inference candidate, but the right choice depends on target model, quality check, and hardware. If the full model fits in VRAM, benchmark GPU-oriented GPTQ or AWQ artifacts against the local runtime; if partial offload is required, GGUF is a useful packaging option.

iMatrix quantization

Importance-matrix quantization uses representative text to estimate which directions are expensive to distort. That extra signal lets IQ formats spend precision where it buys the most quality. Conceptually, it fills the same role as calibration in GPTQ and AWQ: representative data tells the quantizer what errors matter most.

The commands below illustrate a llama.cpp-style conversion and quantization path. Binary names and supported quant types can change, so check the installed revision's documentation before running it.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}

When does iMatrix-style GGUF quantization help most?

Answer

It helps when you are squeezing below comfortable 4-bit settings or using IQ formats. Representative text tells the quantizer which directions matter, similar to calibration data in GPTQ and AWQ.

---

Beyond weight-only: FP8 and KV cache quantization

The low-bit weight artifacts covered above shrink stored model weights. Once the weights are small, the next bottleneck is often the KV cache, the attention state that grows with sequence length.

FP8 sits adjacent to the big three rather than replacing them. It's an 8-bit floating-point format that becomes attractive when the serving hardware has native FP8 kernels.^{[9]Reference 9FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}

FP8 has two common encodings:^{[9]Reference 9FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}

• E4M3: more mantissa precision, less dynamic range
• E5M2: less mantissa precision, more dynamic range

Unlike 4-bit weight-only methods, FP8 is usually chosen when you want a milder accuracy-memory tradeoff and the accelerator is built to exploit FP8 directly.

Quantizing the KV cache is a separate lever. Weight quantization shrinks static model weights. KV-cache quantization shrinks attention state that grows with sequence length. Once weights fit, long-context concurrency can become limited by KV state; a runtime may then offer lower-precision KV storage as another measured tradeoff.

Why can KV-cache quantization matter after weight quantization succeeds?

Answer

Weight quantization shrinks static model weights. KV cache grows with active sequence length and concurrency, so after weights fit, long-context serving may be limited by cached attention state instead.

---

Comparison

When choosing a quantization strategy, don't ask "Which one is best?" Ask "What hardware constraint am I solving for?"

AWQ's paper reports strong 4-bit results by protecting activation-sensitive channels.^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978} GPTQ remains relevant when a runtime or kernel stack supports its packed artifacts efficiently.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323} On GPU servers, benchmark the exact low-bit kernel path. For local or partial-offload deployments, benchmark the GGUF runtime and placement plan rather than assuming a format name decides performance.

What is the fastest decision rule for GPTQ, AWQ, and GGUF?

Answer

If the model fits fully on GPU, start with AWQ or GPTQ based on runtime kernel support. If a local deployment needs CPU/GPU split placement or a portable artifact, start with GGUF and a compatible local runtime.

---

When quantization breaks down

Treating all quantizers as equivalent

• Symptom: You choose "4-bit" from a model hub without checking whether it's GPTQ, AWQ, GGUF, or a runtime quantization path.

• Cause: Bit width describes storage size, not calibration method, tensor layout, kernel support, offload behavior, or quality profile.

• Fix: Name the artifact and the runtime together: "AWQ on vLLM," "GPTQ on Transformers," or "Q4_K_M GGUF on llama.cpp." Then test that exact pair.

The calibration trap

• Symptom: Your quantized French incident-assistant model speaks gibberish, even though the English version quantized fine.

• Cause: GPTQ and AWQ both rely on calibration data to understand which weights matter. If you use English Wikipedia to quantize a model trained on French incident runbooks, the activation statistics are wrong and the quantizer throws away precision in the wrong places.

• Fix: Use calibration text that matches the target domain and language, then verify quality on held-out target tasks.

Confusing weight-only and full quantization

• Symptom: A design doc claims GPTQ or AWQ makes the entire model 4-bit.

• Cause: GPTQ and AWQ are weight-only methods. Activations usually stay at FP16 or BF16, and accumulation happens in higher precision.

• Fix: Write the precision contract explicitly. W4A16 means 4-bit stored weights and 16-bit activations, not full W4A4 inference.

The speed fallacy

• Symptom: You quantize to 4-bit expecting a 4x speedup, but tokens per second barely improve.

• Cause: 4-bit weights save memory bandwidth, but the kernel still has to dequantize them into higher precision before the matrix multiply. If the dequantization code path is slow or the GPU isn't memory-bound to begin with, the speedup shrinks.

• Fix: Measure end-to-end tokens per second on your exact hardware and batch size. Bandwidth savings are real, but they only translate to speed when the runtime is optimized for your GPU.

Treating perplexity as sufficient

• Symptom: The quantized model still chats politely, but it hallucinates deploy status or generates invalid JSON for your incident API.

• Cause: Perplexity on held-out text is a fast sanity check, but a model can show only a small perplexity increase while regressing sharply on structured tasks like code generation or multi-step reasoning. Relying only on perplexity (not ignoring it) is the failure mode.

• Fix: Always pair perplexity with task-specific benchmarks. For an incident-assistant model, run your own production eval set that includes the exact output formats the model must produce.

Confusing GGUF with the quantizer

• Symptom: Someone says "we used GGUF quantization" as if that fully specifies the quality and runtime behavior.

• Cause: GGUF is the container. Q4_K_M, IQ4_XS, Q5_K_M, and related tensor types describe the actual low-bit encoding inside the file.

• Fix: Report both: "GGUF Q4_K_M with 20 GPU layers," "GGUF IQ4_XS with iMatrix," or another concrete artifact/runtime pairing.

---

How to evaluate a quantized model

Perplexity is a good first sanity check, but it's not enough. A quantized model can show a modest change in perplexity while still regressing on code generation, structured output, or domain decisions. A returns-classification model should therefore be tested on representative extraction and action-format cases, not general text alone.

GPTQ and AWQ both report that 4-bit weight-only quantization can preserve language-modeling quality surprisingly well on large models, while more aggressive bit widths degrade more sharply.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}

Three practical rules:

• GPTQ and AWQ results support testing 4-bit weight-only artifacts before pushing to more aggressive bit widths.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}
• Task-specific regressions can't be inferred from a generic quality score; reasoning, structured output, and tool-use tasks need their own gates.
• Group size, calibration data, and the serving kernel can matter as much as the headline format name.

Why isn't perplexity enough to approve a quantized model?

Answer

Perplexity checks broad next-token drift, but structured outputs, code, math, tool calls, and domain-specific decisions can regress without a dramatic perplexity change. Pair it with task evals and serving metrics.

---

Decision guide

Choose based on the bottleneck

Practical rule of thumb: if the whole model fits on GPU, benchmark an AWQ or GPTQ path supported by your runtime. If a local deployment needs CPU/GPU split placement, benchmark a GGUF artifact with a compatible runtime.

For a local workstation GPU and a model too large for full VRAM residency, why is GGUF a reasonable first artifact?

Answer

GGUF works well with local runtimes that can split layers between GPU and system RAM. It may be slower than full-GPU kernels, but it lets the model run when a GPU-only AWQ or GPTQ deployment doesn't fit.

---

Quantization decision checklist

• Quantization stores weights with fewer bits using scales and, sometimes, zero points. The main win is lower memory bandwidth and smaller model artifacts.
• GPTQ uses representative activations and approximate second-order information to minimize output error during post-training quantization.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323}
• AWQ identifies activation-sensitive channels and rescales them so the quantizer spends precision where it matters most.^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}
• GGUF is a portable container for local inference that can store many ggml quantization types; compatible runtimes such as llama.cpp choose CPU/GPU-offload placement.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}
• Benchmark AWQ or GPTQ for fully GPU-resident serving. Benchmark GGUF when portability or partial offload is the main constraint.