Model Quantization: GPTQ, AWQ & GGUF

Model parallelism splits a large model across several accelerators. Quantization starts earlier: what if each weight used fewer bytes before you split anything?

Model quantization stores selected tensors with fewer bits so large models can require less memory and potentially less bandwidth. Imagine serving Qwen3.6-27B for a developer-assistant route. Its dense weights need about 54 GB in FP16 or BF16, which can fit raw H100 capacity but not a typical 24 GB workstation budget before KV cache or runtime buffers.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} Model parallelism can split those bytes across accelerators. Quantization asks whether you can shrink the bytes first.

The technique works like replacing a precise ruler with a coarser grid: storage reduction is predictable, while error depends on which values the grid distorts. A weight-only method packs approximate low-bit values plus metadata. LLM generation is often memory-bandwidth bound, so 4-bit weights also cut raw weight traffic to about one quarter of FP16. Real speedups are smaller than 4x because kernels still have to unpack values and accumulate in higher precision. Compare GPTQ, a Hessian-aware post-training quantization method; AWQ, activation-aware weight quantization; and GGUF, a portable local inference container format. For each one, focus on the serving tradeoff: what changes, which runtime can use the artifact, and what quality you must measure before deployment.

---

What is quantization?

Quantization resembles rounding measurements to coarser marks. A precise ruler has millimeter markings, so you can measure 3.7 mm, 4.2 mm, 12.1 mm. A coarse ruler only has centimeter markings, so those same values become 4 cm, 4 cm, 12 cm. You lose some precision, but the ruler is simpler and cheaper. Quantization does this to every number in a neural network.

A tiny worked example

Before the general formula, walk through one weight by hand. Suppose a weight has the value $w = 0.73$ and you choose a scale of $s = 0.1$. That means each integer step represents $0.1$ in the original space.

1. Divide: $0.73 / 0.1 = 7.3$
2. Round to the nearest integer: $7$
3. Store the integer $q = 7$

During inference you reverse the process:

$\hat{w} = s \cdot q = 0.1 \cdot 7 = 0.7$

The stored value is $0.7$, not $0.73$. The error is tiny ($0.03$), and when this happens across billions of weights the model usually stays useful. If you had chosen a coarser scale of $s = 0.5$, the same weight would become $q = \text{round}(0.73 / 0.5) = 1$ and the reconstructed value would be $0.5$, which is a much larger error. The art of quantization is choosing the right scale so the errors stay small where they matter most.

At the simplest level, quantization maps a floating-point weight to a smaller integer range using a scale and, in some schemes, a zero point:

$q = \text{clip}\left(\text{round}\left(\frac{w}{s}\right) + z,\ q_{\min},\ q_{\max}\right)$

The quantization formula

Take the original weight $w$, divide by the scale $s$ to express it in "integer-sized steps," shift by the zero point $z$, round to the nearest integer, and clamp to the representable range. That's how a high-precision weight becomes an INT8 or INT4 value.

• $w$ is the original floating-point weight
• $q$ is the stored integer
• $s$ is the scale factor
• $z$ is the zero point
• $q_{\min}, q_{\max}$ are the integer limits (for example, 0 to 15 for unsigned 4-bit)

Dequantization reverses the process during inference:

$\hat{w} = s \cdot (q - z)$

Reading the formula

Subtract the zero point from the stored integer, then multiply by the scale. The result $\hat{w}$ is only an approximation of the original weight because rounding already threw away information.

Symmetric vs. asymmetric

A numeric sensor display that stores activation readings with only 16 possible markings has two strategies available:

• Symmetric: center the 16 markings around zero, such as -8 to +7. This is simple and hardware-friendly, but it wastes range if the values are skewed.
• Asymmetric: move the markings to the actual range you observed. This uses the integer range more efficiently, but it requires the extra zero-point offset.

More formally:

1. Symmetric quantization: uses $z = 0$ and maps weights around zero. This is common for weight-only LLM kernels because the math is simpler.
2. Asymmetric quantization: uses a non-zero $z$ to better cover distributions that aren't centered at zero. This is common for activations and some weight formats.

Most LLM weight tensors are roughly zero-centered, so symmetric or near-symmetric per-group quantization often works well for weights. Activations are harder because a few channels can contain very large outliers. Techniques like SmoothQuant^{[2]Reference 2SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.https://arxiv.org/abs/2211.10438} make activation quantization easier by shifting some of that difficulty into the weights.

The pipeline figure walks through the basic quantization flow: start with high-precision weights, estimate the statistics needed to compute scales, then pack the results into a low-bit representation plus metadata.

Memory savings

The most obvious benefit of quantization is memory reduction. That directly translates to lower serving cost, larger batch sizes, and the ability to fit bigger models on smaller devices.

The memory table counts weights only. Actual runtime memory is higher because you still pay for activations, the KV cache, scale metadata, and framework overhead.

Those numbers are idealized weight math. Real packed formats are larger when they also store per-group scales, alignment, or tensors kept at higher precision. For a GGUF artifact, inspect the selected quantization type and actual file/runtime footprint rather than assuming the ideal 13.5 GB number.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}

Because LLM generation is often weight-bandwidth bound, shrinking the weights can also speed up inference. The raw bandwidth demand drops almost linearly with bit width, although the observed throughput gain depends on the kernel, batch size, and how much extra work the runtime does to unpack the weights.

---

Sizing exercise

Try this before moving on. You have a workstation with one NVIDIA RTX 4060 (8 GB VRAM) and you want to run Qwen3.6-27B entirely on the GPU. The model has roughly 27 billion dense parameters.^{[1]Reference 1Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B}

1. How much VRAM would the model need in FP16?
2. How much would it need at INT4?
3. Can you run it on the 4060?

Use this tiny calculator to check the arithmetic without any framework overhead:

Solution

1. FP16/BF16: $27 \times 2 = 54$ GB. The model doesn't fit.
2. INT4: $27 \times 0.5 = 13.5$ GB. The model still doesn't fit on an 8 GB card.
3. You can't run the full model on the GPU alone. One local option is a GGUF artifact loaded by a runtime with heavy CPU offload. The other options are a larger GPU or a smaller model. The INT4 size is a huge improvement, but it doesn't remove the memory budget.

This is the calculation you should do before choosing a quantization strategy. The formula is simple: parameters $\times$ bytes per parameter = weight footprint. Runtime memory is higher because of activations, KV cache, and metadata.

---

Weight-only vs. weight-activation quantization

Two broad approaches to quantization exist, and confusing them causes a lot of interview mistakes.

Weight-only quantization is what GPTQ and AWQ target. The stored weights are low precision, while optimized kernels unpack or dequantize low-bit values during computation and accumulate with higher-precision activations or accumulators. Notation like W4A16 means 4-bit stored weights with 16-bit activations.

Weight-activation quantization pushes both sides of the matmul down, for example W8A8 or W4A8. This can be faster and more memory-efficient, but it's harder because activation distributions are spikier and less stable than weight distributions. Techniques like SmoothQuant^{[2]Reference 2SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.https://arxiv.org/abs/2211.10438} exist specifically to make weight-activation quantization practical.

GPTQ and AWQ are weight-only methods. They capture large weight-memory savings without requiring equally low-bit activations. Lower-bit activation paths such as W4A4 need separate hardware, kernel, and quality validation.

Common mistake: Candidates often claim W4A16 quantization gives a 4x end-to-end speedup. It doesn't guarantee that. Low-bit weights reduce raw weight traffic, but kernel overhead and non-weight work remain, and the workload may not be bandwidth-bound. The guaranteed first-order gain is smaller stored weights, not a fixed tokens-per-second multiplier.

---

GPTQ (post-training quantization)

The intuition: balancing errors like a checkbook

For simplicity, take a layer with only two weights: $w_1 = 1.2$ and $w_2 = 0.8$. A naive quantizer might round both toward the nearest integer, turning them into $1.0$ and $1.0$. The first weight lost $0.2$, the second gained $0.2$. The total output change depends on how the layer uses those weights.

If the calibration data shows that $w_1$ is multiplied by large activations and $w_2$ by small ones, the $-0.2$ error on $w_1$ hurts the output far more than the $+0.2$ error on $w_2$. GPTQ notices this through the Hessian approximation and compensates: it might quantize $w_1$ more carefully, or adjust $w_2$ in the opposite direction to cancel some of the damage. GPTQ isn't trying to make every weight close to its original value. It tries to keep the layer's output on real data as close as possible to the original output.

Algorithm: minimize output error with curvature information

GPTQ works like carefully removing blocks from a Jenga tower. When you remove one block (round one weight), the tower shifts. A careless player ignores the shift and keeps pulling until the tower collapses. GPTQ compensates for the shift. After quantizing one set of weights, it updates the remaining floating-point weights so the layer output changes as little as possible.

GPTQ^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323} is a one-shot post-training quantization method based on approximate second-order information. It builds on the layer-wise Optimal Brain Quantization (OBQ) solver, which quantizes weights one at a time and updates the remaining weights to minimize the layer's output error. GPTQ makes that idea fast enough for billion-parameter models by quantizing weights in a fixed order and using lazy batched updates. The key objective isn't "make the quantized weights numerically close to the originals." It's "make the layer output on real activations stay close to the original output." For a weight row $w$ and calibration activations $X$, GPTQ approximates:

$$\begin{aligned} \hat{w} &= \arg\min_{\tilde{w} \in \mathcal{Q}} \|wX - \tilde{w}X\|_2^2 \\ &\approx \arg\min_{\tilde{w} \in \mathcal{Q}} (w - \tilde{w})^T H (w - \tilde{w}) \\ H &\approx XX^T \end{aligned}$$

Reading the formula

The Hessian approximation $H$ tells GPTQ which input directions matter most on the calibration set. A small error on an unimportant direction is cheap. The same numeric error on a frequently used direction is expensive. That's why GPTQ usually beats plain round-to-nearest quantization at the same bit width.

In practice, GPTQ looks like this:

1. Collect representative activations from a calibration set such as C4^{[5]Reference 5Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683}.
2. Approximate $H \approx XX^T$ for each linear layer.
3. Quantize the weights sequentially while using an approximate inverse Hessian to compensate the remaining floating-point weights.
4. Pack the result into a low-bit format that an inference kernel can consume efficiently.

The original paper reports quantizing 175B-class models (OPT-175B and BLOOM-176B) in about four GPU-hours while preserving strong accuracy at 3-bit and 4-bit settings.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323}

The configuration sketch below shows the shape of a transformers GPTQ workflow. Backend packages and supported arguments change over time, so verify current library documentation and evaluate the resulting artifact before deployment.^{[6]Reference 6GPTQhttps://huggingface.co/docs/transformers/main/en/quantization/gptq}

Granularity: per-tensor, per-channel, per-group

A naive quantizer uses one scale for the entire tensor. This is per-tensor quantization. It's cheap, but one large outlier can ruin the precision of everything else.

Modern LLM quantizers almost always use finer granularity:

Per-group quantization is the common compromise for 4-bit LLM inference. Smaller groups usually improve fidelity, but they also require storing more scale metadata and may reduce kernel efficiency.

---

AWQ (activation-aware weight quantization)

Not all weights are equally dangerous to quantize

The building-renovation analogy is useful here: most walls can be thinned or moved with little impact, but a few are load-bearing. Disturb those and the whole structure becomes unstable. AWQ tries to identify the "load-bearing" weight channels.

AWQ^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978} starts from the observation that activation magnitudes aren't evenly distributed. A small fraction of channels carry disproportionately large activations. If the corresponding weight columns are quantized poorly, the downstream matmul error gets amplified. The AWQ paper reports that protecting only about 1% of salient weights can greatly reduce quantization error.^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}

Why protecting a few weights matters

Now make the two-weight layer's activation pattern extreme: $w_1$ is almost always multiplied by $100$, while $w_2$ is multiplied by $0.1$. A $0.1$ rounding error on $w_1$ becomes a $10$ unit output error, while the same error on $w_2$ becomes only $0.01$ units. AWQ identifies these "high-traffic" channels and rescales them so the quantizer spends more of its limited integer range on the weights that matter most.

Algorithm

AWQ doesn't rebuild the entire weight matrix the way GPTQ does. Instead, it uses an equivalent rescaling trick:

$W x = W \cdot \mathrm{diag}(s) \cdot \mathrm{diag}(s)^{-1} x \approx Q\!\left(W \cdot \mathrm{diag}(s)\right) \cdot \mathrm{diag}(s)^{-1} x$

Reading the formula

Multiply important weight columns by a scaling vector $s$ before quantization so they occupy more of the available integer range. Then divide the corresponding activation channels by the same factor. The floating-point computation stays equivalent, but the quantizer now spends more precision on the columns that matter most.

The practical workflow is:

1. Run representative inputs through the model and collect activation statistics.
2. Identify salient channels with unusually large activation magnitude.
3. Search for scaling factors that reduce the quantization error on those channels.
4. Quantize the rescaled weights into a hardware-friendly 4-bit format.

AWQ artifacts are typically produced offline and loaded by a serving runtime that understands the artifact's quantization metadata, such as group size and zero-point policy. Loader and kernel support varies by runtime version, so verify the chosen artifact/runtime pair before benchmarking.^{[8]Reference 8AWQhttps://huggingface.co/docs/transformers/main/en/quantization/awq}

Compared with GPTQ, AWQ is lighter-weight because it avoids GPTQ's reconstruction step. It often works especially well on instruction-tuned checkpoints, but the final speed and latency picture still depends on the runtime and kernel implementation.^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978[8]Reference 8AWQhttps://huggingface.co/docs/transformers/main/en/quantization/awq}

---

GGUF (llama.cpp format)

What is GGUF?

GGUF is a file format, not a quantization algorithm. It's the container format used by the ggml / llama.cpp ecosystem for local inference.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}

GPTQ and AWQ mainly answer the question "How should I quantize the weights?" GGUF answers a different question: "How should I package model tensors and metadata so local runtimes can load and run them efficiently?"

GGUF matters because it bundles the tensors with the metadata needed to run them:

• tokenizer and vocabulary information
• architecture metadata and tensor shapes
• tensor-by-tensor quantization types inside one portable file
• enough information for compatible local runtimes to execute on CPU or choose partial GPU offload

That last point is why GGUF is so important for local LLMs. The file doesn't place layers by itself. If the full model doesn't fit in VRAM, a llama.cpp-style runtime can keep some layers on the GPU and spill the rest to system memory.

Quantization families inside GGUF

GGUF can store several quantization families. The format doesn't force one specific quantizer.

Q4_K_M is a commonly encountered local-inference candidate, but the right choice depends on target model, quality check, and hardware. If the full model fits in VRAM, benchmark GPU-oriented GPTQ or AWQ artifacts against the local runtime; if partial offload is required, GGUF is a useful packaging option.

iMatrix quantization

Importance-matrix quantization uses representative text to estimate which directions are expensive to distort. That extra signal lets IQ formats spend precision where it buys the most quality. Conceptually, it fills the same role as calibration in GPTQ and AWQ: representative data tells the quantizer what errors matter most.

The commands below illustrate a llama.cpp-style conversion and quantization path. Binary names and supported quant types can change, so check the installed revision's documentation before running it.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}

---

Beyond weight-only: FP8 and KV cache quantization

The low-bit weight artifacts covered above shrink stored model weights. Once the weights are small, the next bottleneck is often the KV cache, the attention state that grows with sequence length.

FP8 sits adjacent to the big three rather than replacing them. It's an 8-bit floating-point format that becomes attractive when the serving hardware has native FP8 kernels.^{[9]Reference 9FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}

FP8 has two common encodings:^{[9]Reference 9FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}

• E4M3: more mantissa precision, less dynamic range
• E5M2: less mantissa precision, more dynamic range

Unlike 4-bit weight-only methods, FP8 is usually chosen when you want a milder accuracy-memory tradeoff and the accelerator is built to exploit FP8 directly.

Quantizing the KV cache is a separate lever. Weight quantization shrinks static model weights. KV-cache quantization shrinks attention state that grows with sequence length. Once weights fit, long-context concurrency can become limited by KV state; a runtime may then offer lower-precision KV storage as another measured tradeoff.

---

Comparison

When choosing a quantization strategy, don't ask "Which one is best?" Ask "What hardware constraint am I solving for?"

AWQ's paper reports strong 4-bit results by protecting activation-sensitive channels.^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978} GPTQ remains relevant when a runtime or kernel stack supports its packed artifacts efficiently.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323} On GPU servers, benchmark the exact low-bit kernel path. For local or partial-offload deployments, benchmark the GGUF runtime and placement plan rather than assuming a format name decides performance.

---

When quantization breaks down

Treating all quantizers as equivalent

• Symptom: You choose "4-bit" from a model hub without checking whether it's GPTQ, AWQ, GGUF, or a runtime quantization path.

• Cause: Bit width describes storage size, not calibration method, tensor layout, kernel support, offload behavior, or quality profile.

• Fix: Name the artifact and the runtime together: "AWQ on vLLM," "GPTQ on Transformers," or "Q4_K_M GGUF on llama.cpp." Then test that exact pair.

The calibration trap

• Symptom: Your quantized French incident-assistant model speaks gibberish, even though the English version quantized fine.

• Cause: GPTQ and AWQ both rely on calibration data to understand which weights matter. If you use English Wikipedia to quantize a model trained on French incident runbooks, the activation statistics are wrong and the quantizer throws away precision in the wrong places.

• Fix: Use calibration text that matches the target domain and language, then verify quality on held-out target tasks.

Confusing weight-only and full quantization

• Symptom: A design doc claims GPTQ or AWQ makes the entire model 4-bit.

• Cause: GPTQ and AWQ are weight-only methods. Activations usually stay at FP16 or BF16, and accumulation happens in higher precision.

• Fix: Write the precision contract explicitly. W4A16 means 4-bit stored weights and 16-bit activations, not full W4A4 inference.

The speed fallacy

• Symptom: You quantize to 4-bit expecting a 4x speedup, but tokens per second barely improve.

• Cause: 4-bit weights save memory bandwidth, but the kernel still has to dequantize them into higher precision before the matrix multiply. If the dequantization code path is slow or the GPU isn't memory-bound to begin with, the speedup shrinks.

• Fix: Measure end-to-end tokens per second on your exact hardware and batch size. Bandwidth savings are real, but they only translate to speed when the runtime is optimized for your GPU.

Ignoring perplexity

• Symptom: The quantized model still chats politely, but it hallucinates deploy status or generates invalid JSON for your incident API.

• Cause: Perplexity on held-out text is a fast sanity check, but a model can show only a small perplexity increase while regressing sharply on structured tasks like code generation or multi-step reasoning.

• Fix: Always pair perplexity with task-specific benchmarks. For an incident-assistant model, run your own production eval set that includes the exact output formats the model must produce.

Confusing GGUF with the quantizer

• Symptom: Someone says "we used GGUF quantization" as if that fully specifies the quality and runtime behavior.

• Cause: GGUF is the container. Q4_K_M, IQ4_XS, Q5_K_M, and related tensor types describe the actual low-bit encoding inside the file.

• Fix: Report both: "GGUF Q4_K_M with 20 GPU layers," "GGUF IQ4_XS with iMatrix," or another concrete artifact/runtime pairing.

---

How to evaluate a quantized model

Perplexity is a good first sanity check, but it's not enough. A quantized model can show a modest change in perplexity while still regressing on code generation, structured output, or domain decisions. A returns-classification model should therefore be tested on representative extraction and action-format cases, not general text alone.

GPTQ and AWQ both report that 4-bit weight-only quantization can preserve language-modeling quality surprisingly well on large models, while more aggressive bit widths degrade more sharply.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}

Three practical rules:

• GPTQ and AWQ results support testing 4-bit weight-only artifacts before pushing to more aggressive bit widths.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}
• Task-specific regressions can't be inferred from a generic quality score; reasoning, structured output, and tool-use tasks need their own gates.
• Group size, calibration data, and the serving kernel can matter as much as the headline format name.

---

Decision guide

Choose based on the bottleneck

Practical rule of thumb: if the whole model fits on GPU, benchmark an AWQ or GPTQ path supported by your runtime. If a local deployment needs CPU/GPU split placement, benchmark a GGUF artifact with a compatible runtime.

---

Mastery check

Check that you can:

• Distinguish symmetric quantization from asymmetric quantization with scale and zero point.
• Explain why GPTQ uses approximate second-order information to minimize layer reconstruction error.
• Describe AWQ's activation-aware channel protection and why activation statistics matter.
• Explain GGUF as a portable container format, not a quantization algorithm.
• Predict how calibration data and group size affect quantization quality.
• Evaluate a quantized model with perplexity, task benchmarks, and serving metrics.
• Separate weight-only quantization, activation quantization, and KV-cache quantization.

What strong answers show

• Needs work: You can repeat bit widths, but you can't explain why low-bit storage helps bandwidth before it helps model quality or speed.
• Developing: You can describe one format such as GPTQ or GGUF, but you still mix up quantization algorithms, container formats, and runtime offload choices.
• Solid: You can size a model at FP16, INT8, and INT4, explain why runtime memory exceeds raw weight math, and choose between GPTQ, AWQ, and GGUF for a concrete hardware limit.
• Strong: You can explain why GPTQ protects layer outputs, why AWQ protects activation-sensitive channels, and why calibration data and kernel support decide whether a quantized checkpoint is deployable.
• Excellent: You can design an approval gate that combines perplexity, task evals, serving metrics, fallback artifacts, and KV-cache checks before routing production traffic to a low-bit model.

Follow-up questions

Your 34B model is 6 GB over the VRAM budget after ordinary FP16 sizing. Which artifact do you try first?

If 4-bit compression would let the full checkpoint stay resident on GPU, benchmark a supported AWQ or GPTQ kernel path. If the model still won't fit after low-bit packing, evaluate GGUF because partial offload is now the main constraint.

Your French incident-assistant model quantizes badly after calibrating on English Wikipedia. What likely went wrong?

The quantizer protected the wrong directions. GPTQ uses calibration activations to estimate which output errors matter, and AWQ uses them to find activation-sensitive channels. If deployment traffic is French incident response, English Wikipedia gives the wrong activation distribution.

A single outlier channel ruins your per-tensor 4-bit result. Why does per-group or per-channel scaling help?

One global scale forces every weight to share the outlier's range. Per-channel or per-group scaling lets smaller neighborhoods keep their own range, which preserves more local precision while adding moderate metadata overhead.

After moving to W4A16, your long-context agent still runs out of memory at high concurrency. What lever do you pull next?

Look at the KV cache. Weight quantization shrinks static parameters, but the KV cache grows with context length and concurrent sessions. Once weights fit, cache compression or shorter context budgets often become the next bottleneck.

You have hardware with strong native FP8 support and an 8B model that already fits cleanly in VRAM. Why might FP8 beat a jump straight to INT4?

FP8 is a milder compression step than INT4 while still reducing memory pressure. If the accelerator has native FP8 kernels, benchmark it as a candidate before accepting the larger compression risk of INT4.

---

Quantization decision checklist

• Quantization stores weights with fewer bits using scales and, sometimes, zero points. The main win is lower memory bandwidth and smaller model artifacts.
• GPTQ uses representative activations and approximate second-order information to minimize output error during post-training quantization.^{[4]Reference 4GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323}
• AWQ identifies activation-sensitive channels and rescales them so the quantizer spends precision where it matters most.^{[7]Reference 7AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}
• GGUF is a portable container for local inference that can store many ggml quantization types; compatible runtimes such as llama.cpp choose CPU/GPU-offload placement.^{[3]Reference 3llama.cpp: Inference of LLaMA model in pure C/C++https://github.com/ggml-org/llama.cpp}
• Benchmark AWQ or GPTQ for fully GPU-resident serving. Benchmark GGUF when portability or partial offload is the main constraint.

Course handoff

Shrinking model weights for production should be a measurable decision: calculate the memory footprint, choose between GPTQ, AWQ, and GGUF based on hardware constraints, and evaluate whether the compressed model is good enough for the task. Those decisions shape cost-effective LLM serving.

The next challenge is local deployment. Even a quantized model needs the right runtime, hardware plan, container, health check, and evaluation gate. In Local LLM Deployment, you'll turn compression choices into a repeatable serving plan.