LearnInference & Production ScaleLow-Precision GPU Programming

🚀HardInference Optimization

Low-Precision GPU Programming

Build numerically sound FP16, BF16, TF32, FP8, and MXFP8 GPU paths by making scale, accumulation, layout, and kernel contracts explicit.

19 min read

Learning path

Step 149 of 192 in the full curriculum

GEMM From Naive CUDA to Tensor Cores Model Quantization: GPTQ, AWQ & GGUF

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

A GPU operation gets faster after you switch it to a smaller number format. Then one request returns a plausible vector with a 98% error. Nothing crashes. Every value is finite. The narrow input format clipped a spike before the wider accumulator ever saw it.

The previous GEMM lesson tuned a general matrix multiplication (GEMM) while keeping operand, accumulator, and output types separate. The mixed-precision training lesson showed how scaling keeps gradients inside a 16-bit format's range. Here, those ideas become an explicit numeric contract for each tensor and kernel.

That incident separates low-precision programming from changing a dtype argument. A production path needs four compatible contracts:

Encoding: which bit patterns represent finite values, zeros, infinities, and NaNs?
Scaling: which values share a scale, when is it measured, and how is it stored?
Arithmetic: which format enters the multiplier, accumulates partial sums, and stores the epilogue?
Kernel application binary interface (ABI): which layouts, alignments, architecture targets, and scale orientations does the implementation accept?

The running operation is a small $Y=XW$ . $X$ has two rows of 32 activations. One row stays near $10^{-3}$ to $1$ ; the other reaches $500$ . $W$ is a 32-element vector. We'll compare 8-bit floating-point (FP8) inputs with a 32-bit floating-point (FP32) accumulator. This setup is small enough to audit every conversion but has the same failure mode as a large projection: one global scale lets the large row erase detail in the small row.

Read formats as range plus spacing

A format name says how many exponent and fraction bits are available. More exponent bits buy dynamic range. More fraction bits buy precision near a fixed magnitude. The table compares FP32, NVIDIA TensorFloat-32 (TF32), bfloat16 (BF16), IEEE half precision (FP16), and two FP8 encodings.

Format	Stored layout	Largest finite magnitude	Smallest positive normal	Typical GPU role
FP32	1 sign, 8 exponent, 23 fraction	about $3.40\times10^{38}$	about $1.18\times10^{-38}$	reference math, accumulation, sensitive reductions
TF32	FP32 storage; multiplier reads 8 exponent and 10 fraction bits	FP32 range	FP32 normal range	faster FP32 matrix math on supported NVIDIA tensor cores
BF16	1 sign, 8 exponent, 7 fraction	about $3.39\times10^{38}$	about $1.18\times10^{-38}$	broad-range activations, weights, gradients
FP16	1 sign, 5 exponent, 10 fraction	65,504	about $6.10\times10^{-5}$	higher local precision, narrower range
FP8 E4M3	1 sign, 4 exponent, 3 fraction	448	$2^{-6}$	forward tensors when precision matters more than range
FP8 E5M2	1 sign, 5 exponent, 2 fraction	57,344	$2^{-14}$	gradients or tensors needing more range

The Open Compute Project (OCP) 8-bit floating-point specification, shortened to OFP8, defines these two encodings. OFP8 E4M3 reaches subnormals down to $2^{-9}$ and uses its extreme exponent patterns for finite numbers, leaving two NaN encodings and no infinity. OFP8 E5M2 reaches subnormals down to $2^{-16}$ and retains IEEE-style infinities and NaNs.^{[1]Reference 1OCP 8-bit Floating Point Specification (OFP8), Revision 1.0https://www.opencompute.org/documents/ocp-8-bit-floating-point-specification-ofp8-revision-1-0-2023-12-01-pdf-1}

Two comparisons prevent common design mistakes:

FP16 versus BF16: both store 16 bits. FP16 has three more fraction bits, while BF16 inherits FP32's eight exponent bits. FP16 resolves nearby values more finely; BF16 survives much larger and smaller magnitudes.
TF32 versus a tensor dtype: TF32 is an NVIDIA compute mode for FP32 matrix operations, not an 11-bit storage type. FP32 tensors enter a tensor-core multiplier with reduced fraction precision, while products normally accumulate in FP32. Current PyTorch documentation exposes explicit TF32 precision controls and notes that defaults differ between matrix multiplication and convolution.^{[2]Reference 2Numerical Accuracyhttps://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html}

Roundoff grows with magnitude. Around 1, E4M3 spacing is $2^{-3}=0.125$ ; around 256, its spacing is 32. Scaling moves values into a useful part of that nonuniform grid.

Follow one value through the kernel

For each operand, use scale $s>0$ and encode

$q = \operatorname{cast}_{F}\left(\frac{x}{s}\right), \qquad \hat{x}=s q$

where $F$ is the low-precision format. The GEMM then computes products from reconstructed values and accumulates them in the declared accumulator type:

$Y_{ij}=\operatorname{cast}_{O}\left(\sum_k \hat{X}_{ik}\hat{W}_{kj}\right)$

The output cast $O$ is another rounding boundary. A BF16 epilogue can hide improvement gained from an FP32 accumulator if the next operator only receives BF16.

Diagram showing BF16 or FP32 X, measure amax, choose X scale, and scale + FP8 cast. — BF16 or FP32 X, measure amax, choose X scale, and scale + FP8 cast.

Treat every arrow as owned state. If a framework exports QX but omits SX, another runtime sees codes without their units. If it keeps a rowwise scale but launches a kernel expecting tensorwise scale, the bytes are valid and the numbers are wrong.

Conversion has three visible failure modes

Saturation handles values beyond the finite range. Under OFP8 saturating conversion, 500 cast directly to E4M3 becomes 448. Under non-saturating conversion, an E4M3 overflow becomes NaN, while E5M2 overflow becomes infinity. CUDA's FP8 conversion API exposes saturation modes and round-to-nearest-even behavior.^{[1]Reference 1OCP 8-bit Floating Point Specification (OFP8), Revision 1.0https://www.opencompute.org/documents/ocp-8-bit-floating-point-specification-ofp8-revision-1-0-2023-12-01-pdf-1}^{[3]Reference 3FP8 Conversion and Data Movementhttps://docs.nvidia.com/cuda/cuda-math-api/cuda_math_api/group__CUDA__MATH__FP8__MISC.html}

Underflow handles values below the smallest subnormal. OFP8 conversion rounds them to signed zero. Values between the smallest normal and smallest subnormal can survive as subnormals, but with reduced significant precision. Hardware may flush some denormals. PyTorch documents a specific ROCm caveat: on MI200-class products, FP16 and BF16 V_DOT2 and matrix fused multiply-add instructions flush input and output denormals; its other supported AMD GPUs don't have this behavior.^{[1]Reference 1OCP 8-bit Floating Point Specification (OFP8), Revision 1.0https://www.opencompute.org/documents/ocp-8-bit-floating-point-specification-ofp8-revision-1-0-2023-12-01-pdf-1}^{[2]Reference 2Numerical Accuracyhttps://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html}

Rounding chooses a nearby representable value. OFP8 conversion requires round-to-nearest, ties-to-even. The rule is deterministic, but repeated casts can still bias a computation when values repeatedly land on one side of a grid boundary.^{[1]Reference 1OCP 8-bit Floating Point Specification (OFP8), Revision 1.0https://www.opencompute.org/documents/ocp-8-bit-floating-point-specification-ofp8-revision-1-0-2023-12-01-pdf-1}

Clipping telemetry and NaN checks observe different failures. Saturation can produce a finite but badly wrong result. A health check that only counts NaNs misses it.

Worked failure: stale scale, finite answer

Suppose delayed E4M3 scaling uses yesterday's amax of 10:

$s_{old}=\frac{10}{448}$

Today, a value spikes to 500. Its scaled magnitude is 22,400, so saturating conversion clamps it to 448. Reconstruction returns

$\hat{x}=448\cdot \frac{10}{448}=10$

The value is finite and has the right sign, but its relative error is 98%. FP32 accumulation faithfully sums the wrong product.

Scale granularity sets who competes for range

Granularity	Scale count for $X\in\mathbb{R}^{M\times K}$	Benefit	Cost or risk
Tensorwise	1	minimal metadata and simple kernels	one outlier controls every value
Rowwise	$M$	separates activation rows or tokens	axis-specific layout and scale loads
Columnwise	$K$	useful when contraction or output channels differ	isn't interchangeable with rowwise packing
Groupwise	roughly $MK/g$	localizes outliers within groups of size $g$	more metadata and indexing work
MXFP8 blockwise	one E8M0 scale per 32 elements	standardized local power-of-two scale	block alignment, padding, and orientation constraints

The best granularity follows the distribution and access pattern. Finer scaling reduces competition between outliers and small values, but kernels must fetch and apply more metadata. A theoretically accurate grouping can lose overall throughput if it breaks coalesced access or lacks a fused kernel.

Current scaling and delayed scaling

An amax is the largest absolute value in a scale group. That measurement is an observation, not a scale by itself.

Current scaling measures the tensor being converted, derives a scale, then casts it. NVIDIA Transformer Engine describes one FP32 scale per tensor for its current-scaling FP8 recipe. The extra amax pass means the input is read twice: once to reduce amax and again to scale and cast.^{[4]Reference 4Using FP8 and FP4 with Transformer Enginehttps://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html}

Delayed scaling chooses today's scale from prior amax values, so a fused kernel can cast today's input while collecting an amax for a future step. An amax history may choose its maximum or its most recent sample. The saved read improves execution, but a sudden spike can clip under a stale scale. A stale historical maximum can cause the opposite problem: no clipping, yet common values occupy too little of the FP8 grid.^{[4]Reference 4Using FP8 and FP4 with Transformer Enginehttps://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html}

Use the recipe name together with its state:

text

fp8_contract = {
  format: E4M3,
  granularity: tensorwise,
  scale_rule: current,
  rounding: nearest_even,
  overflow: saturate,
  accumulator: FP32,
  output: BF16
}

Transformer Engine's common hybrid recipe uses E4M3 for forward activations and weights, then E5M2 for gradients because gradients need more range. That convention is a recipe choice, not a property that forces every inference tensor into E4M3.^{[4]Reference 4Using FP8 and FP4 with Transformer Enginehttps://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html}^{[5]Reference 5FP8 Formats for Deep Learning.https://arxiv.org/abs/2209.05433}

MXFP8 makes scale ownership part of the format

The OCP Microscaling (MX) specification describes an MX tensor as element values $P_i$ , a shared scale type $X$ , and block size $k$ . Microscaling FP8 (MXFP8) uses 32 E4M3 or E5M2 elements with one E8M0 scale per block. E8M0 stores a power-of-two exponent, so scale multiplication can be implemented as exponent adjustment.^{[6]Reference 6OCP Microscaling Formats (MX) Specification, Version 1.0https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf}

One 32-value block contains 256 element bits plus 8 scale bits, an effective payload of $8.25$ bits per value before padding or container overhead. Other standardized MX members trade more precision for density, including MXFP6, MXFP4, and MXINT8.^{[6]Reference 6OCP Microscaling Formats (MX) Specification, Version 1.0https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf}

MX changes the interface, not the need for a kernel contract:

The contraction dimension is typically padded to a multiple of 32.
Scale and element bytes need their specified physical layout.
Rowwise and columnwise quantizations of a matrix are different numeric objects. NVIDIA's MXFP8 documentation warns that a packed rowwise tensor can't be transposed to obtain the columnwise representation; both orientations must be quantized independently from higher precision data.^{[4]Reference 4Using FP8 and FP4 with Transformer Enginehttps://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html}
The OCP general dot-product definition says its result should be FP32, while internal precision and operation order remain implementation-defined. Bitwise equality across kernels isn't implied.^{[6]Reference 6OCP Microscaling Formats (MX) Specification, Version 1.0https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf}

Run the two-row experiment

This exercise uses PyTorch's OCP E4M3 dtype. Tensorwise scaling uses the exact amax ratio. The MXFP8-like path uses one conservative power-of-two scale per 32-value row so every block maximum fits inside E4M3. OCP permits multiple scale-selection algorithms, so the exercise demonstrates granularity rather than claiming one universal MX encoder.

Predict what happens to 0.001 when it shares one scale with 500, then run the code.

fp8-scale-granularity.py

import math
import torch

torch.set_printoptions(precision=6, sci_mode=False)

pattern_small = [0.001, 0.03, 0.117, 0.5, 1.0, -0.25, 0.0625, -0.015625]
pattern_large = [4.0, 16.0, 64.0, 500.0, -2.0, -8.0, -32.0, -250.0]
weight_pattern = [0.5, -0.25, 0.125, -0.0625, 0.03125, -0.5, 0.25, -0.125]

x = torch.tensor([pattern_small * 4, pattern_large * 4], dtype=torch.float32)
w = torch.tensor(weight_pattern * 4, dtype=torch.float32).reshape(32, 1)
fp8_max = torch.finfo(torch.float8_e4m3fn).max

def roundtrip(values: torch.Tensor, scale: torch.Tensor) -> torch.Tensor:
    return (values / scale).to(torch.float8_e4m3fn).float() * scale

tensor_scale_x = x.abs().max() / fp8_max
tensor_scale_w = w.abs().max() / fp8_max
x_tensor = roundtrip(x, tensor_scale_x)
w_tensor = roundtrip(w, tensor_scale_w)

def conservative_power_of_two_scale(block: torch.Tensor) -> float:
    ratio = block.abs().max().item() / fp8_max
    return 2.0 ** math.ceil(math.log2(ratio))

mx_scales_x = torch.tensor(
    [conservative_power_of_two_scale(row) for row in x], dtype=torch.float32
).reshape(2, 1)
mx_scale_w = torch.tensor(conservative_power_of_two_scale(w), dtype=torch.float32)
x_mx = roundtrip(x, mx_scales_x)
w_mx = roundtrip(w, mx_scale_w)

reference_y = (x @ w).squeeze()
tensor_y = (x_tensor @ w_tensor).squeeze()
mx_y = (x_mx @ w_mx).squeeze()

print(f"tensor_scale_x={tensor_scale_x.item():.6f}")
print(f"mxfp8_scales_x={mx_scales_x.squeeze().tolist()}")
print(
    "x[0,0]: "
    f"reference={x[0, 0].item():.8f} "
    f"tensor_fp8={x_tensor[0, 0].item():.8f} "
    f"mxfp8={x_mx[0, 0].item():.8f}"
)
print(
    "x[1,3]: "
    f"reference={x[1, 3].item():.8f} "
    f"tensor_fp8={x_tensor[1, 3].item():.8f} "
    f"mxfp8={x_mx[1, 3].item():.8f}"
)
print(f"reference_y=[{reference_y[0]:.6f}, {reference_y[1]:.6f}]")
print(f"tensor_fp8_y=[{tensor_y[0]:.6f}, {tensor_y[1]:.6f}]")
print(f"mxfp8_y=[{mx_y[0]:.6f}, {mx_y[1]:.6f}]")
print(f"tensor_abs_error={(tensor_y - reference_y).abs().tolist()}")
print(f"mxfp8_abs_error={(mx_y - reference_y).abs().tolist()}")

Output

tensor_scale_x=1.116071
mxfp8_scales_x=[0.00390625, 2.0]
x[0,0]: reference=0.00100000 tensor_fp8=0.00000000 mxfp8=0.00097656
x[1,3]: reference=500.00000000 tensor_fp8=500.00000000 mxfp8=512.00000000
reference_y=[0.600812, 7.750000]
tensor_fp8_y=[0.583104, 7.568359]
mxfp8_y=[0.601562, 7.750000]
tensor_abs_error=[0.017708778381347656, 0.181640625]
mxfp8_abs_error=[0.0007500052452087402, 0.0]

The tensorwise scale is about 1.116, so the smallest nonzero reconstructed E4M3 value is $1.116\times2^{-9}\approx0.00218$ . The first activation, 0.001, falls below half that first step and rounds to zero. Its row loses almost 3% in the final dot product.

The first block's power-of-two scale is $2^{-8}$ . Its smallest subnormal reconstructs as $2^{-17}\approx7.63\times10^{-6}$ , so 0.001 survives as 0.00097656. The large row gets its own scale of 2.0, but its 500 still rounds to 512. The second dot product happens to be exact because positive and negative rounding errors cancel for this weight vector, not because block scaling preserves every operand.

Side-by-side tables compare one tensorwise E4M3 scale with one power-of-two scale per 32-value row. With global scale 1.116071, 0.001 reconstructs as zero. With row scales 0.00390625 and 2, 0.001 reconstructs as 0.00097656 while 500 rounds to 512; the second dot product is exact only because rounding errors cancel for this weight vector. — One scale forces both rows onto the same E4M3 grid, erasing 0.001. Separate 32-value scales recover the small row, while 500 still rounds to 512; the exact second dot product comes from error cancellation, not lossless encoding.

The result doesn't prove MXFP8 always wins. It isolates one mechanism. On a distribution with uniform magnitude, per-block metadata may add traffic without enough accuracy benefit. Measure error and kernel throughput together.

Accumulation is a separate precision decision

A multiply format and an accumulator format solve different problems. FP8 or FP16 operands reduce storage and can expose higher-throughput tensor-core paths. FP32 accumulation reduces loss while many products are added.

FP32 accumulation still has limits:

Products are formed from already rounded inputs.
Some libraries permit reduced-precision reductions for FP16 or BF16 GEMMs on supported hardware. PyTorch exposes flags to disable those paths when numerical error matters more than speed.^{[2]Reference 2Numerical Accuracyhttps://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html}
Split-K or parallel reductions change addition order. Floating-point addition isn't associative, so two correct kernels may differ in low bits.
Bias, activation, residual addition, and output cast can each introduce another precision boundary.

Record a full signature such as E4M3 x E4M3 -> FP32 accumulate -> BF16 output. Saying "FP8 GEMM" leaves the most important correctness choices unstated.

Hardware support is a dated claim

The table below is a documentation snapshot verified 2026-08-29. Architecture capability, installed toolkit, framework release, and chosen kernel all have to agree.

Ecosystem	Documented architecture support in this snapshot	Boundary to keep explicit
NVIDIA TF32	Ampere and later in current PyTorch guidance	enabled state and library heuristic affect whether a specific operation uses TF32
NVIDIA FP8	CUDA capability tables list FP8 from compute capability 8.9; Transformer Engine documents FP8 on Ada, Hopper, and Blackwell	a dtype object or device capability doesn't promise every operator has an FP8 kernel
NVIDIA MXFP8	Transformer Engine documents native MXFP8 on Blackwell SM 10.0 and 10.3	block layout, dual orientation, and recipe support are kernel-specific
AMD FP8	ROCm lists matrix-core FP8 on CDNA3, CDNA4, and RDNA4 families	MI300-class CDNA3 paths use FNUZ; MI350-class CDNA4 paths use OCP FP8; RDNA4 and library support need path-specific checks

CUDA's current programming guide supplies the NVIDIA capability table.^{[7]Reference 7CUDA Programming Guide.https://docs.nvidia.com/cuda/cuda-programming-guide/} AMD's current precision matrix names example products and separates language types, matrix-core support, and library support.^{[8]Reference 8Data Types and Precision Supporthttps://rocm.docs.amd.com/en/latest/reference/precision-support.html}

Interoperability needs particular care on AMD. FNUZ E4M3 and E5M2 encodings differ from OCP variants in special values and signed-zero behavior. Identical raw bytes can mean different numbers. PyTorch exposes OCP-style and FNUZ dtype names, plus E8M0 and packed low-bit types, but its own dtype documentation warns that shell dtype availability doesn't imply broad operator support.^{[9]Reference 9Tensor Attributeshttps://docs.pytorch.org/docs/stable/tensor_attributes.html}

Write the kernel contract before choosing the kernel

Use a contract review that another engineer could implement without guessing:

Field	Example decision	Failure when omitted
Logical shapes	$X:[M,K]$ , $W:[K,N]$ , $K\bmod32=0$	wrong padding or tail handling
Element encodings	OCP E4M3 for both operands	OCP/FNUZ byte mismatch
Scale semantics	E8M0, one per 32 contiguous $K$ values	wrong axis or stale metadata
Operand layouts	row-major $X$ , kernel-packed $W$	valid allocation, scrambled values
Scale layouts	rowwise $X$ , columnwise $W$	transposed scale association
Accumulator	FP32	unexplained reduction drift
Epilogue	FP32 bias, GELU, BF16 store	hidden output rounding
Overflow and rounding	saturate, nearest-even	NaN versus clamp disagreement
Alignment	pointer, stride, and tile constraints	fallback, fault, or slow path
Architecture	compiled target and minimum runtime	unsupported instruction or silent fallback

Treat framework, compiler, and kernel boundaries as serialization boundaries. Pass dtype tags, scales, orientation, padding, and accumulator mode together. Reject ambiguous inputs at dispatch rather than inferring them from tensor shape.

Diagnose the numeric boundary

Start from the first boundary where the low-precision path diverges from an FP32 reference.

Symptom	Likely boundary	Targeted probe	Typical correction
Raw FP8 codes pinned at ±448	E4M3 conversion	clip count and pre-cast amax	refresh scale or use finer granularity
Many exact zeros in one row	scale grouping or underflow	zero rate by row/block before GEMM	rowwise or blockwise scale
NaNs appear only with non-saturating cast	overflow policy	compare saturating and non-saturating conversions	specify policy and fix range
Error starts after reduction	accumulator or split-K order	store FP32 partials and compare reduction modes	wider reduction or stable algorithm
Correct on one vendor, wrong on another	encoding ABI	decode known byte patterns, especially zero/NaN	convert OCP and FNUZ explicitly
Correct values, disappointing speed	layout or kernel dispatch	profiler kernel name, bytes moved, fallback log	pack expected layout or choose supported recipe
MX result breaks after transpose	scale orientation	compare fresh columnwise quantization	quantize each required orientation independently

A useful incident capture includes input histograms, amax per scale group, selected scales, clip and zero counts, accumulator mode, output error against FP32, actual kernel symbol, and device/runtime versions. A single aggregate relative error hides where the contract failed.

Ship low precision as a measured contract

Low-precision GPU code is safe when every lossy boundary is named and observed:

Pick format from range and spacing, not bit width alone.
Make scale value, granularity, axis, update rule, and history explicit.
Count saturation and underflow even when outputs remain finite.
Separate operand encoding from accumulator and epilogue precision.
Version hardware claims and verify actual kernel dispatch.
Compare accuracy and throughput with the same shapes, layouts, and workload distribution used in serving.

Next Step

Continue to Model Quantization: GPTQ, AWQ & GGUF

It moves from floating-point execution formats to model artifacts that compress weights for deployment. Low-precision kernel contracts explain whether those quantized bytes produce a real memory or throughput win.

PreviousGEMM From Naive CUDA to Tensor Cores

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

OCP 8-bit Floating Point Specification (OFP8), Revision 1.0

Open Compute Project · 2023

https://www.opencompute.org/documents/ocp-8-bit-floating-point-specification-ofp8-revision-1-0-2023-12-01-pdf-1

Numerical Accuracy

PyTorch Contributors · 2026 · Official PyTorch Documentation

https://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html

FP8 Conversion and Data Movement

NVIDIA · 2026 · CUDA Math API

https://docs.nvidia.com/cuda/cuda-math-api/cuda_math_api/group__CUDA__MATH__FP8__MISC.html

Using FP8 and FP4 with Transformer Engine

NVIDIA · 2026

https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html

FP8 Formats for Deep Learning.

Micikevicius, P., et al. · 2022

https://arxiv.org/abs/2209.05433

OCP Microscaling Formats (MX) Specification, Version 1.0

Open Compute Project · 2023

https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf

CUDA Programming Guide.

NVIDIA · 2026

https://docs.nvidia.com/cuda/cuda-programming-guide/

Data Types and Precision Support

AMD · 2026 · ROCm Documentation

https://rocm.docs.amd.com/en/latest/reference/precision-support.html

Tensor Attributes

PyTorch Contributors · 2026 · Official PyTorch Documentation

https://docs.pytorch.org/docs/stable/tensor_attributes.html

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Back to Topics

LearnInference & Production ScaleLow-Precision GPU Programming

🚀HardInference Optimization

Low-Precision GPU Programming

Build numerically sound FP16, BF16, TF32, FP8, and MXFP8 GPU paths by making scale, accumulation, layout, and kernel contracts explicit.

19 min read

Learning path

Step 149 of 192 in the full curriculum

GEMM From Naive CUDA to Tensor Cores Model Quantization: GPTQ, AWQ & GGUF

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

That incident separates low-precision programming from changing a dtype argument. A production path needs four compatible contracts:

Encoding: which bit patterns represent finite values, zeros, infinities, and NaNs?
Scaling: which values share a scale, when is it measured, and how is it stored?
Arithmetic: which format enters the multiplier, accumulates partial sums, and stores the epilogue?
Kernel application binary interface (ABI): which layouts, alignments, architecture targets, and scale orientations does the implementation accept?

Read formats as range plus spacing

Format	Stored layout	Largest finite magnitude	Smallest positive normal	Typical GPU role
FP32	1 sign, 8 exponent, 23 fraction	about $3.40\times10^{38}$	about $1.18\times10^{-38}$	reference math, accumulation, sensitive reductions
TF32	FP32 storage; multiplier reads 8 exponent and 10 fraction bits	FP32 range	FP32 normal range	faster FP32 matrix math on supported NVIDIA tensor cores
BF16	1 sign, 8 exponent, 7 fraction	about $3.39\times10^{38}$	about $1.18\times10^{-38}$	broad-range activations, weights, gradients
FP16	1 sign, 5 exponent, 10 fraction	65,504	about $6.10\times10^{-5}$	higher local precision, narrower range
FP8 E4M3	1 sign, 4 exponent, 3 fraction	448	$2^{-6}$	forward tensors when precision matters more than range
FP8 E5M2	1 sign, 5 exponent, 2 fraction	57,344	$2^{-14}$	gradients or tensors needing more range

Two comparisons prevent common design mistakes:

FP16 versus BF16: both store 16 bits. FP16 has three more fraction bits, while BF16 inherits FP32's eight exponent bits. FP16 resolves nearby values more finely; BF16 survives much larger and smaller magnitudes.
TF32 versus a tensor dtype: TF32 is an NVIDIA compute mode for FP32 matrix operations, not an 11-bit storage type. FP32 tensors enter a tensor-core multiplier with reduced fraction precision, while products normally accumulate in FP32. Current PyTorch documentation exposes explicit TF32 precision controls and notes that defaults differ between matrix multiplication and convolution.^{[2]Reference 2Numerical Accuracyhttps://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html}

Roundoff grows with magnitude. Around 1, E4M3 spacing is $2^{-3}=0.125$ ; around 256, its spacing is 32. Scaling moves values into a useful part of that nonuniform grid.

Follow one value through the kernel

For each operand, use scale $s>0$ and encode

$q = \operatorname{cast}_{F}\left(\frac{x}{s}\right), \qquad \hat{x}=s q$

where $F$ is the low-precision format. The GEMM then computes products from reconstructed values and accumulates them in the declared accumulator type:

$Y_{ij}=\operatorname{cast}_{O}\left(\sum_k \hat{X}_{ik}\hat{W}_{kj}\right)$

The output cast $O$ is another rounding boundary. A BF16 epilogue can hide improvement gained from an FP32 accumulator if the next operator only receives BF16.

Conversion has three visible failure modes

Clipping telemetry and NaN checks observe different failures. Saturation can produce a finite but badly wrong result. A health check that only counts NaNs misses it.

Worked failure: stale scale, finite answer

Suppose delayed E4M3 scaling uses yesterday's amax of 10:

$s_{old}=\frac{10}{448}$

Today, a value spikes to 500. Its scaled magnitude is 22,400, so saturating conversion clamps it to 448. Reconstruction returns

$\hat{x}=448\cdot \frac{10}{448}=10$

The value is finite and has the right sign, but its relative error is 98%. FP32 accumulation faithfully sums the wrong product.

Scale granularity sets who competes for range

Granularity	Scale count for $X\in\mathbb{R}^{M\times K}$	Benefit	Cost or risk
Tensorwise	1	minimal metadata and simple kernels	one outlier controls every value
Rowwise	$M$	separates activation rows or tokens	axis-specific layout and scale loads
Columnwise	$K$	useful when contraction or output channels differ	isn't interchangeable with rowwise packing
Groupwise	roughly $MK/g$	localizes outliers within groups of size $g$	more metadata and indexing work
MXFP8 blockwise	one E8M0 scale per 32 elements	standardized local power-of-two scale	block alignment, padding, and orientation constraints

Current scaling and delayed scaling

An amax is the largest absolute value in a scale group. That measurement is an observation, not a scale by itself.

Use the recipe name together with its state:

text

fp8_contract = {
  format: E4M3,
  granularity: tensorwise,
  scale_rule: current,
  rounding: nearest_even,
  overflow: saturate,
  accumulator: FP32,
  output: BF16
}

MXFP8 makes scale ownership part of the format

MX changes the interface, not the need for a kernel contract:

The contraction dimension is typically padded to a multiple of 32.
Scale and element bytes need their specified physical layout.
Rowwise and columnwise quantizations of a matrix are different numeric objects. NVIDIA's MXFP8 documentation warns that a packed rowwise tensor can't be transposed to obtain the columnwise representation; both orientations must be quantized independently from higher precision data.^{[4]Reference 4Using FP8 and FP4 with Transformer Enginehttps://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html}
The OCP general dot-product definition says its result should be FP32, while internal precision and operation order remain implementation-defined. Bitwise equality across kernels isn't implied.^{[6]Reference 6OCP Microscaling Formats (MX) Specification, Version 1.0https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf}

Run the two-row experiment

Predict what happens to 0.001 when it shares one scale with 500, then run the code.

fp8-scale-granularity.py

import math
import torch

torch.set_printoptions(precision=6, sci_mode=False)

pattern_small = [0.001, 0.03, 0.117, 0.5, 1.0, -0.25, 0.0625, -0.015625]
pattern_large = [4.0, 16.0, 64.0, 500.0, -2.0, -8.0, -32.0, -250.0]
weight_pattern = [0.5, -0.25, 0.125, -0.0625, 0.03125, -0.5, 0.25, -0.125]

x = torch.tensor([pattern_small * 4, pattern_large * 4], dtype=torch.float32)
w = torch.tensor(weight_pattern * 4, dtype=torch.float32).reshape(32, 1)
fp8_max = torch.finfo(torch.float8_e4m3fn).max

def roundtrip(values: torch.Tensor, scale: torch.Tensor) -> torch.Tensor:
    return (values / scale).to(torch.float8_e4m3fn).float() * scale

tensor_scale_x = x.abs().max() / fp8_max
tensor_scale_w = w.abs().max() / fp8_max
x_tensor = roundtrip(x, tensor_scale_x)
w_tensor = roundtrip(w, tensor_scale_w)

def conservative_power_of_two_scale(block: torch.Tensor) -> float:
    ratio = block.abs().max().item() / fp8_max
    return 2.0 ** math.ceil(math.log2(ratio))

mx_scales_x = torch.tensor(
    [conservative_power_of_two_scale(row) for row in x], dtype=torch.float32
).reshape(2, 1)
mx_scale_w = torch.tensor(conservative_power_of_two_scale(w), dtype=torch.float32)
x_mx = roundtrip(x, mx_scales_x)
w_mx = roundtrip(w, mx_scale_w)

reference_y = (x @ w).squeeze()
tensor_y = (x_tensor @ w_tensor).squeeze()
mx_y = (x_mx @ w_mx).squeeze()

print(f"tensor_scale_x={tensor_scale_x.item():.6f}")
print(f"mxfp8_scales_x={mx_scales_x.squeeze().tolist()}")
print(
    "x[0,0]: "
    f"reference={x[0, 0].item():.8f} "
    f"tensor_fp8={x_tensor[0, 0].item():.8f} "
    f"mxfp8={x_mx[0, 0].item():.8f}"
)
print(
    "x[1,3]: "
    f"reference={x[1, 3].item():.8f} "
    f"tensor_fp8={x_tensor[1, 3].item():.8f} "
    f"mxfp8={x_mx[1, 3].item():.8f}"
)
print(f"reference_y=[{reference_y[0]:.6f}, {reference_y[1]:.6f}]")
print(f"tensor_fp8_y=[{tensor_y[0]:.6f}, {tensor_y[1]:.6f}]")
print(f"mxfp8_y=[{mx_y[0]:.6f}, {mx_y[1]:.6f}]")
print(f"tensor_abs_error={(tensor_y - reference_y).abs().tolist()}")
print(f"mxfp8_abs_error={(mx_y - reference_y).abs().tolist()}")

Output

tensor_scale_x=1.116071
mxfp8_scales_x=[0.00390625, 2.0]
x[0,0]: reference=0.00100000 tensor_fp8=0.00000000 mxfp8=0.00097656
x[1,3]: reference=500.00000000 tensor_fp8=500.00000000 mxfp8=512.00000000
reference_y=[0.600812, 7.750000]
tensor_fp8_y=[0.583104, 7.568359]
mxfp8_y=[0.601562, 7.750000]
tensor_abs_error=[0.017708778381347656, 0.181640625]
mxfp8_abs_error=[0.0007500052452087402, 0.0]

Accumulation is a separate precision decision

FP32 accumulation still has limits:

Products are formed from already rounded inputs.
Some libraries permit reduced-precision reductions for FP16 or BF16 GEMMs on supported hardware. PyTorch exposes flags to disable those paths when numerical error matters more than speed.^{[2]Reference 2Numerical Accuracyhttps://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html}
Split-K or parallel reductions change addition order. Floating-point addition isn't associative, so two correct kernels may differ in low bits.
Bias, activation, residual addition, and output cast can each introduce another precision boundary.

Record a full signature such as E4M3 x E4M3 -> FP32 accumulate -> BF16 output. Saying "FP8 GEMM" leaves the most important correctness choices unstated.

Hardware support is a dated claim

The table below is a documentation snapshot verified 2026-08-29. Architecture capability, installed toolkit, framework release, and chosen kernel all have to agree.

Ecosystem	Documented architecture support in this snapshot	Boundary to keep explicit
NVIDIA TF32	Ampere and later in current PyTorch guidance	enabled state and library heuristic affect whether a specific operation uses TF32
NVIDIA FP8	CUDA capability tables list FP8 from compute capability 8.9; Transformer Engine documents FP8 on Ada, Hopper, and Blackwell	a dtype object or device capability doesn't promise every operator has an FP8 kernel
NVIDIA MXFP8	Transformer Engine documents native MXFP8 on Blackwell SM 10.0 and 10.3	block layout, dual orientation, and recipe support are kernel-specific
AMD FP8	ROCm lists matrix-core FP8 on CDNA3, CDNA4, and RDNA4 families	MI300-class CDNA3 paths use FNUZ; MI350-class CDNA4 paths use OCP FP8; RDNA4 and library support need path-specific checks

Write the kernel contract before choosing the kernel

Use a contract review that another engineer could implement without guessing:

Field	Example decision	Failure when omitted
Logical shapes	$X:[M,K]$ , $W:[K,N]$ , $K\bmod32=0$	wrong padding or tail handling
Element encodings	OCP E4M3 for both operands	OCP/FNUZ byte mismatch
Scale semantics	E8M0, one per 32 contiguous $K$ values	wrong axis or stale metadata
Operand layouts	row-major $X$ , kernel-packed $W$	valid allocation, scrambled values
Scale layouts	rowwise $X$ , columnwise $W$	transposed scale association
Accumulator	FP32	unexplained reduction drift
Epilogue	FP32 bias, GELU, BF16 store	hidden output rounding
Overflow and rounding	saturate, nearest-even	NaN versus clamp disagreement
Alignment	pointer, stride, and tile constraints	fallback, fault, or slow path
Architecture	compiled target and minimum runtime	unsupported instruction or silent fallback

Diagnose the numeric boundary

Start from the first boundary where the low-precision path diverges from an FP32 reference.

Symptom	Likely boundary	Targeted probe	Typical correction
Raw FP8 codes pinned at ±448	E4M3 conversion	clip count and pre-cast amax	refresh scale or use finer granularity
Many exact zeros in one row	scale grouping or underflow	zero rate by row/block before GEMM	rowwise or blockwise scale
NaNs appear only with non-saturating cast	overflow policy	compare saturating and non-saturating conversions	specify policy and fix range
Error starts after reduction	accumulator or split-K order	store FP32 partials and compare reduction modes	wider reduction or stable algorithm
Correct on one vendor, wrong on another	encoding ABI	decode known byte patterns, especially zero/NaN	convert OCP and FNUZ explicitly
Correct values, disappointing speed	layout or kernel dispatch	profiler kernel name, bytes moved, fallback log	pack expected layout or choose supported recipe
MX result breaks after transpose	scale orientation	compare fresh columnwise quantization	quantize each required orientation independently

Ship low precision as a measured contract

Low-precision GPU code is safe when every lossy boundary is named and observed:

Pick format from range and spacing, not bit width alone.
Make scale value, granularity, axis, update rule, and history explicit.
Count saturation and underflow even when outputs remain finite.
Separate operand encoding from accumulator and epilogue precision.
Version hardware claims and verify actual kernel dispatch.
Compare accuracy and throughput with the same shapes, layouts, and workload distribution used in serving.

Next Step

Continue to Model Quantization: GPTQ, AWQ & GGUF

PreviousGEMM From Naive CUDA to Tensor Cores

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

OCP 8-bit Floating Point Specification (OFP8), Revision 1.0

Open Compute Project · 2023

https://www.opencompute.org/documents/ocp-8-bit-floating-point-specification-ofp8-revision-1-0-2023-12-01-pdf-1

Numerical Accuracy

PyTorch Contributors · 2026 · Official PyTorch Documentation

https://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html

FP8 Conversion and Data Movement

NVIDIA · 2026 · CUDA Math API

https://docs.nvidia.com/cuda/cuda-math-api/cuda_math_api/group__CUDA__MATH__FP8__MISC.html

Using FP8 and FP4 with Transformer Engine

NVIDIA · 2026

https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html

FP8 Formats for Deep Learning.

Micikevicius, P., et al. · 2022

https://arxiv.org/abs/2209.05433

OCP Microscaling Formats (MX) Specification, Version 1.0

Open Compute Project · 2023

https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf

CUDA Programming Guide.

NVIDIA · 2026

https://docs.nvidia.com/cuda/cuda-programming-guide/

Data Types and Precision Support

AMD · 2026 · ROCm Documentation

https://rocm.docs.amd.com/en/latest/reference/precision-support.html

Tensor Attributes

PyTorch Contributors · 2026 · Official PyTorch Documentation

https://docs.pytorch.org/docs/stable/tensor_attributes.html

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Low-Precision GPU Programming

A tensor-core kernel accepts FP8 inputs and accumulates into FP32. Does FP32 accumulation guarantee a correct answer?

Read formats as range plus spacing

Follow one value through the kernel

Why is an FP8 tensor plus its scale closer to a typed value than the FP8 bytes alone?

Conversion has three visible failure modes

Worked failure: stale scale, finite answer

Scale granularity sets who competes for range

Current scaling and delayed scaling

A delayed-scaling run has no NaNs, but clip count jumps after a distribution shift. Which state should be inspected first?

MXFP8 makes scale ownership part of the format

Run the two-row experiment

Accumulation is a separate precision decision

Hardware support is a dated claim

Write the kernel contract before choosing the kernel

A framework exposes `float8_e4m3fn`, and the target GPU advertises FP8 instructions. Is that enough to dispatch an arbitrary FP8 operator?

Diagnose the numeric boundary

Ship low precision as a measured contract

Mastery Check

Discussion

Low-Precision GPU Programming

A tensor-core kernel accepts FP8 inputs and accumulates into FP32. Does FP32 accumulation guarantee a correct answer?

Read formats as range plus spacing

Follow one value through the kernel

Why is an FP8 tensor plus its scale closer to a typed value than the FP8 bytes alone?

Conversion has three visible failure modes

Worked failure: stale scale, finite answer

Scale granularity sets who competes for range

Current scaling and delayed scaling

A delayed-scaling run has no NaNs, but clip count jumps after a distribution shift. Which state should be inspected first?

MXFP8 makes scale ownership part of the format

Run the two-row experiment

Accumulation is a separate precision decision

Hardware support is a dated claim

Write the kernel contract before choosing the kernel

A framework exposes `float8_e4m3fn`, and the target GPU advertises FP8 instructions. Is that enough to dispatch an arbitrary FP8 operator?

Diagnose the numeric boundary

Ship low precision as a measured contract

Mastery Check

Discussion

A tensor-core kernel accepts FP8 inputs and accumulates into FP32. Does FP32 accumulation guarantee a correct answer?

A tensor-core kernel accepts FP8 inputs and accumulates into FP32. Does FP32 accumulation guarantee a correct answer?

Why is an FP8 tensor plus its scale closer to a typed value than the FP8 bytes alone?

Why is an FP8 tensor plus its scale closer to a typed value than the FP8 bytes alone?

A delayed-scaling run has no NaNs, but clip count jumps after a distribution shift. Which state should be inspected first?

A delayed-scaling run has no NaNs, but clip count jumps after a distribution shift. Which state should be inspected first?

A framework exposes float8_e4m3fn, and the target GPU advertises FP8 instructions. Is that enough to dispatch an arbitrary FP8 operator?

A framework exposes float8_e4m3fn, and the target GPU advertises FP8 instructions. Is that enough to dispatch an arbitrary FP8 operator?

Mastery Check

Discussion

A tensor-core kernel accepts FP8 inputs and accumulates into FP32. Does FP32 accumulation guarantee a correct answer?

A tensor-core kernel accepts FP8 inputs and accumulates into FP32. Does FP32 accumulation guarantee a correct answer?

Why is an FP8 tensor plus its scale closer to a typed value than the FP8 bytes alone?

Why is an FP8 tensor plus its scale closer to a typed value than the FP8 bytes alone?

A delayed-scaling run has no NaNs, but clip count jumps after a distribution shift. Which state should be inspected first?

A delayed-scaling run has no NaNs, but clip count jumps after a distribution shift. Which state should be inspected first?

A framework exposes float8_e4m3fn, and the target GPU advertises FP8 instructions. Is that enough to dispatch an arbitrary FP8 operator?

A framework exposes float8_e4m3fn, and the target GPU advertises FP8 instructions. Is that enough to dispatch an arbitrary FP8 operator?

Mastery Check

Discussion

A framework exposes `float8_e4m3fn`, and the target GPU advertises FP8 instructions. Is that enough to dispatch an arbitrary FP8 operator?

A framework exposes `float8_e4m3fn`, and the target GPU advertises FP8 instructions. Is that enough to dispatch an arbitrary FP8 operator?