Skip to content
Skip to lesson
Back to Topics
LearnComputing FoundationsCUDA for ML Training
⚡EasyFine-Tuning & Training

CUDA for ML Training

Four token sequences can have the right shape and still fail: wrong device, a batch that doesn't fit, or a timer that stops before the GPU finishes.

25 min read
Learning path
Step 5 of 199 in the full curriculum

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

The NumPy lesson tracked what each tensor axis means: examples, token positions, and features. Reuse that shape reasoning for a small sequence classifier: four sequences, eight token positions, and 16 embedding features give shape (4, 8, 16). The model pools each sequence's tokens and predicts three class scores.

That same tensor also has to live somewhere. An operation can have the right shape and still fail when its disagree about CPU memory versus GPU memory. The same four sequences can be indexed correctly and still sit in the wrong memory, exhaust the device, or look fast because the timer stopped before the GPU finished.

is NVIDIA's platform and programming model for running parallel work on NVIDIA GPUs. It doesn't turn every Python line into GPU code. Python still runs on the CPU, while launches GPU functions called for tensor operations whose data lives on a CUDA device.[1]

Platform path

If you train on a Mac with Apple silicon, continue through this lesson for the shared accelerator concepts, then use MPS & Metal for ML on Mac for its backend and memory differences.

Eight tokens onto two blocks

Would the four-sequence miniature automatically run faster on a GPU? Not necessarily. A GPU earns its keep when many items perform similar arithmetic; a tiny tensor, branch-heavy Python loop, or copy on every line can lose to launch and transfer overhead.

A CPU is tuned for low-latency control flow and varied work. A GPU is tuned for high throughput. Large multiplications, convolutions, and attention scores usually offer enough parallel arithmetic to justify the handoff.

Part of a training stepCPU host jobGPU device job
data preparationread, tokenize, pad, and assemble a batchno work yet
device transferrequest a copyreceive tensor data in device memory
forward and backwardlaunch PyTorch operationsexecute tensor kernels
reportingrequest a Python number or file writefinish and return needed values

The CPU prepares and launches; the GPU executes. Inside one kernel launch, a thread handles one logical slice of work. Threads are grouped into thread blocks, and all blocks for one kernel form a grid. CUDA schedules each whole block onto one , a hardware processor inside the GPU. Several blocks may be active on one SM, and CUDA doesn't promise which block runs first.[1]

Use the eight positions of the first sequence, not all four sequences and not every feature. Two blocks with four threads each make the indexing concrete. For a one-dimensional grid of one-dimensional blocks, each thread computes a unique global index:

global_id=block_id×threads_per_block+thread_id\text{global\_id} = \text{block\_id} \times \text{threads\_per\_block} + \text{thread\_id}

For block 1, thread 2, the global index is 1 × 4 + 2 = 6, so that thread handles token position T6. The Python trace below checks all eight assignments, taking block and thread counts as input and printing each block's token positions:

map-token-positions.py
threads_per_block = 4
blocks = 2

for block_id in range(blocks):
    positions = []
    for thread_id in range(threads_per_block):
        global_id = block_id * threads_per_block + thread_id
        positions.append(f"T{global_id}")
    print(f"block {block_id}: {positions}")
Output
block 0: ['T0', 'T1', 'T2', 'T3']
block 1: ['T4', 'T5', 'T6', 'T7']
Eight token positions are divided between two four-thread blocks. The formula global id equals block id times four plus thread id maps block 1, thread 2 to T6. Each block occupies its own 32-lane warp: lanes 0 through 3 are active and lanes 4 through 31 are idle, so four of 32 lanes do useful work in this teaching kernel.
Each of the eight token positions gets one global id. Warps don't cross thread-block boundaries, so each four-thread block uses only 4 of its 32 lanes (12.5%). This lane fraction isn't a measured throughput efficiency. Multiples of 32 avoid partial warps caused by the block size; bounds checks and branches can still leave lanes inactive.

The trace has eight useful threads, but hardware groups threads into a larger unit. An NVIDIA contains 32 threads from one block. Warp lanes execute through a single-instruction, multiple-thread (SIMT) model. If lanes take different branches, CUDA masks inactive lanes while it executes each required path, so divergence can waste throughput.[1]

The four-thread teaching block is legal, and it still occupies a full warp. CUDA fills that warp with consecutive thread ids from the same block, so 4 lanes process token positions and 28 lanes sit unused for the whole launch. A production kernel should choose its block size to keep more lanes active.

Read the hierarchy from a kernel launch downward: its grid contains blocks; each block is scheduled on one SM and advances in warps; each thread owns its index. A PyTorch operation can launch several kernels, and a shape-only view can launch none. Use the table to name what each level contributes:

LevelMeaningBeginner question
PyTorch operationa request such as x * 2 or x @ wIs the tensor on CUDA?
kernelGPU function launched to perform part of the workIs the work large enough to justify a launch?
gridevery block in one launchHow much work exists?
blockthreads that run on one SM and can cooperateWhich slice of work stays together?
warp32 threads scheduled togetherAre branches, or a short last warp, leaving lanes idle?
threadone logical worker with its own indexWhich data item does it handle?

PyTorch's matrix-multiply kernels use optimized tiling, not one thread per token. Treat the eight-position trace as a map for CUDA's indexing and scheduling terms, not as a model of the library kernel.

What if there are ten positions instead of eight? Two four-thread blocks miss the last two positions; three blocks create twelve threads. The final two threads need a bounds check before reading or writing an array. This CPU simulation checks both coverage and the unused tail:

guard-the-last-block.py
positions = 10
threads_per_block = 4
blocks = (positions + threads_per_block - 1) // threads_per_block
visited = []
skipped = []
for block_id in range(blocks):
    for thread_id in range(threads_per_block):
        i = block_id * threads_per_block + thread_id
        if i < positions:
            visited.append(i)
        else:
            skipped.append(i)
assert visited == list(range(positions))
print("blocks:", blocks)
print("guarded indexes:", skipped)
Output
blocks: 3
guarded indexes: [10, 11]

Removing the guard would attempt indexes 10 and 11, beyond the ten-element array. These are ordinary Python simulations of CUDA indexing, not GPU kernels. PyTorch handles the indexing and bounds checks for the tensor operations used below.

Those launches only happen if this process can see a CUDA device. The next check is whether the driver, the PyTorch build, and the GPU agree.

Check driver, PyTorch build, and device

The phrase "my CUDA version" separates several layers of the software stack, and each command checks a different piece. The NVIDIA System Management Interface command, nvidia-smi, reads driver and GPU state. nvcc is NVIDIA's CUDA compiler:

SignalWhat it tells youWhat to check next
nvidia-smiNVIDIA driver can see a GPU. The CUDA Version header (documented as CUDA UMD Version; the older CUDA Version label is deprecated) is the latest CUDA version that driver supportswhich CUDA toolkit is installed, or whether this Python has a CUDA-enabled PyTorch build
torch.__version__installed PyTorch version and often its build suffixwhether GPU access works
torch.version.cudaCUDA version used to build this PyTorch package, or None for a build without CUDAwhether driver and device are reachable
torch.cuda.is_available()CUDA is currently available to this processwhether a particular kernel can execute, fits, or runs fast
nvcc --versionversion of local CUDA toolkit compiler, when installedwhich runtime a prebuilt PyTorch package uses

The CUDA UMD Version describes driver capability, not an installed compiler. A machine can run a prebuilt CUDA-enabled PyTorch package without having nvcc installed.[2] It can't prove that this Python process loaded a CUDA-enabled PyTorch build.

PyTorch's installer offers builds for supported compute platforms, and its verification step uses torch.cuda.is_available().[3] For ordinary prebuilt use, a local toolkit and nvcc matter when you build PyTorch from source or compile custom CUDA extensions. Version strings don't need to match; the driver must support the runtime used by the package.

Run the driver check first, then inspect the Python environment. The shell command produces driver output, PyTorch version and runtime, availability, device name, capability, and architectures when CUDA is reachable:

terminal
nvidia-smi
python3 - <<'PY'
import torch

print("PyTorch:", torch.__version__)
print("PyTorch CUDA runtime:", torch.version.cuda)
print("CUDA available:", torch.cuda.is_available())

if torch.cuda.is_available():
    print("device:", torch.cuda.get_device_name(0))
    print("compute capability:", torch.cuda.get_device_capability(0))
    print("build architectures:", torch.cuda.get_arch_list())
PY

The compute capability pair describes hardware features supported by a GPU generation. get_arch_list() reports architectures included in the PyTorch build. Read the first disagreement, not the most alarming line:

Observed stateLikely layerNext check
nvidia-smi failsdriver, hardware, or container accessdriver installation and device exposure
nvidia-smi works, torch.version.cuda is NonePyTorch package wasn't built for CUDAselect the CUDA-enabled package for this environment
PyTorch has a CUDA runtime, but availability is Falsedriver compatibility or process visibilitydriver support, container GPU access, CUDA_VISIBLE_DEVICES
availability is True, but kernel reports unsupported architectureinstalled binary doesn't support this GPUcheck support in PyTorch and any custom extension; older GPUs can also lose support
availability is True and device name is correctbasic stack workstest placement, memory, and timing

CUDA_VISIBLE_DEVICES changes the GPU indexes visible inside the process. If physical GPU 1 is the only visible device, PyTorch may call it cuda:0. Inspect tensor.device and device names inside the running process instead of assuming host indexes survive a mask.

Once the stack answers "yes," move the miniature sequence batch and check the result. GPU visibility confirms device access; validate tensor placement, memory use, outputs, and timing in the training step below.

Move the sequence batch once

Return to the (4, 8, 16) sequence tensor. What changes during .to(device): its shape or its residence? Shape stays constant. Residence changes from host memory to CUDA device memory.

On a discrete NVIDIA GPU, a copies data from CPU memory to GPU memory. Device-to-host (D2H) is the return path. Keep model weights, batch tensors, activations, , and on the GPU through the hot part of a training step. Bring back the small value needed for reporting.

The runnable step uses random feature values so it needs no dataset download. It tests execution, not classifier quality. It averages eight token into one vector per sequence, then scores three classes indexed 0, 1, and 2.

nn.Linear(16, 3) multiplies each 16-feature vector by learned weights and adds a bias, producing three scores called . Cross-entropy turns those scores and the correct label into one error value, the loss. backward() computes how each weight affects that loss; those derivatives are gradients. Stochastic gradient descent (SGD) updates weights opposite the gradient, aiming to reduce the loss.[4] The later PyTorch training lesson derives the loop; here, watch where its tensors live.

Checking shape and device placement confirms that (4, 8, 16) becomes sequence vectors (4, 16) and logits (4, 3). On CUDA, model and batch agree; on CPU, the same shape and loss checks still run:

one-cuda-training-step.py
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(7)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = nn.Linear(16, 3).to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

cpu_batch = {
    "token_features": torch.randn(4, 8, 16),
    "labels": torch.tensor([0, 2, 1, 0]),
}
batch = {name: tensor.to(device) for name, tensor in cpu_batch.items()}

optimizer.zero_grad()
seq_vectors = batch["token_features"].mean(dim=1)
logits = model(seq_vectors)
loss = F.cross_entropy(logits, batch["labels"])
loss.backward()
assert torch.isfinite(loss).item()
assert all(parameter.grad is not None and torch.isfinite(parameter.grad).all().item()
           for parameter in model.parameters())
optimizer.step()

model_device = next(model.parameters()).device
assert all(tensor.device == model_device for tensor in batch.values())
assert logits.device == loss.device == model_device
assert seq_vectors.shape == (4, 16) and logits.shape == (4, 3)
reported_loss = loss.detach().cpu().item()
print("selected accelerator:", device.type == "cuda")
print("model and batch agree:", model_device == batch["token_features"].device)
print("sequence vectors:", tuple(seq_vectors.shape))
print("logits:", tuple(logits.shape))
print("finite loss:", -float("inf") < reported_loss < float("inf"))

On an accessible NVIDIA GPU, the first two lines should report True. A CPU-only machine reports False for the accelerator line but still validates tensor and training logic. The finite-value assertions check this smoke test before the update; their .item() calls can wait for GPU work, so keep them outside a throughput measurement. If a shape fails, debug the reduction or linear layer first; a device error needs a different branch.

Device placement applies to every tensor an operation touches. Moving features but leaving labels on CPU still fails during a CUDA loss calculation. Deliberately leave the features on CPU in the next fragment; on a CUDA machine this raises a device error, while a CPU-only machine prints the fallback message:

reproduce-device-mismatch.py
import torch
import torch.nn as nn

if torch.cuda.is_available():
    cuda_model = nn.Linear(16, 3).to("cuda")
    cpu_features = torch.randn(4, 16)
    try:
        cuda_model(cpu_features)
    except RuntimeError as error:
        if "device" not in str(error).lower():
            raise
        print("device mismatch:", str(error))
    else:
        raise AssertionError("expected a mixed-device forward pass to fail")
else:
    print("CUDA unavailable: real mixed-device failure wasn't executed")

The symptom is an error saying tensors or arguments aren't on the same device. Cause is placement, not shape. Move every participating tensor to the model's device before the forward or loss call. Synchronizing won't repair a tensor stored on the wrong device.

The miniature step can succeed while a training-sized batch still fails. Shapes and device placement answer whether an operation is legal; memory pressure decides whether the full forward and backward pass can coexist.

Count memory before the first big batch

The small step fits. Before increasing batch size, ask which values must survive between steps and which exist only while one kernel runs. CUDA exposes several storage levels, and they aren't interchangeable:

StorageScopeTypical training data
host RAMCPU processdataset objects, decoded examples, CPU batches
device global memory, often called video RAM (VRAM) or high-bandwidth memory (HBM)all SMs on one GPUweights, activations, gradients, optimizer state
L2 and L1 cachesGPU hardwarerecently accessed device data
shared memorythreads in one blocktiles reused inside a kernel
registersone threadcounters, addresses, and small working values

PyTorch allocates model tensors in device global memory. Optimized kernels decide when to reuse tiles through caches, shared memory, or registers. Calling .to("cuda") doesn't place a whole tensor in a register or in shared memory. An adds its own long-lived state during training.

The miniature batch itself contains 4 × 8 × 16 = 512 float32 values, only 2,048 bytes. A larger (32, 128, 768) float32 batch contains 32 × 128 × 768 × 4 = 12,582,912 bytes, exactly 12.0 MiB of input features. That input is only one term in a training step:

training memory≈weights+activations+gradients+optimizer state+temporary workspaces\text{training memory} \approx \text{weights} + \text{activations} + \text{gradients} + \text{optimizer state} + \text{temporary workspaces}

The example uses SGD without momentum, which doesn't keep per-parameter running statistics. To see why the optimizer choice changes memory, consider Adam, an optimizer that tracks two statistics for each parameter: a running average of gradients and a running average of squared gradients.[5]

For a simplified full-precision Adam budget, count 4 bytes for each weight, 4 for its gradient, and 8 for its two statistics. One billion parameters therefore require 16 billion bytes for these tensors when they're all present. Activations, temporary buffers, allocator overhead, and any extra parameter copies still sit outside this accounting.

A CPU batch crosses to CUDA device global memory. GPU kernels access model tensors through the device memory system; L2 is shared across SMs, L1 is per SM, shared memory is per block, and registers belong to threads. Below, four equal 4-byte quantities make a simplified 16-byte FP32 Adam budget per parameter: weight, gradient, first moment, and second moment. One billion parameters therefore account for 14.90 GiB before activations, workspaces, and allocator overhead.
Registers belong to threads, shared memory to a block, and caches to the hardware. Their bandwidth and latency depend on the GPU and access pattern. For this simplified FP32 Adam budget, weights occupy only one quarter of the parameter-related tensors. Include gradients, optimizer state, and activations when estimating training-step memory.

Compute those two scales before touching a GPU. The first value is a batch input; the second is a parameter-related floor. The script takes the parameter count and feature shape as constants and prints both in human-readable units:

adam-memory-floor.py
parameters = 1_000_000_000
bytes_per_parameter = 4 + 4 + 8
total_bytes = parameters * bytes_per_parameter
feature_bytes = 32 * 128 * 768 * 4

print(f"bytes per parameter: {bytes_per_parameter}")
print(f"parameter-related floor: {total_bytes / (1024 ** 3):.2f} GiB")
print(f"training-sized sequence features: {feature_bytes / (1024 ** 2):.1f} MiB")
print("activations and temporary work: add more memory")
Output
bytes per parameter: 16
parameter-related floor: 14.90 GiB
training-sized sequence features: 12.0 MiB
activations and temporary work: add more memory

A model can load and still hit an failure on its first forward or backward pass. Loading checks that current model state fits. The real batch also needs activations and gradients. Adam's moment tensors are commonly created on the first optimizer step, so that step can fail even after backward succeeds.

For the running text batch, compare halving the batch size with halving the sequence length. Both halve the simple B × T count, but attention storage responds differently. This small calculation makes that distinction visible:

activation-position-budget.py
batch_size = 32
sequence_length = 128
baseline = batch_size * sequence_length

for name, batch, tokens in [
    ("baseline", 32, 128),
    ("half batch", 16, 128),
    ("half length", 32, 64),
]:
    share = batch * tokens / baseline
    print(f"{name:11s}: {share:.0%} of token positions")
Output
baseline   : 100% of token positions
half batch : 50% of token positions
half length: 50% of token positions

The output counts token positions, not full memory. Our mean-pooling classifier has no attention matrix. A later transformer compares each token with every other token; storing those scores can require a tensor shaped (B, H, T, T), where B is batch size, H is the number of attention heads, and T is sequence length. Halving B halves that tensor; halving T quarters it. Memory-efficient attention can avoid storing the full score matrix, so this isn't a universal prediction of peak memory. Feature width D = 768 doesn't change the token-position count.

is another memory and throughput tool, but it isn't a safe synonym for calling .half() on everything. PyTorch's automatic mixed precision (AMP) uses torch.autocast(device_type="cuda") to choose an operation-specific dtype. (16-bit floating point) training may need torch.amp.GradScaler("cuda") to prevent small gradients from underflowing, and some models overflow in FP16.[6] Confirm finite loss, finite gradients, and expected model quality. The dedicated mixed-precision lesson covers that workflow.

TensorFloat-32 (TF32) is a separate choice on Ampere and newer NVIDIA GPUs: eligible operations keep float32 tensor storage but use reduced-precision inputs internally, with FP32 accumulation. It can change numerical results without halving tensor memory. Current PyTorch documentation recommends fp32_precision controls such as torch.backends.cuda.matmul.fp32_precision = "ieee" or "tf32"; mixing these with legacy allow_tf32 controls isn't supported. Record the precision setting when comparing timings, and check accuracy before accepting a faster setting.[7]

Memory pressure answers "does it fit?" It doesn't answer whether the GPU is moving data efficiently while it works. Access order explains that difference.

Let access order decide bandwidth

Two kernels can read the same number of float32 values and still ask the memory system for very different work. On GPUs with compute capability 6.0 or newer, a warp's global-memory requests are serviced in 32-byte segments. Reading adjacent words lets the lanes share segments, an access pattern called coalescing. Reading a column in a row-major matrix can instead make neighboring lanes jump by a of 32 words, so each lane touches a different segment.

Keep eight lanes in the illustration so the addresses stay readable. For a full warp, the adjacent pattern needs four 32-byte transactions when the first word is aligned. The stride-32 pattern can need one transaction per lane. That's why a mathematically identical operation can land on opposite sides of a bandwidth limit.[8]

Shared memory can repair the global access pattern, but it introduces banks, independently accessible storage partitions. In the standard 32-bank layout for 32-bit words, the bank index is the word index modulo 32. For column zero of a 32 × 32 tile, lanes 0, 1, and 2 access word indexes 0, 32, and 64, all in bank 0. These are different addresses, so the accesses serialize. With a 32 × 33 allocation, the indexes become 0, 33, and 66, which map to banks 0, 1, and 2. Padding changes storage layout, not the logical matrix result. Multiple lanes reading the same address are a broadcast case, not this bank-conflict case.[8]

Eight adjacent 4-byte words fit in one aligned 32-byte global-memory segment; column reads at stride 32 words scatter across eight separate segments. In shared memory, column-zero addresses 0, 32, through 224 all map to bank zero and serialize as an eight-way conflict. Padding the tile stride from 32 to 33 maps those lanes to banks zero through seven.
For these eight active lanes and aligned 4-byte words, adjacent reads use all 32 requested bytes; strided reads use only 32 of 256 requested bytes. These ratios describe requested segment use. Caching and other hardware behavior determine actual bus traffic and latency. Shared-memory stride 32 creates an eight-way conflict; stride 33 removes that conflict.

Verify the figure's address mapping directly in Python before analyzing kernel speed. With a base address aligned to 32 bytes, dividing each byte address by 32 identifies its global-memory segment. Taking each word index modulo 32 identifies its shared-memory bank. This CPU trace checks the address math without claiming to measure CUDA transactions:

trace-segments-and-banks.py
lanes = range(8)
adjacent_segments = [(lane * 4) // 32 for lane in lanes]
strided_segments = [(lane * 32 * 4) // 32 for lane in lanes]
unpadded_banks = [(lane * 32) % 32 for lane in lanes]
padded_banks = [(lane * 33) % 32 for lane in lanes]

assert len(set(adjacent_segments)) == 1
assert len(set(strided_segments)) == 8
assert len(set(unpadded_banks)) == 1
assert len(set(padded_banks)) == 8
print("adjacent segments:", adjacent_segments)
print("stride-32 segments:", strided_segments)
print("stride-32 banks:", unpadded_banks)
print("stride-33 banks:", padded_banks)
Output
adjacent segments: [0, 0, 0, 0, 0, 0, 0, 0]
stride-32 segments: [0, 4, 8, 12, 16, 20, 24, 28]
stride-32 banks: [0, 0, 0, 0, 0, 0, 0, 0]
stride-33 banks: [0, 1, 2, 3, 4, 5, 6, 7]

The same stride of 32 is costly in both examples for different reasons: global reads request eight segments, while shared reads target eight distinct words in one bank. A broadcast would repeat the same word address instead.

Use this compact table after you've inspected the access pattern. It suggests changes to test; measurements on the target kernel show which ones help:

Observed accessWhat the hardware seesFirst experiment
adjacent global wordsa few 32-byte transactionskeep the layout and measure another bottleneck
strided global wordsmany transactions for the same useful valuesload coalesced rows, then reorder in shared memory
shared tile stride 32several lanes contend for one bankadd a padding column and remeasure
shared tile stride 33lanes spread across bankskeep padding only if output and end-to-end time improve

The table explains why "put it in shared memory" isn't a complete optimization. Shared memory helps when it removes global traffic or fixes an access pattern; bank conflicts can give that shortcut a new bottleneck.[8]

Memory layout answers where time can go. Next, check whether arithmetic or bytes set the upper bound.

Choose the next optimization with a bound

Suppose a kernel performs 1,000 floating-point operations (FLOPs) and transfers 4,000 bytes to or from device memory. That's 0.25 FLOPs/byte: little arithmetic for each byte fetched. This ratio is called arithmetic intensity:

I=floating-point operationsbytes movedI = \frac{\text{floating-point operations}}{\text{bytes moved}}

Double the bytes without changing the math and intensity falls to 0.125. The Roofline performance model combines this ratio with two hardware limits: arithmetic per second and bytes per second. Its upper bound is the smaller of the compute ceiling and bandwidth multiplied by intensity. Count bytes and bandwidth at the same memory level; a device-memory Roofline uses traffic beyond the caches, not every load instruction.[9][10]

On a device with a 100 GFLOP/s compute ceiling and 40 GB/s memory bandwidth, intensity 0.25 gives a bandwidth ceiling of 40 × 0.25 = 10 GFLOP/s, making the workload memory-bound at 10 GFLOP/s.

When intensity is low and measured bandwidth is high, adding more arithmetic will not make the kernel faster. Reuse data, reduce bytes, or make requests coalesced first. When intensity is high but achieved floating-point rate stays far below the compute ceiling, inspect instruction choice, layout, and launch resources instead.

Occupancy is the fraction of an SM's maximum resident warps that are currently resident. A resident warp holds execution resources even while waiting for data. Having other ready warps helps the scheduler hide that wait, but 100% occupancy isn't a target by itself. Registers and shared memory are finite; using more of either can reduce the number of resident blocks. A kernel with lower occupancy can still win if its tiles reuse data or its threads do more useful work. Treat occupancy as a constraint and a clue, then let measured throughput and stall reasons decide.[8]

NVIDIA's Assess, Parallelize, Optimize, Deploy (APOD) cycle gives the work a safe rhythm. At this level, combine its Parallelize and Optimize stages into one controlled change:

  1. Measure (Assess): profile a representative shape and record end-to-end time, kernel time, output, and memory behavior.
  2. Change (Parallelize, then Optimize): alter access order, reuse, dtype, launch shape, or transfer behavior because the evidence points there.
  3. Verify, Deploy, and repeat: compare output with a trusted reference, rerun the same measurement, keep the change only when the whole step improves, then start the next pass from a fresh assessment.

Pick the tool from the question. Nsight Systems answers whether the CPU feeds the GPU and where transfers, kernels, streams, or host waits leave gaps. Nsight Compute answers what one kernel does with its launch resources, memory hierarchy, and compute or bandwidth ceilings. Compute Sanitizer checks custom CUDA work for illegal memory access, shared-memory hazards, and synchronization errors. For a custom kernel, combine those checks with reference-output comparisons; neither alone proves the algorithm is correct.[11][10][12]

QuestionFirst toolEvidence to read
Is the step CPU-, transfer-, or synchronization-bound?Nsight Systems (nsys profile --trace=cuda)CUDA API and GPU workload timelines, H2D copies, kernels, streams, and idle gaps
Is one kernel limited by bytes, math, or launch resources?Nsight Compute (ncu)launch statistics, SpeedOfLight, Roofline, cache traffic, bank conflicts, and warp stalls
Did a custom kernel produce invalid work?Compute Sanitizermemcheck, racecheck, and synccheck reports with source, block, and thread context

Run correctness checks before comparing numbers. Once the bound and the failure surface are named, the diagnosis table can point to a narrow next action instead of turning every slowdown into a memory tweak.

CUDA's default async launch makes timing another measurement that is easy to get wrong.

Measure asynchronous work honestly

PyTorch queues CUDA operations asynchronously by default. The CPU can finish a Python call while its GPU kernels are still waiting or running. Operations in the same CUDA stream keep their order, so later GPU work sees correct results without a host wait. A host-visible value or explicit synchronization makes the CPU wait.[7]

Suppose a larger classifier's forward pass takes 40 ms on the GPU, while Python needs only 2 ms to queue it. Those timings raise a direct question: what does a host timer around the call report? It measures the caller's elapsed time, which may end before device work completes.

On one elapsed-time axis, Python enqueues CUDA work in 2 ms and returns while device work continues. CUDA events span 40 ms on the GPU stream. A synchronized host timer can report 42 ms in this example because it includes caller overhead and waiting; submission can overlap execution, so 2 plus 40 is not a general rule.
In this scenario, submission takes 2 ms while events span 40 ms on the device stream, so an unsynchronized timer understates completion time roughly twentyfold. A synchronized host timer includes caller overhead and waiting, whereas CUDA events measure a stream interval. They answer different questions and do not have to match.

The naive timer stops at 2 ms while the GPU continues. Events record timestamps on a CUDA stream, an ordered queue of device work. Their interval includes work and any gaps between the events on that stream; it isn't automatically the sum of individual kernel durations. A synchronized host timer can also include Python overhead, so it does not have to match event time exactly.

Warm up first because initial calls may include library setup or kernel selection. This script uses PyTorch 2.9 or later and times ten matrix multiplications. On CUDA it prints three timings for the same loop: host submission, host completion, and the event interval. On CPU it uses a completed host timer. It selects IEEE FP32 matmul on CUDA and checks against a CPU reference. Input allocation, warmup, transfers, and the correctness check stay outside the timed region; result allocation and Python loop overhead remain inside:

honest-matmul-timing.py
import time

import torch

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
torch.manual_seed(7)
x_cpu = torch.randn(256, 256, dtype=torch.float32)
x = x_cpu.to(device)
if device.type == "cuda":
    torch.backends.cuda.matmul.fp32_precision = "ieee"

for _ in range(3):
    result = x @ x

if device.type == "cuda":
    torch.cuda.synchronize()
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    start.record()
    host_start = time.perf_counter()
    for _ in range(10):
        result = x @ x
    enqueue_ms = (time.perf_counter() - host_start) * 1000
    end.record()
    end.synchronize()
    completed_ms = (time.perf_counter() - host_start) * 1000
    elapsed_ms = start.elapsed_time(end)
    print(f"mean host submission: {enqueue_ms / 10:.3f} ms")
    print(f"mean host completion: {completed_ms / 10:.3f} ms")
    print("CUDA matmul precision:", torch.backends.cuda.matmul.fp32_precision)
else:
    start_time = time.perf_counter()
    for _ in range(10):
        result = x @ x
    elapsed_ms = (time.perf_counter() - start_time) * 1000

torch.testing.assert_close(result.cpu(), x_cpu @ x_cpu, rtol=1e-4, atol=1e-4)
print("timing backend:", device.type)
print("PyTorch:", torch.__version__)
print("dtype:", x.dtype)
print("result shape:", tuple(result.shape))
interval = "CUDA stream interval" if device.type == "cuda" else "completed CPU loop"
print(f"mean {interval}: {elapsed_ms / 10:.3f} ms")
print("completed timing is positive:", elapsed_ms > 0)

The event interval includes the work and gaps between its two timestamps. The host completion timer includes recording and waiting for the end event; its submission timer stops before that wait. These tiny matrices may be dominated by Python and launch overhead, so a run does not have to reproduce the diagram's 20-fold gap. Record the device name from the setup check and repeat with representative shapes before drawing a throughput conclusion. Passing assert_close validates this check at its stated tolerance, not every model's accuracy under a precision change.

Common host waits include:

  • loss.item() for a Python scalar
  • tensor.cpu() before CPU analysis
  • tensor.cpu().numpy() before NumPy work
  • printing a CUDA tensor's values
  • torch.cuda.synchronize()

Those operations aren't bugs. They become performance bugs when they sit inside a hot loop more often than reporting or correctness requires. A scalar needed once per full pass is different from a scalar pulled back every step.

Asynchrony also affects error location. A bad kernel may report its error on a later Python line that finally waits for the device. For one debugging reproduction, run with CUDA_LAUNCH_BLOCKING=1 to make CUDA calls synchronous and recover a more useful . Remove it before performance measurement because it changes execution behavior.[7]

When the training step is slow, empty, or dead, start with one question: did the CPU fail to feed the GPU, did the GPU run work that doesn't fit, or did the measurement hide the wait? The diagnosis table turns that question into first actions.

Diagnose setup, memory, and throughput failures

nvidia-smi is a useful first observation, not a kernel profiler. Use it to confirm device visibility, process attachment, rough memory pressure, and utilization samples. Before comparing its memory number with PyTorch, ask which counter includes reserved-but-unused allocator blocks: PyTorch's caching allocator can hold blocks for reuse, so nvidia-smi may show more memory than live tensors occupy.[2][7]

PyTorch separates two allocator views:

  • torch.cuda.memory_allocated() counts memory occupied by live tensors.
  • torch.cuda.memory_reserved() counts the larger pool managed by PyTorch's caching allocator.

torch.cuda.empty_cache() releases unused cached blocks for other applications. It doesn't free live tensors or shrink the workload's required memory. Cached blocks are normally reusable by the same job already; repeatedly clearing them can add allocation overhead. nvidia-smi also includes the CUDA context, which PyTorch's allocator doesn't count, so it does not have to equal memory_reserved().[2][13]

Use symptom, cause, and next action together. Choose the first row that matches observed evidence instead of tuning every knob at once:

SymptomLikely causeFirst action
torch.cuda.is_available() is Falsedriver, visibility, or PyTorch build mismatchcompare nvidia-smi, torch.version.cuda, and process visibility
forward says tensors are on different devicesmodel and one batch field disagreemove every tensor used by model or loss to model device
model loads, backward OOMsactivations, gradients, optimizer state, or workspace exceed free memoryadd up weights, gradients, and optimizer state; if they fit, lower per-step batch size, then sequence length; read allocation size in OOM message
effective batch must stay largesmaller steps change optimization batchaccumulate gradients across several smaller steps and scale loss correctly
loss or gradients become not-a-number (NaN) under FP16numerical overflow or invalid mixed-precision pathdisable AMP; if the NaN goes away, the precision path is implicated; inspect dtypes, scaling, and any changed kernels with finite-value checks
host timer says a kernel took almost zero timeCPU timed enqueue onlywarm up and use CUDA events or explicit synchronization
memory bar is high but examples per second are lowallocation isn't utilization; data, copies, sync, or tiny kernels may stallinspect dataloading and host waits before blaming matrix kernels
GPU utilization repeatedly falls to zeropossible input starvation, host waits, or brief work missed by samplingprofile data loading, transfer cadence, and synchronization gaps
CUDA error points at an innocent later lineasynchronous error surfaced at next waitreproduce once with CUDA_LAUNCH_BLOCKING=1

Pinned host memory can speed H2D copies. DataLoader(..., pin_memory=True) prepares CPU tensors in page-locked memory, and .to(device, non_blocking=True) can let the host continue without waiting for each transfer. This doesn't by itself overlap the copy with a later kernel in the same stream: their order is preserved. Actual copy/compute overlap requires suitable hardware, separate streams, and correct dependencies. Don't modify a pinned source buffer until its asynchronous copy finishes.[14]

The next snippet is a host-to-device transfer fragment, not a full script. It assumes a CPU dataset yielding (features, labels) and a model already exist. Deriving the device from the model gives an explicit index such as cuda:0, not an unindexed cuda specification:

pinned-batch-transfer.py
from torch.utils.data import DataLoader

device = next(model.parameters()).device
loader = DataLoader(dataset, batch_size=32, pin_memory=(device.type == "cuda"))

for features, labels in loader:
    features = features.to(device, non_blocking=True)
    labels = labels.to(device, non_blocking=True)
    assert features.device == device and labels.device == device
    # Forward, backward, and optimizer work stay on device.

Measure before and after. Pinned memory and non-blocking copies help a transfer bottleneck; they won't fix a kernel that's already compute-bound or an OOM caused by live tensors. A faster copy doesn't change the amount of model state that must fit.

Trace the training step end-to-end

Treat the complete training step as a reproducible GPU smoke test. "GPU works" hides setup, shape, placement, memory, and timing in one unverifiable claim. Record evidence for each stage.

  1. Check shapes: batch (4, 8, 16), pooled vectors (4, 16), logits (4, 3).
  2. Run nvidia-smi and the PyTorch environment check. Record driver, PyTorch runtime, device name, and availability separately.
  3. Run the training step. Confirm model, features, labels, logits, and loss use one device until reporting.
  4. Deliberately leave features on CPU while model is on CUDA. Capture the device-mismatch symptom, then restore .to(device).
  5. Time ten matrix multiplies with CUDA events. Compare that result with a naive host timer around the same loop.
  6. Write the terms that would make a larger run OOM: weights, activations, gradients, optimizer state, workspaces, and allocator overhead.

Compare expected output with this table, then fix the first stage that fails. A later symptom can be downstream noise:

EvidencePass conditionIf it fails
shape trace(4, 8, 16) → (4, 16) → (4, 3)revisit reduction axis or linear-layer input width
device traceevery training tensor matches model devicemove missing batch field before operation that uses it
timing traceevent reports completed GPU workadd warmup and synchronization after end event
memory tracefull training footprint is namedadd activations, gradients, optimizer state, and temporary work

Check the reasoning without running code. Each answer should name a mechanism, not only repeat a command:

Carry shape, device, and timing forward

A CUDA training step depends on three invariants:

  • Shape: each axis still means what the model expects.
  • Device: every tensor used together lives on a compatible device.
  • Time: measurements include the GPU work whose latency you intend to report.

The CPU prepares data and launches work. CUDA maps kernels onto grids, blocks, warps, and SMs. Device memory holds long-lived training state, while kernels use caches, shared memory, and registers for smaller working sets. Copies and synchronization are explicit costs, so placement, OOM diagnosis, and timing all follow from the same execution path.

Place the sequence batch, check the driver version against a usable PyTorch build, and see whether a slow step is waiting on bytes, math, or a timer that returned too early. If a run fails, inspect shape, then device, then time.

The Mac lesson keeps those checks but changes the hardware picture: Apple silicon doesn't give you a separate VRAM pool copied over PCIe.

Complete the lesson

Mastery Check

Answer every question, then check your score. Score 75% or higher to mark this lesson complete.

1.A one-dimensional CUDA kernel assigns one thread per array position, starting at index 0. The caller launches the fewest 8-thread blocks that cover all 19 positions. Which block and local thread own index 18, and how many launched indexes fall outside the array?
2.Only physical GPU 5 is exposed, as logical cuda:0 in this process. Trace A has model/features on cuda:0 and labels on CPU. Trace B has model/labels on cuda:0 and features on CPU. Which paired transfers preserve a GPU forward and loss?
3.A float32 tensor has shape (7, 11, 20) in (sequences, tokens, features) order. A classifier averages the tokens into one vector per sequence and applies Linear(20, 5). What are the shapes of the pooled tensor and the logits, and how many bytes do the input values occupy?
4.A float32 tensor contains 1,000,000 values. An eligible operation changes from IEEE FP32 arithmetic to TF32 while the tensor dtype stays float32. Which storage count and required validation fit?
5.A model has 300 million parameters. Count 4 bytes of weights, 4 of gradients and 8 of Adam moments per parameter. The device has 6 GiB. What conclusion follows before activations and workspaces?
6.In one load, the 32 lanes of a warp read consecutive 4-byte global words starting at byte address 80, one word per lane. In a separate load, the same lanes read shared words starting at word 1 with a stride of 38 words. Use the standard 32-bank layout for 4-byte words, with word 0 in bank 0. How many aligned 32-byte global segments does the warp touch, and how do the shared words map to banks?
7.A block launches 8 threads for the 6 input values [5, 1, 4, 2, 8, 0]. Shared memory has one slot per thread. After the block barrier, thread 0 sums all 8 slots. How should the two tail threads handle their slots, and what mean should the block return for the input?
8.A warmed modeled run records a CUDA start event at 5 ms and end event at 23 ms. Its host timer starts at 4 ms and stops after the scalar is ready at 24 ms. Which event interval and completed host interval are being reported?
9.Suppose an allocator model has 5 GiB live tensors, 2 GiB unused cache and 1 GiB context outside the allocator. Assume no other memory. After all unused cache is released, what are allocated, reserved and driver-visible modeled totals?
10.Suppose a kernel performs 2,400 FLOPs and moves 600 bytes beyond cache. At the same memory level, bandwidth is 60 GB/s and the compute ceiling is 200 GFLOP/s. A larger tile lowers occupancy but reduces those bytes further. Which bound and conclusion follow?

10 questions remaining.

References

CUDA Programming Guide

NVIDIA · 2026

https://docs.nvidia.com/cuda/cuda-programming-guide/

nvidia-smi documentation

NVIDIA · 2026

https://docs.nvidia.com/deploy/nvidia-smi/index.html

Get Started.

PyTorch Contributors · 2026

https://pytorch.org/get-started/locally/

Optimizing Model Parameters

PyTorch Contributors · 2026 · Official tutorial

https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html

Adam

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html

Automatic Mixed Precision package - torch.amp

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/2.14/amp.html

CUDA semantics

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/2.14/notes/cuda.html

CUDA C++ Best Practices Guide

NVIDIA · 2026 · Official documentation

https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/

Roofline: An Insightful Visual Performance Model for Multicore Architectures

Williams, S., Waterman, A., & Patterson, D. · 2009

https://doi.org/10.1145/1498765.1498785

Nsight Compute Profiling Guide

NVIDIA · 2026 · Official documentation

https://docs.nvidia.com/nsight-compute/ProfilingGuide/

Nsight Systems User Guide

NVIDIA · 2026 · Official documentation

https://docs.nvidia.com/nsight-systems/UserGuide/

Compute Sanitizer

NVIDIA · 2026 · Official documentation

https://docs.nvidia.com/compute-sanitizer/ComputeSanitizer/

Why reserved_memory+allocated memory is bigger than the memory shown in nvidia-smi?

PyTorch Forums, reply by ptrblck · 2021

https://discuss.pytorch.org/t/why-reserved-memory-allocated-memory-is-bigger-than-the-memory-shown-in-nvidia-smi/138815

A guide on good usage of non_blocking and pin_memory() in PyTorch

PyTorch Contributors · 2026

https://docs.pytorch.org/tutorials/intermediate/pinmem_nonblock.html

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.