Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
The NumPy lesson tracked what each tensor axis means: examples, token positions, and features. Reuse that shape reasoning for a small sequence classifier: four sequences, eight token positions, and 16 embedding features give shape (4, 8, 16). The model pools each sequence's tokens and predicts three class scores.
That same tensor also has to live somewhere. An operation can have the right shape and still fail when its tensors disagree about CPU memory versus GPU memory. The same four sequences can be indexed correctly and still sit in the wrong memory, exhaust the device, or look fast because the timer stopped before the GPU finished.
CUDA is NVIDIA's platform and programming model for running parallel work on NVIDIA GPUs. It doesn't turn every Python line into GPU code. Python still runs on the CPU, while PyTorch launches GPU functions called CUDA kernels for tensor operations whose data lives on a CUDA device.[1]
If you train on a Mac with Apple silicon, continue through this lesson for the shared accelerator concepts, then use MPS & Metal for ML on Mac for its backend and memory differences.
Eight tokens onto two blocks
Would the four-sequence miniature automatically run faster on a GPU? Not necessarily. A GPU earns its keep when many items perform similar arithmetic; a tiny tensor, branch-heavy Python loop, or copy on every line can lose to launch and transfer overhead.
A CPU is tuned for low-latency control flow and varied work. A GPU is tuned for high throughput. Large matrix multiplications, convolutions, and attention scores usually offer enough parallel arithmetic to justify the handoff.
| Part of a training step | CPU host job | GPU device job |
|---|---|---|
| data preparation | read, tokenize, pad, and assemble a batch | no work yet |
| device transfer | request a copy | receive tensor data in device memory |
| forward and backward | launch PyTorch operations | execute tensor kernels |
| reporting | request a Python number or file write | finish and return needed values |
The CPU prepares and launches; the GPU executes. Inside one kernel launch, a thread handles one logical slice of work. Threads are grouped into thread blocks, and all blocks for one kernel form a grid. CUDA schedules each whole block onto one streaming multiprocessor (SM), a hardware processor inside the GPU. Several blocks may be active on one SM, and CUDA doesn't promise which block runs first.[1]
Use the eight token positions of the first sequence, not all four sequences and not every feature. Two blocks with four threads each make the indexing concrete. For a one-dimensional grid of one-dimensional blocks, each thread computes a unique global index:
For block 1, thread 2, the global index is 1 × 4 + 2 = 6, so that thread handles token position T6. The Python trace below checks all eight assignments, taking block and thread counts as input and printing each block's token positions:
threads_per_block = 4
blocks = 2
for block_id in range(blocks):
positions = []
for thread_id in range(threads_per_block):
global_id = block_id * threads_per_block + thread_id
positions.append(f"T{global_id}")
print(f"block {block_id}: {positions}")block 0: ['T0', 'T1', 'T2', 'T3']
block 1: ['T4', 'T5', 'T6', 'T7']
The trace has eight useful threads, but hardware groups threads into a larger unit. An NVIDIA CUDA warp contains 32 threads from one block. Warp lanes execute through a single-instruction, multiple-thread (SIMT) model. If lanes take different branches, CUDA masks inactive lanes while it executes each required path, so divergence can waste throughput.[1]
The four-thread teaching block is legal, and it still occupies a full warp. CUDA fills that warp with consecutive thread ids from the same block, so 4 lanes process token positions and 28 lanes sit unused for the whole launch. A production kernel should choose its block size to keep more lanes active.
Read the hierarchy from a kernel launch downward: its grid contains blocks; each block is scheduled on one SM and advances in warps; each thread owns its index. A PyTorch operation can launch several kernels, and a shape-only view can launch none. Use the table to name what each level contributes:
| Level | Meaning | Beginner question |
|---|---|---|
| PyTorch operation | a request such as x * 2 or x @ w | Is the tensor on CUDA? |
| kernel | GPU function launched to perform part of the work | Is the work large enough to justify a launch? |
| grid | every block in one launch | How much work exists? |
| block | threads that run on one SM and can cooperate | Which slice of work stays together? |
| warp | 32 threads scheduled together | Are branches, or a short last warp, leaving lanes idle? |
| thread | one logical worker with its own index | Which data item does it handle? |
PyTorch's matrix-multiply kernels use optimized tiling, not one thread per token. Treat the eight-position trace as a map for CUDA's indexing and scheduling terms, not as a model of the library kernel.
What if there are ten positions instead of eight? Two four-thread blocks miss the last two positions; three blocks create twelve threads. The final two threads need a bounds check before reading or writing an array. This CPU simulation checks both coverage and the unused tail:
positions = 10
threads_per_block = 4
blocks = (positions + threads_per_block - 1) // threads_per_block
visited = []
skipped = []
for block_id in range(blocks):
for thread_id in range(threads_per_block):
i = block_id * threads_per_block + thread_id
if i < positions:
visited.append(i)
else:
skipped.append(i)
assert visited == list(range(positions))
print("blocks:", blocks)
print("guarded indexes:", skipped)blocks: 3
guarded indexes: [10, 11]Removing the guard would attempt indexes 10 and 11, beyond the ten-element array. These are ordinary Python simulations of CUDA indexing, not GPU kernels. PyTorch handles the indexing and bounds checks for the tensor operations used below.
Those launches only happen if this process can see a CUDA device. The next check is whether the driver, the PyTorch build, and the GPU agree.
Check driver, PyTorch build, and device
The phrase "my CUDA version" separates several layers of the software stack, and each command checks a different piece. The NVIDIA System Management Interface command, nvidia-smi, reads driver and GPU state. nvcc is NVIDIA's CUDA compiler:
| Signal | What it tells you | What to check next |
|---|---|---|
nvidia-smi | NVIDIA driver can see a GPU. The CUDA Version header (documented as CUDA UMD Version; the older CUDA Version label is deprecated) is the latest CUDA version that driver supports | which CUDA toolkit is installed, or whether this Python has a CUDA-enabled PyTorch build |
torch.__version__ | installed PyTorch version and often its build suffix | whether GPU access works |
torch.version.cuda | CUDA version used to build this PyTorch package, or None for a build without CUDA | whether driver and device are reachable |
torch.cuda.is_available() | CUDA is currently available to this process | whether a particular kernel can execute, fits, or runs fast |
nvcc --version | version of local CUDA toolkit compiler, when installed | which runtime a prebuilt PyTorch package uses |
The CUDA UMD Version describes driver capability, not an installed compiler. A machine can run a prebuilt CUDA-enabled PyTorch package without having nvcc installed.[2] It can't prove that this Python process loaded a CUDA-enabled PyTorch build.
PyTorch's installer offers builds for supported compute platforms, and its verification step uses torch.cuda.is_available().[3] For ordinary prebuilt use, a local toolkit and nvcc matter when you build PyTorch from source or compile custom CUDA extensions. Version strings don't need to match; the driver must support the runtime used by the package.
Run the driver check first, then inspect the Python environment. The shell command produces driver output, PyTorch version and runtime, availability, device name, capability, and architectures when CUDA is reachable:
nvidia-smi
python3 - <<'PY'
import torch
print("PyTorch:", torch.__version__)
print("PyTorch CUDA runtime:", torch.version.cuda)
print("CUDA available:", torch.cuda.is_available())
if torch.cuda.is_available():
print("device:", torch.cuda.get_device_name(0))
print("compute capability:", torch.cuda.get_device_capability(0))
print("build architectures:", torch.cuda.get_arch_list())
PYThe compute capability pair describes hardware features supported by a GPU generation. get_arch_list() reports architectures included in the PyTorch build. Read the first disagreement, not the most alarming line:
| Observed state | Likely layer | Next check |
|---|---|---|
nvidia-smi fails | driver, hardware, or container access | driver installation and device exposure |
nvidia-smi works, torch.version.cuda is None | PyTorch package wasn't built for CUDA | select the CUDA-enabled package for this environment |
PyTorch has a CUDA runtime, but availability is False | driver compatibility or process visibility | driver support, container GPU access, CUDA_VISIBLE_DEVICES |
availability is True, but kernel reports unsupported architecture | installed binary doesn't support this GPU | check support in PyTorch and any custom extension; older GPUs can also lose support |
availability is True and device name is correct | basic stack works | test placement, memory, and timing |
CUDA_VISIBLE_DEVICES changes the GPU indexes visible inside the process. If physical GPU 1 is the only visible device, PyTorch may call it cuda:0. Inspect tensor.device and device names inside the running process instead of assuming host indexes survive a mask.
Once the stack answers "yes," move the miniature sequence batch and check the result. GPU visibility confirms device access; validate tensor placement, memory use, outputs, and timing in the training step below.
Move the sequence batch once
Return to the (4, 8, 16) sequence tensor. What changes during .to(device): its shape or its residence? Shape stays constant. Residence changes from host memory to CUDA device memory.
On a discrete NVIDIA GPU, a host-to-device (H2D) transfer copies data from CPU memory to GPU memory. Device-to-host (D2H) is the return path. Keep model weights, batch tensors, activations, gradients, and optimizer state on the GPU through the hot part of a training step. Bring back the small value needed for reporting.
The runnable step uses random feature values so it needs no dataset download. It tests execution, not classifier quality. It averages eight token vectors into one vector per sequence, then scores three classes indexed 0, 1, and 2.
nn.Linear(16, 3) multiplies each 16-feature vector by learned weights and adds a bias, producing three scores called logits. Cross-entropy turns those scores and the correct label into one error value, the loss. backward() computes how each weight affects that loss; those derivatives are gradients. Stochastic gradient descent (SGD) updates weights opposite the gradient, aiming to reduce the loss.[4] The later PyTorch training lesson derives the loop; here, watch where its tensors live.
Checking shape and device placement confirms that (4, 8, 16) becomes sequence vectors (4, 16) and logits (4, 3). On CUDA, model and batch agree; on CPU, the same shape and loss checks still run:
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(7)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = nn.Linear(16, 3).to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
cpu_batch = {
"token_features": torch.randn(4, 8, 16),
"labels": torch.tensor([0, 2, 1, 0]),
}
batch = {name: tensor.to(device) for name, tensor in cpu_batch.items()}
optimizer.zero_grad()
seq_vectors = batch["token_features"].mean(dim=1)
logits = model(seq_vectors)
loss = F.cross_entropy(logits, batch["labels"])
loss.backward()
assert torch.isfinite(loss).item()
assert all(parameter.grad is not None and torch.isfinite(parameter.grad).all().item()
for parameter in model.parameters())
optimizer.step()
model_device = next(model.parameters()).device
assert all(tensor.device == model_device for tensor in batch.values())
assert logits.device == loss.device == model_device
assert seq_vectors.shape == (4, 16) and logits.shape == (4, 3)
reported_loss = loss.detach().cpu().item()
print("selected accelerator:", device.type == "cuda")
print("model and batch agree:", model_device == batch["token_features"].device)
print("sequence vectors:", tuple(seq_vectors.shape))
print("logits:", tuple(logits.shape))
print("finite loss:", -float("inf") < reported_loss < float("inf"))On an accessible NVIDIA GPU, the first two lines should report True. A CPU-only machine reports False for the accelerator line but still validates tensor and training logic. The finite-value assertions check this smoke test before the update; their .item() calls can wait for GPU work, so keep them outside a throughput measurement. If a shape fails, debug the reduction or linear layer first; a device error needs a different branch.
Device placement applies to every tensor an operation touches. Moving features but leaving labels on CPU still fails during a CUDA loss calculation. Deliberately leave the features on CPU in the next fragment; on a CUDA machine this raises a device error, while a CPU-only machine prints the fallback message:
import torch
import torch.nn as nn
if torch.cuda.is_available():
cuda_model = nn.Linear(16, 3).to("cuda")
cpu_features = torch.randn(4, 16)
try:
cuda_model(cpu_features)
except RuntimeError as error:
if "device" not in str(error).lower():
raise
print("device mismatch:", str(error))
else:
raise AssertionError("expected a mixed-device forward pass to fail")
else:
print("CUDA unavailable: real mixed-device failure wasn't executed")The symptom is an error saying tensors or arguments aren't on the same device. Cause is placement, not shape. Move every participating tensor to the model's device before the forward or loss call. Synchronizing won't repair a tensor stored on the wrong device.
The miniature step can succeed while a training-sized batch still fails. Shapes and device placement answer whether an operation is legal; memory pressure decides whether the full forward and backward pass can coexist.
Count memory before the first big batch
The small step fits. Before increasing batch size, ask which values must survive between steps and which exist only while one kernel runs. CUDA exposes several storage levels, and they aren't interchangeable:
| Storage | Scope | Typical training data |
|---|---|---|
| host RAM | CPU process | dataset objects, decoded examples, CPU batches |
| device global memory, often called video RAM (VRAM) or high-bandwidth memory (HBM) | all SMs on one GPU | weights, activations, gradients, optimizer state |
| L2 and L1 caches | GPU hardware | recently accessed device data |
| shared memory | threads in one block | tiles reused inside a kernel |
| registers | one thread | counters, addresses, and small working values |
PyTorch allocates model tensors in device global memory. Optimized kernels decide when to reuse tiles through caches, shared memory, or registers. Calling .to("cuda") doesn't place a whole tensor in a register or in shared memory. An optimizer adds its own long-lived state during training.
The miniature batch itself contains 4 × 8 × 16 = 512 float32 values, only 2,048 bytes. A larger (32, 128, 768) float32 batch contains 32 × 128 × 768 × 4 = 12,582,912 bytes, exactly 12.0 MiB of input features. That input is only one term in a training step:
The example uses SGD without momentum, which doesn't keep per-parameter running statistics. To see why the optimizer choice changes memory, consider Adam, an optimizer that tracks two statistics for each parameter: a running average of gradients and a running average of squared gradients.[5]
For a simplified full-precision Adam budget, count 4 bytes for each weight, 4 for its gradient, and 8 for its two statistics. One billion parameters therefore require 16 billion bytes for these tensors when they're all present. Activations, temporary buffers, allocator overhead, and any extra parameter copies still sit outside this accounting.

Compute those two scales before touching a GPU. The first value is a batch input; the second is a parameter-related floor. The script takes the parameter count and feature shape as constants and prints both in human-readable units:
parameters = 1_000_000_000
bytes_per_parameter = 4 + 4 + 8
total_bytes = parameters * bytes_per_parameter
feature_bytes = 32 * 128 * 768 * 4
print(f"bytes per parameter: {bytes_per_parameter}")
print(f"parameter-related floor: {total_bytes / (1024 ** 3):.2f} GiB")
print(f"training-sized sequence features: {feature_bytes / (1024 ** 2):.1f} MiB")
print("activations and temporary work: add more memory")bytes per parameter: 16
parameter-related floor: 14.90 GiB
training-sized sequence features: 12.0 MiB
activations and temporary work: add more memoryA model can load and still hit an out-of-memory (OOM) failure on its first forward or backward pass. Loading checks that current model state fits. The real batch also needs activations and gradients. Adam's moment tensors are commonly created on the first optimizer step, so that step can fail even after backward succeeds.
For the running text batch, compare halving the batch size with halving the sequence length. Both halve the simple B × T count, but attention storage responds differently. This small calculation makes that distinction visible:
batch_size = 32
sequence_length = 128
baseline = batch_size * sequence_length
for name, batch, tokens in [
("baseline", 32, 128),
("half batch", 16, 128),
("half length", 32, 64),
]:
share = batch * tokens / baseline
print(f"{name:11s}: {share:.0%} of token positions")baseline : 100% of token positions
half batch : 50% of token positions
half length: 50% of token positionsThe output counts token positions, not full memory. Our mean-pooling classifier has no attention matrix. A later transformer compares each token with every other token; storing those scores can require a tensor shaped (B, H, T, T), where B is batch size, H is the number of attention heads, and T is sequence length. Halving B halves that tensor; halving T quarters it. Memory-efficient attention can avoid storing the full score matrix, so this isn't a universal prediction of peak memory. Feature width D = 768 doesn't change the token-position count.
Mixed precision is another memory and throughput tool, but it isn't a safe synonym for calling .half() on everything. PyTorch's automatic mixed precision (AMP) uses torch.autocast(device_type="cuda") to choose an operation-specific dtype. FP16 (16-bit floating point) training may need torch.amp.GradScaler("cuda") to prevent small gradients from underflowing, and some models overflow in FP16.[6] Confirm finite loss, finite gradients, and expected model quality. The dedicated mixed-precision lesson covers that workflow.
TensorFloat-32 (TF32) is a separate choice on Ampere and newer NVIDIA GPUs: eligible operations keep float32 tensor storage but use reduced-precision inputs internally, with FP32 accumulation. It can change numerical results without halving tensor memory. Current PyTorch documentation recommends fp32_precision controls such as torch.backends.cuda.matmul.fp32_precision = "ieee" or "tf32"; mixing these with legacy allow_tf32 controls isn't supported. Record the precision setting when comparing timings, and check accuracy before accepting a faster setting.[7]
Memory pressure answers "does it fit?" It doesn't answer whether the GPU is moving data efficiently while it works. Access order explains that difference.
Let access order decide bandwidth
Two kernels can read the same number of float32 values and still ask the memory system for very different work. On GPUs with compute capability 6.0 or newer, a warp's global-memory requests are serviced in 32-byte segments. Reading adjacent words lets the lanes share segments, an access pattern called coalescing. Reading a column in a row-major matrix can instead make neighboring lanes jump by a stride of 32 words, so each lane touches a different segment.
Keep eight lanes in the illustration so the addresses stay readable. For a full warp, the adjacent pattern needs four 32-byte transactions when the first word is aligned. The stride-32 pattern can need one transaction per lane. That's why a mathematically identical operation can land on opposite sides of a bandwidth limit.[8]
Shared memory can repair the global access pattern, but it introduces banks, independently accessible storage partitions. In the standard 32-bank layout for 32-bit words, the bank index is the word index modulo 32. For column zero of a 32 × 32 tile, lanes 0, 1, and 2 access word indexes 0, 32, and 64, all in bank 0. These are different addresses, so the accesses serialize. With a 32 × 33 allocation, the indexes become 0, 33, and 66, which map to banks 0, 1, and 2. Padding changes storage layout, not the logical matrix result. Multiple lanes reading the same address are a broadcast case, not this bank-conflict case.[8]

Verify the figure's address mapping directly in Python before analyzing kernel speed. With a base address aligned to 32 bytes, dividing each byte address by 32 identifies its global-memory segment. Taking each word index modulo 32 identifies its shared-memory bank. This CPU trace checks the address math without claiming to measure CUDA transactions:
lanes = range(8)
adjacent_segments = [(lane * 4) // 32 for lane in lanes]
strided_segments = [(lane * 32 * 4) // 32 for lane in lanes]
unpadded_banks = [(lane * 32) % 32 for lane in lanes]
padded_banks = [(lane * 33) % 32 for lane in lanes]
assert len(set(adjacent_segments)) == 1
assert len(set(strided_segments)) == 8
assert len(set(unpadded_banks)) == 1
assert len(set(padded_banks)) == 8
print("adjacent segments:", adjacent_segments)
print("stride-32 segments:", strided_segments)
print("stride-32 banks:", unpadded_banks)
print("stride-33 banks:", padded_banks)adjacent segments: [0, 0, 0, 0, 0, 0, 0, 0]
stride-32 segments: [0, 4, 8, 12, 16, 20, 24, 28]
stride-32 banks: [0, 0, 0, 0, 0, 0, 0, 0]
stride-33 banks: [0, 1, 2, 3, 4, 5, 6, 7]The same stride of 32 is costly in both examples for different reasons: global reads request eight segments, while shared reads target eight distinct words in one bank. A broadcast would repeat the same word address instead.
Use this compact table after you've inspected the access pattern. It suggests changes to test; measurements on the target kernel show which ones help:
| Observed access | What the hardware sees | First experiment |
|---|---|---|
| adjacent global words | a few 32-byte transactions | keep the layout and measure another bottleneck |
| strided global words | many transactions for the same useful values | load coalesced rows, then reorder in shared memory |
| shared tile stride 32 | several lanes contend for one bank | add a padding column and remeasure |
| shared tile stride 33 | lanes spread across banks | keep padding only if output and end-to-end time improve |
The table explains why "put it in shared memory" isn't a complete optimization. Shared memory helps when it removes global traffic or fixes an access pattern; bank conflicts can give that shortcut a new bottleneck.[8]
Memory layout answers where time can go. Next, check whether arithmetic or bytes set the upper bound.
Choose the next optimization with a bound
Suppose a kernel performs 1,000 floating-point operations (FLOPs) and transfers 4,000 bytes to or from device memory. That's 0.25 FLOPs/byte: little arithmetic for each byte fetched. This ratio is called arithmetic intensity:
Double the bytes without changing the math and intensity falls to 0.125. The Roofline performance model combines this ratio with two hardware limits: arithmetic per second and bytes per second. Its upper bound is the smaller of the compute ceiling and bandwidth multiplied by intensity. Count bytes and bandwidth at the same memory level; a device-memory Roofline uses traffic beyond the caches, not every load instruction.[9][10]
On a device with a 100 GFLOP/s compute ceiling and 40 GB/s memory bandwidth, intensity 0.25 gives a bandwidth ceiling of 40 × 0.25 = 10 GFLOP/s, making the workload memory-bound at 10 GFLOP/s.
When intensity is low and measured bandwidth is high, adding more arithmetic will not make the kernel faster. Reuse data, reduce bytes, or make requests coalesced first. When intensity is high but achieved floating-point rate stays far below the compute ceiling, inspect instruction choice, layout, and launch resources instead.
Occupancy is the fraction of an SM's maximum resident warps that are currently resident. A resident warp holds execution resources even while waiting for data. Having other ready warps helps the scheduler hide that wait, but 100% occupancy isn't a target by itself. Registers and shared memory are finite; using more of either can reduce the number of resident blocks. A kernel with lower occupancy can still win if its tiles reuse data or its threads do more useful work. Treat occupancy as a constraint and a clue, then let measured throughput and stall reasons decide.[8]
NVIDIA's Assess, Parallelize, Optimize, Deploy (APOD) cycle gives the work a safe rhythm. At this level, combine its Parallelize and Optimize stages into one controlled change:
- Measure (Assess): profile a representative shape and record end-to-end time, kernel time, output, and memory behavior.
- Change (Parallelize, then Optimize): alter access order, reuse, dtype, launch shape, or transfer behavior because the evidence points there.
- Verify, Deploy, and repeat: compare output with a trusted reference, rerun the same measurement, keep the change only when the whole step improves, then start the next pass from a fresh assessment.
Pick the tool from the question. Nsight Systems answers whether the CPU feeds the GPU and where transfers, kernels, streams, or host waits leave gaps. Nsight Compute answers what one kernel does with its launch resources, memory hierarchy, and compute or bandwidth ceilings. Compute Sanitizer checks custom CUDA work for illegal memory access, shared-memory hazards, and synchronization errors. For a custom kernel, combine those checks with reference-output comparisons; neither alone proves the algorithm is correct.[11][10][12]
| Question | First tool | Evidence to read |
|---|---|---|
| Is the step CPU-, transfer-, or synchronization-bound? | Nsight Systems (nsys profile --trace=cuda) | CUDA API and GPU workload timelines, H2D copies, kernels, streams, and idle gaps |
| Is one kernel limited by bytes, math, or launch resources? | Nsight Compute (ncu) | launch statistics, SpeedOfLight, Roofline, cache traffic, bank conflicts, and warp stalls |
| Did a custom kernel produce invalid work? | Compute Sanitizer | memcheck, racecheck, and synccheck reports with source, block, and thread context |
Run correctness checks before comparing numbers. Once the bound and the failure surface are named, the diagnosis table can point to a narrow next action instead of turning every slowdown into a memory tweak.
CUDA's default async launch makes timing another measurement that is easy to get wrong.
Measure asynchronous work honestly
PyTorch queues CUDA operations asynchronously by default. The CPU can finish a Python call while its GPU kernels are still waiting or running. Operations in the same CUDA stream keep their order, so later GPU work sees correct results without a host wait. A host-visible value or explicit synchronization makes the CPU wait.[7]
Suppose a larger classifier's forward pass takes 40 ms on the GPU, while Python needs only 2 ms to queue it. Those timings raise a direct question: what does a host timer around the call report? It measures the caller's elapsed time, which may end before device work completes.

The naive timer stops at 2 ms while the GPU continues. Events record timestamps on a CUDA stream, an ordered queue of device work. Their interval includes work and any gaps between the events on that stream; it isn't automatically the sum of individual kernel durations. A synchronized host timer can also include Python overhead, so it does not have to match event time exactly.
Warm up first because initial calls may include library setup or kernel selection. This script uses PyTorch 2.9 or later and times ten matrix multiplications. On CUDA it prints three timings for the same loop: host submission, host completion, and the event interval. On CPU it uses a completed host timer. It selects IEEE FP32 matmul on CUDA and checks against a CPU reference. Input allocation, warmup, transfers, and the correctness check stay outside the timed region; result allocation and Python loop overhead remain inside:
import time
import torch
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
torch.manual_seed(7)
x_cpu = torch.randn(256, 256, dtype=torch.float32)
x = x_cpu.to(device)
if device.type == "cuda":
torch.backends.cuda.matmul.fp32_precision = "ieee"
for _ in range(3):
result = x @ x
if device.type == "cuda":
torch.cuda.synchronize()
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
host_start = time.perf_counter()
for _ in range(10):
result = x @ x
enqueue_ms = (time.perf_counter() - host_start) * 1000
end.record()
end.synchronize()
completed_ms = (time.perf_counter() - host_start) * 1000
elapsed_ms = start.elapsed_time(end)
print(f"mean host submission: {enqueue_ms / 10:.3f} ms")
print(f"mean host completion: {completed_ms / 10:.3f} ms")
print("CUDA matmul precision:", torch.backends.cuda.matmul.fp32_precision)
else:
start_time = time.perf_counter()
for _ in range(10):
result = x @ x
elapsed_ms = (time.perf_counter() - start_time) * 1000
torch.testing.assert_close(result.cpu(), x_cpu @ x_cpu, rtol=1e-4, atol=1e-4)
print("timing backend:", device.type)
print("PyTorch:", torch.__version__)
print("dtype:", x.dtype)
print("result shape:", tuple(result.shape))
interval = "CUDA stream interval" if device.type == "cuda" else "completed CPU loop"
print(f"mean {interval}: {elapsed_ms / 10:.3f} ms")
print("completed timing is positive:", elapsed_ms > 0)The event interval includes the work and gaps between its two timestamps. The host completion timer includes recording and waiting for the end event; its submission timer stops before that wait. These tiny matrices may be dominated by Python and launch overhead, so a run does not have to reproduce the diagram's 20-fold gap. Record the device name from the setup check and repeat with representative shapes before drawing a throughput conclusion. Passing assert_close validates this check at its stated tolerance, not every model's accuracy under a precision change.
Common host waits include:
loss.item()for a Python scalartensor.cpu()before CPU analysistensor.cpu().numpy()before NumPy work- printing a CUDA tensor's values
torch.cuda.synchronize()
Those operations aren't bugs. They become performance bugs when they sit inside a hot loop more often than reporting or correctness requires. A scalar needed once per full pass is different from a scalar pulled back every step.
Asynchrony also affects error location. A bad kernel may report its error on a later Python line that finally waits for the device. For one debugging reproduction, run with CUDA_LAUNCH_BLOCKING=1 to make CUDA calls synchronous and recover a more useful stack trace. Remove it before performance measurement because it changes execution behavior.[7]
When the training step is slow, empty, or dead, start with one question: did the CPU fail to feed the GPU, did the GPU run work that doesn't fit, or did the measurement hide the wait? The diagnosis table turns that question into first actions.
Diagnose setup, memory, and throughput failures
nvidia-smi is a useful first observation, not a kernel profiler. Use it to confirm device visibility, process attachment, rough memory pressure, and utilization samples. Before comparing its memory number with PyTorch, ask which counter includes reserved-but-unused allocator blocks: PyTorch's caching allocator can hold blocks for reuse, so nvidia-smi may show more memory than live tensors occupy.[2][7]
PyTorch separates two allocator views:
torch.cuda.memory_allocated()counts memory occupied by live tensors.torch.cuda.memory_reserved()counts the larger pool managed by PyTorch's caching allocator.
torch.cuda.empty_cache() releases unused cached blocks for other applications. It doesn't free live tensors or shrink the workload's required memory. Cached blocks are normally reusable by the same job already; repeatedly clearing them can add allocation overhead. nvidia-smi also includes the CUDA context, which PyTorch's allocator doesn't count, so it does not have to equal memory_reserved().[2][13]
Use symptom, cause, and next action together. Choose the first row that matches observed evidence instead of tuning every knob at once:
| Symptom | Likely cause | First action |
|---|---|---|
torch.cuda.is_available() is False | driver, visibility, or PyTorch build mismatch | compare nvidia-smi, torch.version.cuda, and process visibility |
| forward says tensors are on different devices | model and one batch field disagree | move every tensor used by model or loss to model device |
| model loads, backward OOMs | activations, gradients, optimizer state, or workspace exceed free memory | add up weights, gradients, and optimizer state; if they fit, lower per-step batch size, then sequence length; read allocation size in OOM message |
| effective batch must stay large | smaller steps change optimization batch | accumulate gradients across several smaller steps and scale loss correctly |
loss or gradients become not-a-number (NaN) under FP16 | numerical overflow or invalid mixed-precision path | disable AMP; if the NaN goes away, the precision path is implicated; inspect dtypes, scaling, and any changed kernels with finite-value checks |
| host timer says a kernel took almost zero time | CPU timed enqueue only | warm up and use CUDA events or explicit synchronization |
| memory bar is high but examples per second are low | allocation isn't utilization; data, copies, sync, or tiny kernels may stall | inspect dataloading and host waits before blaming matrix kernels |
| GPU utilization repeatedly falls to zero | possible input starvation, host waits, or brief work missed by sampling | profile data loading, transfer cadence, and synchronization gaps |
| CUDA error points at an innocent later line | asynchronous error surfaced at next wait | reproduce once with CUDA_LAUNCH_BLOCKING=1 |
Pinned host memory can speed H2D copies. DataLoader(..., pin_memory=True) prepares CPU tensors in page-locked memory, and .to(device, non_blocking=True) can let the host continue without waiting for each transfer. This doesn't by itself overlap the copy with a later kernel in the same stream: their order is preserved. Actual copy/compute overlap requires suitable hardware, separate streams, and correct dependencies. Don't modify a pinned source buffer until its asynchronous copy finishes.[14]
The next snippet is a host-to-device transfer fragment, not a full script. It assumes a CPU dataset yielding (features, labels) and a model already exist. Deriving the device from the model gives an explicit index such as cuda:0, not an unindexed cuda specification:
from torch.utils.data import DataLoader
device = next(model.parameters()).device
loader = DataLoader(dataset, batch_size=32, pin_memory=(device.type == "cuda"))
for features, labels in loader:
features = features.to(device, non_blocking=True)
labels = labels.to(device, non_blocking=True)
assert features.device == device and labels.device == device
# Forward, backward, and optimizer work stay on device.Measure before and after. Pinned memory and non-blocking copies help a transfer bottleneck; they won't fix a kernel that's already compute-bound or an OOM caused by live tensors. A faster copy doesn't change the amount of model state that must fit.
Trace the training step end-to-end
Treat the complete training step as a reproducible GPU smoke test. "GPU works" hides setup, shape, placement, memory, and timing in one unverifiable claim. Record evidence for each stage.
- Check shapes: batch
(4, 8, 16), pooled vectors(4, 16), logits(4, 3). - Run
nvidia-smiand the PyTorch environment check. Record driver, PyTorch runtime, device name, and availability separately. - Run the training step. Confirm model, features, labels, logits, and loss use one device until reporting.
- Deliberately leave features on CPU while model is on CUDA. Capture the device-mismatch symptom, then restore
.to(device). - Time ten matrix multiplies with CUDA events. Compare that result with a naive host timer around the same loop.
- Write the terms that would make a larger run OOM: weights, activations, gradients, optimizer state, workspaces, and allocator overhead.
Compare expected output with this table, then fix the first stage that fails. A later symptom can be downstream noise:
| Evidence | Pass condition | If it fails |
|---|---|---|
| shape trace | (4, 8, 16) → (4, 16) → (4, 3) | revisit reduction axis or linear-layer input width |
| device trace | every training tensor matches model device | move missing batch field before operation that uses it |
| timing trace | event reports completed GPU work | add warmup and synchronization after end event |
| memory trace | full training footprint is named | add activations, gradients, optimizer state, and temporary work |
Check the reasoning without running code. Each answer should name a mechanism, not only repeat a command:
Why can model(x) return to Python before its CUDA kernels finish?
Answer
PyTorch queues CUDA work asynchronously. Python can continue after enqueueing while the device executes operations in stream order.
Why can a model fit in device memory and then OOM during backward?
Answer
Model loading accounts for current state, but training also needs activations, gradients, optimizer state, temporary workspaces, and allocator overhead.
Why can nvidia-smi show more memory than memory_allocated()?
Answer
PyTorch's caching allocator can reserve unused blocks for fast reuse. memory_allocated() counts live tensors, while nvidia-smi can reflect the larger reserved pool.
What does the CUDA value printed by nvidia-smi mean?
Answer
It reports the latest CUDA version supported by the installed driver (CUDA UMD Version; the older CUDA Version label is deprecated). Check the installed toolkit and this PyTorch environment separately to confirm they can use CUDA.
Carry shape, device, and timing forward
A CUDA training step depends on three invariants:
- Shape: each axis still means what the model expects.
- Device: every tensor used together lives on a compatible device.
- Time: measurements include the GPU work whose latency you intend to report.
The CPU prepares data and launches work. CUDA maps kernels onto grids, blocks, warps, and SMs. Device memory holds long-lived training state, while kernels use caches, shared memory, and registers for smaller working sets. Copies and synchronization are explicit costs, so placement, OOM diagnosis, and timing all follow from the same execution path.
Place the sequence batch, check the driver version against a usable PyTorch build, and see whether a slow step is waiting on bytes, math, or a timer that returned too early. If a run fails, inspect shape, then device, then time.
The Mac lesson keeps those checks but changes the hardware picture: Apple silicon doesn't give you a separate VRAM pool copied over PCIe.