GPU Failure Operations

At 14:03:17 UTC, request r-7812 asks replica summarizer-7f9d6 for another 1.50 GiB. The allocation fails. Node gpu-a17 still answers health checks, peer replicas still serve traffic, and the alert says only CUDA out of memory.

The first receipt is more specific:

There is no safe universal command after this receipt. A request retry, process restart, device quarantine, and node drain act at different scopes. Choose one only after evidence identifies the smallest broken scope.

Why is CUDA out of memory insufficient evidence for draining gpu-a17?

Answer

The error names a failed allocation in one process. It doesn't yet distinguish live application demand, allocator fragmentation, memory held outside PyTorch, a stale process, or a device fault. Draining the node would evict unrelated workloads before that scope is known.

Build an incident receipt before changing state

An incident receipt is a time-aligned record of request, process, device, node, and cluster evidence. Capture it while the failed process still exists. PyTorch can record allocator history and dump a snapshot for memory_viz; its snapshot includes allocator state and OOM events, but only for memory visible to the PyTorch allocator.^{[1]Reference 1Understanding CUDA Memory Usagehttps://docs.pytorch.org/docs/main/torch_cuda_memory}

For r-7812, preservation comes before restart:

1. Remove the replica from application readiness so new work stops arriving.
2. Record request ID, pod UID, container restart count, PID, node, GPU UUID, model revision, allocator backend and configuration, CUDA runtime, driver, and timestamps.
3. Dump the PyTorch memory snapshot if the process still responds.
4. Record device-wide memory, process accounting, ECC counters, clocks, power, temperature, Xid events, and device-plugin health.
5. Copy logs and artifacts to incident storage with access controls.
6. Only then choose containment and repair scope.

This order matters. A process exit destroys allocator history. A device reset can destroy volatile fault context. A node reboot can erase local logs. Preservation is an operational gate, not permission to collect another tenant's prompts, model inputs, or secrets.

The receipt separates several memory quantities:

reserved - allocated is an accounting gap, not a reusable-byte counter. Stronger evidence comes from the snapshot: 6.10 GiB is inactive across split blocks trapped in partly active segments, but no such block exceeds 0.88 GiB, so those fragments can't satisfy a 1.50 GiB request. Meanwhile, the 2.68 GiB device-wide delta is worth attribution, but it isn't large enough to explain the whole failure. Together, these numbers support a working classification of allocator fragmentation under high application demand, with a small outside-PyTorch component. That classification remains a hypothesis until segment and block history confirm why the layout formed.

Keep memory layers separate

GPU memory pressure has four common software layers. Combining them into one “leak” diagnosis produces the wrong repair.

PyTorch explicitly warns that its memory profiler can't see allocations made directly through CUDA APIs. NCCL is a documented example. torch.cuda.device_memory_used() provides a device-wide cross-check, and NVML exposes device and process telemetry for independent attribution.^{[1]Reference 1Understanding CUDA Memory Usagehttps://docs.pytorch.org/docs/main/torch_cuda_memory[2]Reference 2NVML API Reference Guidehttps://docs.nvidia.com/deploy/nvml-api/nvml-api-reference.html}

Don't treat reserved - allocated as wasted memory by definition. The caching allocator holds segments so later allocations avoid expensive device synchronization. Fragmentation becomes plausible when an OOM snapshot shows enough inactive split bytes but no fragment large enough for the failed request. PyTorch allocator configuration can change split behavior, yet tuning it before identifying the workload shape can hide an admission-control bug.^{[3]Reference 3CUDA semantics.https://docs.pytorch.org/docs/2.13/notes/cuda.html}

This receipt assumes PyTorch's native caching allocator. With backend:cudaMallocAsync, some allocator statistics have backend-specific or undefined meaning, and native split-block tuning options don't apply. Record the backend and use its documented evidence instead of copying this inference unchanged.^{[3]Reference 3CUDA semantics.https://docs.pytorch.org/docs/2.13/notes/cuda.html}

Record a finite allocator history

Enable history before the incident window when overhead and privacy policy permit:

The initial _ marks these as private APIs, so pin and validate the PyTorch version used by your fleet. Open the pickle in the local memory_viz page. PyTorch says the browser processes the file locally without uploading it.^{[1]Reference 1Understanding CUDA Memory Usagehttps://docs.pytorch.org/docs/main/torch_cuda_memory}

History is a ring buffer, not an infinite flight recorder. Once max_entries fills, the oldest entries are replaced. Large histories can produce multi-gigabyte snapshots. Set a bounded window, record the start time and entry limit in the receipt, and trigger a dump near the failure. A snapshot with no OOM event may mean capture began too late or history wrapped, not that the alert was false.

A snapshot shows 6.10 GiB across inactive split blocks, but NVML reports 2.68 GiB more device use than PyTorch reserved. What should you conclude?

Answer

Two hypotheses remain. Allocator layout may block the 1.50 GiB request, while another CUDA user owns about 2.68 GiB. Inspect snapshot blocks and attribute device processes or libraries. Neither number alone proves a leak.

Classify beyond memory

An OOM can be the visible end of a different failure. A collective stalls, a peer disappears, requests accumulate, and healthy ranks hold buffers until an allocation fails. Build a parallel device and node view.

NVML clock-event reasons name the active limiter, not its root cause. GPU idle and configured application clocks can be expected. Software power cap means clocks are respecting a configured power limit, while thermal slowdown points to temperature protection. Correlate reason, workload phase, configured limits, measured power, and temperature before calling it a fault.^{[2]Reference 2NVML API Reference Guidehttps://docs.nvidia.com/deploy/nvml-api/nvml-api-reference.html}

An Xid entry is not a root cause label. NVIDIA states that Xid events can arise from hardware, NVIDIA software, or an application. Recovery actions differ by event, and newer catalog guidance can attach a specific recovery action to the reported event.^{[4]Reference 4Introduction to Xid Errorshttps://docs.nvidia.com/deploy/xid-errors/introduction.html[5]Reference 5Analyzing Xid Errors Using the Xid Cataloghttps://docs.nvidia.com/deploy/xid-errors/analyzing-xid-catalog.html}

The GPU Operator device plugin listens for health events and can report a GPU unhealthy. Kubernetes then lowers allocatable device count, but pods already assigned to that device remain assigned and may fail or crash-loop.^{[6]Reference 6NVIDIA GPU Operator Troubleshootinghttps://docs.nvidia.com/datacenter/cloud-native/gpu-operator/26.3/troubleshooting.html[7]Reference 7Device Pluginshttps://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/} Application readiness and workload containment still need an explicit controller or operator action. Device-plugin health alone is not retrospective eviction.

Respect asynchronous error reporting

CUDA work is asynchronous. An API call can enqueue work and return before the device executes it; an error may surface at a later synchronization boundary. Capture the first observed failure, stream, request, kernel, and correlation timestamp. CUDA_LAUNCH_BLOCKING=1 can help reproduce ordering in a controlled debug run, but it changes execution and should not become a fleet-wide production setting.^{[8]Reference 8CUDA Programming Guide.https://docs.nvidia.com/cuda/cuda-programming-guide/}

Some CUDA runtime errors explicitly leave the process in an inconsistent state. NVIDIA documents illegal address, kernel-execution launch timeout, launch failure, and several hardware-stack errors as requiring process termination and relaunch. Retrying another request in that same process is unsafe. This rule doesn't apply to every CUDA error, so preserve the exact error enum and originating API operation rather than matching the word “CUDA.”^{[9]Reference 9CUDA Runtime API Error Typeshttps://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html}

Contain at smallest broken scope

Use a ladder. Move outward only when evidence says the smaller boundary can't protect users or hardware.

For r-7812, the request is safe to retry on a different healthy replica only if its operation is idempotent or deduplicated. The failed replica leaves readiness. Its snapshot and receipt are copied. Because evidence points to allocator layout and the CUDA error is OOM rather than an error documented as process-corrupting, policy may allow graceful process teardown and relaunch. No ECC, Xid, link, stack, thermal, or power signal currently supports a node drain.

An illegal-address error appears after a request, but the next health check passes. May the worker serve another request?

Answer

No. CUDA documents illegal address as leaving the process in an inconsistent state. Remove the worker from readiness, preserve evidence, terminate and relaunch the process, then apply re-admission gates. A passing application probe doesn't repair CUDA process state.

Run diagnostics with an explicit question

DCGM diagnostics are active tests, not repair commands. NVIDIA describes four levels:

Exact tests depend on DCGM version, hardware, driver, and installed plugins. A named test may be unavailable. In plugin output, Pass applies only to checks that ran, Fail may be a device finding or a test-setup error, and Skip gives no health verdict. Treating Skip as Pass turns missing coverage into false readiness.^{[10]Reference 10DCGM Diagnosticshttps://docs.nvidia.com/datacenter/dcgm/latest/user-guide/dcgm-diagnostics.html[11]Reference 11DCGM Diagnostic Pluginshttps://docs.nvidia.com/datacenter/dcgm/latest/reference/diagnostics/plugins/index.html}

Ask one question per run:

• “Can this GPU enumerate and pass deployment checks?” points toward Level 1.
• “Did memory or PCIe fail around this incident?” points toward selected Level 2 tests.
• “Can the quarantined tensor-parallel topology sustain fabric traffic?” points toward nvbandwidth or NCCL plugins at an admin level.
• “Does the device reproduce a memory or power failure under burn-in?” points toward selected stress tests on idle hardware.

Record command, DCGM version, selected entities, test names, parameters, wall time, plugin availability, raw result, and exit status. DCGM doesn't claim comprehensive diagnosis or automatic repair. A passing suite narrows evidence; it doesn't erase an Xid or guarantee future health.^{[10]Reference 10DCGM Diagnosticshttps://docs.nvidia.com/datacenter/dcgm/latest/user-guide/dcgm-diagnostics.html}

Make ownership part of automation

A multi-tenant GPU node has at least four owners:

Error-aware automation consumes structured facts, not log substrings alone. A useful event includes GPU UUID, node UID, workload UID, process identity, exact CUDA or Xid code, source, first and last timestamps, recurrence count, and evidence links. Its transition should be idempotent: one device fault creates one quarantine record, not a storm of repeated drains.

Never let a workload controller reset a shared GPU or drain a node by inference. NVIDIA's own GPU Operator upgrade workflow separates cordon, workload termination, validation, and uncordon. Its documentation warns that full drain can evict unrelated pods and can permanently remove emptyDir data.^{[12]Reference 12Automated Driver Upgradehttps://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html} Production remediation needs equivalent scope checks and authority.

Preserved evidence also needs tenancy controls:

• Snapshot allocator metadata, not prompt bodies, unless incident policy explicitly requires content.
• Restrict process lists and command lines when they expose another tenant's paths or arguments.
• Attach retention, encryption, and access policy to receipts.
• Store stable IDs and hashes so responders can correlate without copying sensitive payloads.

Repair path for gpu-a17

The repair is intentionally narrow:

1. Contain: Set summarizer-7f9d6 not ready. Stop scheduler admission to its process.
2. Preserve: Save snapshot, application logs, exact versions, NVML memory and process view, ECC counters, clocks, power, temperature, and kernel Xid window.
3. Classify: Confirm 30.10 GiB allocated, 36.70 GiB reserved, 6.10 GiB inactive across split blocks, 0.88 GiB largest fragment, and 39.38 GiB device-wide use. Attribute the 2.68 GiB delta.
4. Repair process: Finish or reject in-flight work according to the request contract. Relaunch the serving process under its normal supervisor.
5. Prove baseline: Load the same model revision and configuration. Confirm expected idle memory and one process owner.
6. Burn in: Replay a bounded canary workload that includes the 1.50 GiB allocation shape. Watch allocator headroom, device-wide delta, Xid/ECC, link status, clocks, temperature, power, and latency.
7. Re-admit: Restore only a small traffic fraction. Expand after a fixed observation window with no gate failure.

If fragmentation returns under the same bounded workload, process restart was containment, not repair. Revisit admission envelopes, KV-cache sizing, batch and sequence caps, workspace lifetime, CUDA graph pools, and allocator configuration. If outside-PyTorch use grows, attribute NCCL, custom CUDA, peer processes, or context lifecycle before changing allocator settings.

Re-admission gates

Use explicit pass/fail gates rather than “looks stable”:

Burn-in must resemble the failed allocation and communication shape without carrying production traffic. “One inference succeeded” misses fragmentation that emerges after many allocation cycles and fabric faults that need multi-GPU load. Define duration, load, pass thresholds, and abort conditions before starting.

A Level 2 DCGM run reports PCIe Pass and nvbandwidth Skip. Can automation re-admit a topology whose incident involved NVLink timeouts?

Answer

No. PCIe Pass covers only the check that ran. Skip is no health verdict, and the missing fabric coverage is relevant to the incident. Keep the topology quarantined until the required test can run or an authorized owner chooses another evidence path.

Downloadable receipt analyzer

The fixture contains eight synthetic receipts spanning allocator fragmentation, application capacity, outside-PyTorch ownership, device fault, fabric fault, clock limiting, process-corrupting CUDA error, and driver/runtime failure.

• Download fixture
• Download read-only analyzer

The analyzer doesn't reset, kill, drain, or delete anything. It sorts evidence into a first hypothesis and a scoped next step. Its memory rows assume PyTorch's native allocator. new_xids and new_uncorrectable_ecc cover only the incident window. clock_event_reasons contains only limiters that were unexpected for the workload phase and correlated with the regression. cuda_error is normalized only after recording the exact enum and originating API operation; launch_timeout here means the process-invalidating kernel-execution error. dcgm_findings contains defect classes after an operator or version-pinned parser has inspected each Fail code and message; raw Fail and Skip results don't qualify. The fixture's 4.0 GiB outside-PyTorch threshold is illustrative, not universal. Production automation still needs workload-specific bands, platform policy, event-specific guidance, and ownership checks.

Notice the precedence. A new uncorrectable ECC event and decoded fabric finding outrank an OOM-derived memory guess. An Xid takes its scope from the event-specific catalog instead of being called a hardware fault by default. Process-invalidating CUDA errors outrank allocator arithmetic. Only then does the analyzer compare device-wide use, allocator reserve, inactive split blocks, and largest fragment. This is triage, not autonomous remediation.

Mastery rubric

You can operate this incident when you can:

• Reconcile allocated, reserved, inactive, largest-block, and device-wide memory without calling every delta a leak.
• Explain what a PyTorch snapshot omits and how finite history can erase the event you need.
• Preserve request, process, device, node, and cluster evidence before destructive state changes.
• Separate ECC/Xid, fabric, thermal/power, stack, and process-state evidence.
• Choose retry, process restart, device quarantine, or node drain from explicit scope and authority.
• Interpret DCGM level, selected plugin, Pass, Fail, and Skip without inflating coverage.
• Define readiness and burn-in gates tied to the original failure shape.

Score yourself on one unseen receipt:

Pitfalls that widen blast radius

• Calling cached reserve a leak. Reserved memory is allocator policy; snapshot block history decides whether fragmentation is plausible.
• Reading NVML delta as proof of another process. Direct CUDA, NCCL, graphs, libraries, and peer processes all need attribution.
• Retrying in a corrupted CUDA process. Exact runtime error semantics decide whether relaunch is mandatory.
• Treating every Xid alike. Xid is a structured clue; catalog guidance and surrounding evidence decide scope.
• Treating Skip as Pass. Missing plugin coverage can't satisfy a readiness gate.
• Restarting before preservation. The restart may clear the only allocator or kernel evidence.
• Draining for convenience. Shared-node eviction and ephemeral-data loss require platform authority and review.
• Re-admitting after one probe. Short probes miss allocation-cycle fragmentation, topology load, and thermal soak failures.

Check your understanding