AI-Generated Kernel Engineering

A generated CUDA extension clears its visible test and reports a 1.4x speedup. During review, an engineer finds one branch for the exact public shape. Every other shape returns zeros. Code generation succeeded: the source compiled, loaded, and produced a benchmark number. Kernel engineering failed because the candidate never implemented the operator contract.

That distinction drives this lesson. Generated source is a proposal. Promotion requires evidence that the candidate preserves semantics across its supported domain, executes safely, beats a comparable baseline, and can be withdrawn when deployment disagrees with the lab result.

The GPU Runtime Execution Lab separated host submission from device completion and made dispatch, static storage, fallback, and timing scope explicit. Generated-kernel evaluation takes ownership one layer lower: prove that the candidate symbol behind that dispatch computes the correct operator before the runtime is allowed to replay or promote it.

One operator will carry the full process. weighted_rmsnorm(x, weight, eps) normalizes each last-dimension row and applies one learned weight per feature:

$$y_{i,j} = x_{i,j}\, w_j\left(\frac{1}{H}\sum_{k=1}^{H}x_{i,k}^{2}+\epsilon\right)^{-1/2}$$

The reference is ordinary PyTorch. A candidate may use CUDA, Triton, another GPU language, or generated compiler code. The promotion gates stay the same.

Write operator contract before prompt

Prompt quality can't repair an underspecified operation. A generator needs both the mathematical expression and the library behavior that surrounds it. PyTorch custom operators declare mutation and aliasing in their schema, while opcheck tests registration, schema, fake-tensor, and compilation integration rather than full numerical correctness.^{[1]Reference 1The Custom Operators Manualhttps://docs.pytorch.org/docs/main/library.html} Treat those as separate obligations.

Here is the contract for the running operator:

The dispatch predicate is part of the contract. Suppose the first optimized kernel supports CUDA, a contiguous last dimension in x, contiguous weight, H divisible by 128, and all three dtypes. The wrapper must check those facts and call the reference outside that predicate. A fast-path restriction is acceptable. An unannounced semantic restriction isn't.

Use a prompt specification that exposes obligations without exposing secret fixtures:

Exact hidden shapes and values stay outside the model context. Structural rules remain visible so the generator can implement the intended program rather than guess the benchmark.

What must reviewer be able to state before generation begins?

Answer

Reviewer must state accepted shapes, dtypes, strides, devices, aliasing, error behavior, numerical policy, and fallback predicate without reading candidate. If any axis is unclear, contract isn't frozen.

Isolate generated code and toolchain

Generated kernel source is untrusted build input. Compilation can invoke preprocessors, linkers, build scripts, package hooks, and dynamic loading. Run the agent and compiler in a disposable sandbox with read-only fixtures, a pinned toolchain image, no credentials, no network, tight CPU and memory limits, a wall-clock timeout, and one writable build directory. Allow-list compiler commands and artifact types. Keep the promotion registry and production library outside the sandbox.

Device execution needs a second boundary. Run untrusted binaries on a short-lived GPU worker with no production workload or other tenant, expose only evaluator-owned inputs and output buffers, kill the worker on timeout or device fault, and quarantine or reset the device before reuse. A container around a process doesn't make arbitrary device code safe to run beside production. The compiler sandbox limits host-side build capabilities; the disposable GPU worker reduces the blast radius of hangs, illegal accesses, and driver-facing failures during execution.

These boundaries follow the same principle as a code-generation agent sandbox: model output gets the capabilities needed for the task, not ambient authority over the repository or secrets. Record every tool call, command, exit code, and artifact hash. Reject attempts to edit the harness, reference, timing code, or test data.

The controlled loop can iterate across these workers:

1. Generate source and an explicit dispatch predicate.
2. Compile with pinned flags and capture the complete log.
3. Run visible correctness tests.
4. Inspect the artifact and launched symbol.
5. Run hidden tests and safety tools through evaluator-controlled commands.
6. Benchmark only after the earlier gates pass.
7. Feed a coarse failure category back without disclosing the hidden fixture.

Even coarse feedback leaks information over repeated attempts. Cap repair attempts, keep an untouched final holdout that never feeds the generation loop, and rotate hidden families between evaluator versions. Promotion uses the final holdout once after the candidate and dispatch predicate are frozen. A holdout failure rejects that candidate without repair feedback.

KernelAgent is a direct implementation example of hardware-guided iteration: agents inspect, profile, modify, and verify kernels rather than relying on one-shot generation.^{[2]Reference 2KernelAgent: Hardware-Guided GPU Kernel Optimization via Multi-Agent Orchestrationhttps://pytorch.org/blog/kernelagent-hardware-guided-gpu-kernel-optimization-via-multi-agent-orchestration/} Its architecture is useful as workflow evidence, not proof that any generated candidate is safe to ship.

The failure loop never turns a hidden test into a public answer. The evaluator may report stride coverage failed or non-finite case failed; it shouldn't return a tensor that exposes the fixture. Raw hidden-test output, sanitizer logs, and tensors remain evaluator-only.

Catch candidate that memorizes visible input

A small evaluator makes reward hacking visible. The next example freezes a deliberately bad candidate, then changes values and contract axes after that freeze. The candidate computes the requested operator only for the public 2 x 4 contiguous float32 shape and returns zeros otherwise.

Scale and sign transformations preserve shape, so the hardcoded shape branch still passes them. The new shape and noncontiguous view expose the missing domain. A strong hidden suite varies independent axes rather than treating more random seeds as new coverage.

Build correctness suite by failure mode

Original KernelBench evaluates generated kernels against PyTorch across 250 workloads and reports fast_p, the fraction that are correct and faster than a chosen threshold.^{[3]Reference 3KernelBench: Can LLMs Write Efficient GPU Kernels?https://arxiv.org/abs/2502.10517} That metric is useful for generation research. It doesn't define a production contract. KernelBench-Verified showed why stronger harnesses matter: a TF32-enabled PyTorch baseline, four post-generation input distributions, and peak-memory measurement exposed false wins and reward hacking.^{[4]Reference 4KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?https://arxiv.org/abs/2607.16241}

Use separate test families:

Input-blind testing means the generator doesn't receive exact evaluation values before candidate source and dispatch are frozen. The running kernel necessarily reads those values as operator inputs. A hidden distribution draws them from undisclosed families rather than reusing one secret tensor. Keep families versioned so the evaluator can reproduce a result while the generator can't tune to literals.

Numerical acceptance needs declared formula. For finite reference $r$ and candidate $c$, a common componentwise rule is:

$$|c-r| \le \mathrm{atol} + \mathrm{rtol}|r|$$

PyTorch allclose uses this relative-plus-absolute rule and lets the evaluator decide whether NaNs at matching positions compare equal.^{[5]Reference 5torch.allclosehttps://docs.pytorch.org/docs/stable/generated/torch.allclose} The harness should assert dtype, shape, and any promised output layout separately. Choose tolerance from the operator, dtype, reduction length, and downstream sensitivity. One loose tolerance for every task can reward wrong algorithms. Exact equality can reject a valid reassociation. Also report maximum absolute error, maximum relative error away from zero, and failing count; an aggregate alone can hide one catastrophic row.

Adversarial values must match policy. If reference propagates NaN, candidate should not replace it with zero merely to improve finite-case score. If infinities are outside accepted domain, reject input consistently at wrapper rather than letting different kernel paths invent behavior.

What makes hidden correctness suite independent enough to resist benchmark gaming?

Answer

It changes shape, stride, scale, sign, dtype, special values, and call order independently, reproduces failures without revealing fixtures, caps adaptive feedback, and keeps untouched final holdout.

Prove runtime safety and inspect what compiled

Numerical equality on one run can't reveal undefined execution that happened to survive. NVIDIA Compute Sanitizer splits relevant checks: memcheck finds out-of-bounds and misaligned memory access, racecheck reports shared-memory hazards, initcheck catches uninitialized device-memory reads, and synccheck detects invalid synchronization.^{[6]Reference 6Compute Sanitizerhttps://docs.nvidia.com/compute-sanitizer/ComputeSanitizer/} Run memcheck first because the other tools don't perform memory-access checking.

Exercise the sanitizer suite with tail sizes, minimal and maximal supported shapes, and nontrivial storage offsets. Test repeated calls and concurrent streams separately. racecheck sees on-chip shared-memory hazards, not arbitrary global-memory races between launches, so a kernel with process-global scratch storage can still corrupt concurrent invocations. Avoid such scratch state when possible; otherwise add explicit ownership or synchronization plus stress and output checks. Capture the tool version, command, exit status, and raw report. Any unexplained report blocks timing.

The compile gate needs evidence beyond an exit code:

• Pin compiler, driver-facing toolkit, target architecture, language/runtime package, flags, and environment variables.
• Save source hash, build log, binary hash, symbols, PTX or other intermediate code when available, and final device artifact.
• Confirm benchmark launches candidate symbol rather than PyTorch fallback.
• Inspect dispatch predicate and fallback counters. Zero candidate launches means a fast baseline disguised as generated result.
• Search source and intermediate artifacts for public tensor constants, shape-only branches, host callbacks, embedded binaries, network code, and writes outside output.
• Reject unexpected dynamic dependencies, architecture targets, or runtime compilation paths.

CUDA binary utilities can inspect cubin and host objects, while nvcc documents compilation phases and generated artifacts.^{[7]Reference 7CUDA Binary Utilitieshttps://docs.nvidia.com/cuda/cuda-binary-utilities/[8]Reference 8NVIDIA CUDA Compiler Driver NVCChttps://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/} Inspection doesn't prove semantics. It closes a different gap: the evidence corresponds to the code that the reviewer thinks ran.

Benchmark only accepted program

The speed score starts after correctness, safety, and artifact gates. Compare the exact operator scope under baseline parity.

CUDA launches are asynchronous. Host call duration can measure enqueue, not completed device work. Events in the relevant stream provide GPU timing when the end event completes.^{[9]Reference 9CUDA semantics.https://docs.pytorch.org/docs/2.13/notes/cuda.html} PyTorch benchmarking utilities also perform warmup and synchronization, but the evaluator still must state what the timed region includes.^{[10]Reference 10PyTorch Benchmarkhttps://docs.pytorch.org/tutorials/recipes/recipes/benchmark.html}

Alternate baseline and candidate rounds, such as A-B-B-A, to reduce thermal and clock-order bias. Report the distribution, not one minimum. Profile separately. Collection replay and counters perturb runtime, so profiler duration explains the mechanism while clean event timing supports the speed claim.

Measure memory as a promotion gate. A candidate that saves 8 microseconds by caching one compiled binary per shape may exhaust the process over real traffic. Track peak device allocation, reserved memory, scratch workspace, compile artifacts, host memory, and cache cardinality across the shape sequence. Compare steady-state and first-use growth.

Hardware-specific tuning is valid when the receipt says so. Tile size, vector width, tensor-core path, occupancy, and shared-memory budget can change across GPU architectures. KernelBenchX evaluates category-aware Triton tasks, low-precision variants, and hardware efficiency across multiple GPU platforms; its reported cross-hardware speedup variance shows why one target's win doesn't transfer automatically.^{[11]Reference 11KernelBench-X: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelshttps://arxiv.org/abs/2605.04956} Ship an explicit architecture predicate and per-target evidence. Unknown hardware should take a tested fallback, not the closest-looking binary.

Which parity facts must a generated-kernel speedup receipt preserve?

Answer

Baseline and candidate must execute same accepted work in same runtime state under same timing and cache policy. Report compile latency, steady-state latency, peak memory, raw paired samples, and target GPU separately.

Read benchmark scores within their boundary

Kernel benchmarks answer research questions about generators and harnesses. They can't replace library review.

An aggregate pass rate can hide severity. One wrong element on a rare stride might count the same as a complete failure; neither is acceptable for the library contract. A speed threshold can hide a regression below its cutoff. An average can hide an architecture-specific loss. Hidden tests reduce hardcoding only along the axes that harness authors encoded. Treat the benchmark as generator evaluation, then run the operator-specific promotion pipeline.

The baseline definition deserves versioning. A weak or mismatched baseline inflates apparent progress. Record the framework commit, eager or compiled mode, math settings, graph transforms, and exact dispatch. Re-run the baseline when the toolchain changes.

Preserve promotion receipt

Evidence should be machine-readable enough for rerun and compact enough for review. Example receipt fields:

The receipt uses placeholders because fabricated hardware results would defeat its purpose. A real promotion fills values from the evaluator, links immutable artifacts, and signs the record.

A human reviewer checks more than the pass/fail summary:

• Contract and dispatch predicate match library semantics.
• Candidate source is understandable enough to own after generator disappears.
• Generated build dependencies and licenses are allowed.
• Hidden-suite coverage and tolerances fit operator risk.
• Performance mechanism agrees with artifact and profile evidence.
• Memory growth and fallback behavior remain bounded.
• Target architecture, rollout scope, owner, alarms, and rollback are explicit.

Promote through library staging, then canary a small traffic slice only on the receipt's target architecture and dispatch domain. Compare fallback rate, sampled shadow outputs where affordable, latency distribution, device errors, memory, compile/cache growth, and downstream model quality signals. Keep the reference callable behind a feature flag and test that rollback before increasing traffic. Roll back on a correctness mismatch, sanitizer-equivalent device fault, memory growth, tail-latency regression, or unsupported-hardware dispatch. Preserve the failed receipt and traffic signature for the next candidate.

What proves team owns generated kernel after generator session disappears?

Answer

Named reviewer approves exact receipt, and on-call engineer can identify candidate, disable dispatch, restore reference, and retrieve raw evidence without generator session. Otherwise candidate stays in staging.

Promotion rubric

Use hard gates before any weighted score:

No latency gain compensates for a failed contract gate. Once all hard gates pass, the team may rank candidates by latency, memory, maintainability, compile cost, portability, or energy within deployment priorities.

Mastery check

Evaluation rubric

• Foundational: Distinguish generated source, benchmark acceptance, and production promotion as separate states.
• Intermediate: Design contract-complete hidden suite, sanitizer plan, artifact checks, and baseline-parity benchmark for one operator.
• Advanced: Defend hardware-specific dispatch using reproducible receipt, human review, canary signals, and rollback trigger.

Common pitfalls

• Showing secret fixtures to generator, then calling later pass hidden evaluation.
• Treating more random seeds as coverage for missing shapes, strides, aliasing, or devices.
• Loosening tolerance until wrong reduction passes instead of deriving policy from operator and dtype.
• Timing candidate before proving launched path, completed work, and baseline parity.
• Ignoring compile caches, workspace, and persistent memory while reporting kernel latency.
• Promoting generated binary without source ownership, toolchain provenance, or tested fallback.