Capstone: Reproducible ML Study

Two training curves can make an unchanged algorithm look like an improvement. One run discovers the goal early; another spends its budget exploring. Henderson and colleagues documented how seeds, hyperparameters, implementations, and other choices affect comparisons of policy-gradient algorithms in continuous-control benchmarks.^{[1]Reference 1Deep Reinforcement Learning That Mattershttps://ojs.aaai.org/index.php/AAAI/article/view/11694} Selecting only the best seed hides precisely the variation a comparison needs to measure.

How can you tell whether a change helped? Specify the comparison before inspecting new results, keep every planned seed, and show uncertainty. Pairing can reduce shared noise when arm outcomes covary positively. Bootstrap intervals still depend on sampling assumptions.

File hashes let reviewers compare bytes. Repeatability requires an actual rerun under a specified environment. PyTorch releases and platforms can still produce different results from identical seeds.^{[2]Reference 2Reproducibilityhttps://docs.pytorch.org/docs/stable/notes/randomness.html}

Work through those distinctions with a small reinforcement-learning study: does potential-based reward shaping improve early goal-reaching in a stochastic corridor while keeping final greedy success similar under the original environment reward? You'll run the comparison, inspect its failures, and design the larger artifact an independent reviewer would need.

View larger

The study runs locally with synthetic transitions and NumPy. Both arms share the environment reward and indexed random inputs. Candidate training adds $F$ as a learning signal; final evaluation scores frozen policies with the original reward. The package uses code-generated data and needs no private logs, live users, paid APIs, or specialized hardware.

This chapter contains a worked run whose results are already known. Its plan was not independently preregistered. Repeating seeds 0 through 23 checks repeatability. For your own confirmatory study, finish pilot debugging, freeze the design, and reserve a new untouched seed set before inspecting results.

Read the claim before building the simulator

The first implementation decision is whether the source contains one claim small enough to test honestly. Start with Ng, Harada, and Russell's paper on policy invariance under reward transformations.^{[3]Reference 3Policy Invariance Under Reward Transformations: Theory and Application to Reward Shapinghttps://wordpress.andrewng.org/index.php/publication/policy-invariance-under-reward-transformations-theory-and-application-to-reward-shaping/} Its main claim is precise enough for a small simulator: a potential-based shaping reward can guide learning while preserving optimal policies under the theorem's assumptions.

Don't begin by reproducing every table or proof. Keshav's three-pass reading method gives a practical order: classify the paper, inspect its evidence, then reconstruct the part you plan to test.^{[4]Reference 4How to Read a Paperhttps://svr-sk818-web.cl.cam.ac.uk/keshav/papers/07/paper-reading.pdf} That order keeps an attractive equation from turning into an unbounded reproduction project.

Use this triage card while reading. Fill it with page or section pointers so a reviewer can trace your interpretation back to source text.

First pass: classify the claim

Keshav's first pass is a short classification scan. An evidence audit takes a closer look: read the title, abstract, and introduction; skim section headings; glance at the mathematical content; read the conclusion and references. Save detailed figure inspection for the second pass. The author-hosted version includes revisions from February 2016.^{[4]Reference 4How to Read a Paperhttps://svr-sk818-web.cl.cam.ac.uk/keshav/papers/07/paper-reading.pdf}

Write one sentence for the problem, method, comparison, and claimed result. If those sentences stay vague, the study isn't ready to design.

For this paper, the method derives a transition reward from a potential function. The policy-invariance theorem covers this construction; arbitrary bonuses can change which policy is optimal.

Stop the first pass with a scope decision: reproduce, extend, or decline. This capstone performs a small empirical check of one implication and adds an uncertainty-focused artifact.

Second pass: inspect evidence and assumptions

Now inspect the figures, axes, and error bars, then read the equation, experimental setup, and failure caveats. First write down what stays constant, what changes, and what the paper measures.

Missing implementation detail belongs in an assumptions log, not in a silent guess.

Pay close attention to evaluation reward. A candidate trained with shaped reward can look better if its score includes the extra training bonus. Compare both learned policies using the original environment reward or you have changed the measurement.

Record every adaptation. The corridor, Q-learning hyperparameters, action-slip probability, seed budget, and bootstrap rule are capstone choices rather than claims copied from the paper.

Third pass: reconstruct the test

Explain the shaping equation without looking. Then rebuild the smallest environment where moving toward a goal changes potential. Predict what should happen to early learning and final policy before running code.

Then challenge the design. Could the candidate receive more steps, a friendlier random stream, or an easier evaluation? Could a single lucky seed drive the mean? Every plausible shortcut becomes a control, ablation, or stated limitation.

Finish paper_triage.md with three labels: supported by source, study adaptation, and unresolved. That file prevents a plausible implementation detail from turning into a misattributed paper claim.

Turn curiosity into a falsifiable contract

The triage card names a paper claim and a smallest test. Add the observation that would count against the idea. "Does reward shaping help agents learn?" is too broad because it leaves the agent, environment, shaping rule, metric, sample, and outcome undefined.

A falsifiable hypothesis names an observation that can contradict it. Narrow scope makes the capstone stronger: every noun can become a file, parameter, table column, or assertion.

Use this research question:

In a stochastic seven-state corridor, does potential-based shaping increase tabular Q-learning's first-30-episode goal-reaching rate versus sparse reward, while keeping final greedy-policy success unchanged under original reward?

Lock the contract before inspecting confirmatory results:

State the expected result

Prediction comes before output. A distance-based potential supplies feedback before the first goal, so improved early goal-reaching is a plausible hypothesis. Policy-invariance theory concerns optimal policies under its assumptions. The policies produced by 80 finite training episodes require empirical evaluation; similar final greedy success is a hypothesis to test.^{[3]Reference 3Policy Invariance Under Reward Transformations: Theory and Application to Reward Shapinghttps://wordpress.andrewng.org/index.php/publication/policy-invariance-under-reward-transformations-theory-and-application-to-reward-shaping/}

Record that expectation before seeing confirmatory values so reviewers can compare the prediction with the outcome. Label any later explanations as exploratory.

For your own study, write this expected pattern in study_plan.md before running the untouched confirmatory seeds. Seeds 0 through 23 here are already observed:

• positive early paired difference
• interval above zero
• near-zero final difference
• timestamp and retained commit for the frozen plan

A retained commit identifies a version; preserving the plan history makes later edits reviewable. Independent preregistration requires hypotheses, variables, tests, decision criteria, and exclusions in a timestamped read-only system before confirmatory data collection.^{[5]Reference 5Welcome to Registrationshttps://help.osf.io/article/330-welcome-to-registrations}

Write the rejection rule

Support the capstone's narrow claim only if the mean early difference is at least +0.10 and the 90% interval's lower bound is above zero. Also require the final-policy guardrail.

Withhold support for the study claim when either requirement fails. A mean of +0.09 is below the declared point-estimate threshold even if its interval is positive. A mean of +0.20 also fails if final original-reward performance shifts beyond the guardrail. A claim of at least +0.10 with 90% confidence requires the interval's lower bound to clear +0.10.

"Inconclusive" can be accurate when interval width leaves multiple practical interpretations. Keep that label distinct from "no effect" and from "the candidate is worse."

Keep the study public-safe

Small research work still needs a data and authority boundary. Synthetic data removes many privacy risks, but a public repository can still leak credentials, copied code, or outputs from proprietary environments. Decide that boundary before collecting artifacts.

The corridor generator owns every transition and uses only a small configuration plus a pseudorandom seed. It runs without customer data, service scraping, external models, or a live agent.

Add this boundary to the README before code:

Run a secret scanner before publishing. Review generated files for machine paths and usernames. Use an open license only for code and assets you have authority to release.

Build the smallest controlled environment

The corridor has states 0 through 6. Every episode starts at 0; reaching 6 yields environment reward 1; every other transition yields 0. An intended action reverses with probability 0.15, and an episode ends after 12 steps.

Write the environment contract before writing the agent. Otherwise a reward, reset, or ending-condition change can enter one arm unnoticed and look like a learning effect.

terminated means the task reached the goal. truncated means an external rollout budget ended collection. The learner estimates discounted return in an underlying task without that 12-step deadline; it still bootstraps at a timeout, but not at the true terminal goal. Our reported success metric is instead goal-reaching within 12 steps. These are related objectives, not identical ones.

If the deadline were part of the task itself, remaining time would belong in the state and deadline expiry would be terminal. Don't silently switch between these formulations.^{[6]Reference 6Handling Time Limitshttps://gymnasium.farama.org/tutorials/gymnasium_basics/handling_time_limits/} The compact script records episode success and step counts; a full capstone should also retain diagnostic transition traces.

Treating a timeout as terminal success, or bootstrapping through a true terminal, changes the learning target.

Tabular Q-learning stores one value for each state-action pair. During training it moves the chosen action toward observed reward plus the best estimated next-state value. That's the off-policy update from that earlier lesson.^{[7]Reference 7Reinforcement Learning: An Introductionhttp://incompleteideas.net/book/the-book-2nd.html}

At each transition, a left/right action has a 15% chance to reverse before the new position is clamped to the corridor. Both arms observe the same environment reward $R$. Candidate training adds the potential difference $F$ to its update; final evaluation scores both policies on $R$.

Define distance potential as:

States nearer the goal have a larger potential. Candidate training adds:

With discount 0.95, a step toward the goal usually produces a positive shaping signal. A step away usually produces a negative one. Check that on the first two cells before running the 24-seed study. $\Phi(0)=-1$ and $\Phi(1)=-5/6$, so a right move from the start yields $F=0.95\cdot(-5/6)-(-1)=+0.208$, while the reverse move yields $-0.117$.

A left action at state 0 can be clamped to a 0 → 0 self-loop, where its potential-difference bonus is still +0.050.

View larger

The reason this particular bonus preserves the discounted objective is telescoping. For a trajectory of length $T$:

At the goal, $\Phi(6)=0$, so every completed trajectory from state 0 receives the same total discounted addition, $+1$. A bounded potential also makes the final term vanish in the infinite-horizon discounted limit. At an artificial timeout, the remaining term generally isn't zero. That's why terminal potential and timeout bootstrapping matter; merely writing the equation isn't enough.^{[3]Reference 3Policy Invariance Under Reward Transformations: Theory and Application to Reward Shapinghttps://wordpress.andrewng.org/index.php/publication/policy-invariance-under-reward-transformations-theory-and-application-to-reward-shaping/}

The snippet below is the contract for that one transition. It uses the Python standard library so you can verify the arithmetic without NumPy.

GAMMA = 0.95
GOAL = 6

def potential(state: int) -> float:
    return -(GOAL - state) / GOAL

def shaping(state: int, next_state: int) -> float:
    return GAMMA * potential(next_state) - potential(state)

toward = shaping(0, 1)
away = shaping(1, 0)
assert potential(0) == -1
assert abs(potential(1) - (-5 / 6)) < 1e-12
assert toward > 0
assert away < 0
assert abs(shaping(0, 0) - 0.05) < 1e-12
print(f"Phi(0)={potential(0):.3f}  Phi(1)={potential(1):.3f}")
print(f"F(0->1)={toward:+.3f}")
print(f"F(1->0)={away:+.3f}")
print(f"F(0->0)={shaping(0, 0):+.3f}")

# Two completed trajectories take different lengths but get the same addition.
for path in ([0, 1, 2, 3, 4, 5, 6], [0, 1, 0, 1, 2, 3, 4, 5, 6]):
    bonus = sum(GAMMA**t * shaping(s, n) for t, (s, n) in enumerate(zip(path, path[1:])))
    assert abs(bonus - 1.0) < 1e-12
print("Completed-path discounted bonus: +1.000 for both paths")

Phi(0)=-1.000  Phi(1)=-0.833
F(0->1)=+0.208
F(1->0)=-0.117
F(0->0)=+0.050
Completed-path discounted bonus: +1.000 for both paths

A left move clamped at state 0 adds $F(0,0)=+0.05$ despite making no progress. Evaluate success with the discounted total, terminal handling, and unchanged evaluation target; the shaping reward alone does not measure task success.

The comparison table records what stayed paired and what was allowed to differ:

Now the two arms differ in one intended place: the candidate's training update. Controls and paired randomness can test whether that boundary held in code.

Make randomness and controls explicit

Reinforcement learning measurements can vary wildly with random seeds, environment randomness, codebase differences, and hyperparameters. Henderson and colleagues showed that two runs of the same algorithm differing only by random seed can look like two completely different algorithms.^{[1]Reference 1Deep Reinforcement Learning That Mattershttps://ojs.aaai.org/index.php/AAAI/article/view/11694} Treat the random stream as part of the formal experimental design, not as an invisible background detail.

The paired seed trial protocol

Suppose each arm has $n$ independent, identically distributed runs, and the two arms are sampled independently. If $X_i$ is baseline performance and $Y_j$ is candidate performance, the variance of the difference between sample means is:

Seed variation contributes to both arm variances. How many runs you need depends on the effect, variability, desired precision, and sampling design; there is no fixed requirement of hundreds.

In this paired seed trial, seed $i$ supplies both arms with the same indexed random draws. Their states and chosen actions can still differ, so their environment transitions needn't match. Keep the pairing when computing the difference:

For independent, identically distributed seed pairs, the variance of the sample mean paired difference $\bar{\Delta}$ is:

Positive covariance reduces variance relative to independent arms with the same marginal variances. Zero covariance gives no reduction; negative covariance increases it. Shared draws don't determine the sign: reversing a right action moves away from the goal, while reversing a left action moves toward it. The same slip can affect two policies differently.

Try this counterexample before assuming pairing always buys precision. The five score pairs below are invented, not corridor results. Candidate scores decrease whenever baseline scores increase:

from statistics import covariance, mean, variance

baseline = [0.0, 0.25, 0.5, 0.75, 1.0]
candidate = [1.0 - score for score in baseline]
differences = [y - x for x, y in zip(baseline, candidate)]
n = len(baseline)
paired_estimate = variance(differences) / n
independent_estimate = (variance(candidate) + variance(baseline)) / n
assert paired_estimate == 2 * independent_estimate
print(f"Mean difference: {mean(differences):+.3f}")
print(f"Sample covariance: {covariance(baseline, candidate):+.4f}")
print(f"Estimated mean variance, paired / independent: {paired_estimate:.4f} / {independent_estimate:.4f}")

Mean difference: +0.000
Sample covariance: -0.1562
Estimated mean variance, paired / independent: 0.1250 / 0.0625

Pairing preserves the intended comparison unit even when it doesn't improve precision. Inspect the arm relationship and retain the whole paired distribution.

Synchronizing randomness with a 2D tape

Episode lengths differ. Suppose one arm reaches the goal in six steps while the other takes twelve. If a sequential slip generator draws once per step and keeps going across episodes, episode 1 starts at offset 6 in one arm and offset 12 in the other. The streams no longer align by episode and step. They aren't thereby completely uncorrelated, and extra conditional random calls can change alignment further.

To keep the arms strictly synchronized, pre-generate a 2D random tape shaped (TRAIN_EPISODES, MAX_STEPS) for each random decision:

1. explore: continuous uniform float in $[0, 1)$ for $\epsilon$-greedy exploration checks.
2. action: discrete uniform integer in $\{0, 1\}$ for random action draws.
3. tie: discrete uniform integer in $\{0, 1\}$ for reproducible random Q-value tie-breaking.
4. slip: continuous uniform float in $[0, 1)$ for the 15% environment action reversal.

Indexing by (episode, step) gives both arms the same available slip and exploration draws at that index, regardless of earlier episode lengths. An arm that has already reached the goal doesn't consume the remaining steps. Shared draws still don't force shared actions or trajectories.

Separate pilot and confirmatory seeds

Use seeds 900, 901, and 902 to debug code, plots, and manifests. Never include them in confirmatory metrics. You have already observed their behavior, so adding them later would make the primary sample outcome-dependent.

Integers 0 through 23 are the worked sample here. Reserve new seeds for your confirmatory study after pilot debugging. Twenty-four pairs fit this capstone and expose heterogeneity; choose a study's sample size to match its precision target and design.

Evaluation uses streams seeded with 10000 + training_seed, separate from the training tape. No evaluation reward updates Q-values. Don't choose a potential, threshold, or hyperparameter by these final evaluations and then reuse them as an untouched test. A seed split prevents direct reuse; transfer to a new environment family requires its own evaluation.

Run every planned pair. Stopping after ten favorable seeds makes sample size depend on the observed outcome, invalidating the fixed-sample analysis used here. A sequential study needs its own stopping and inference procedure, as discussed in the experiment-design lesson.

If a run fails mechanically, record the failure and rerun policy before looking at aggregate results.

Add controls before interpretation

Zero-potential control sets $\Phi(s)=0$. Under the same tape, it must match sparse training exactly. If it doesn't, candidate code changed more than reward. Stop before interpreting any treatment result.

Original-reward evaluation is another control. Both arms receive identical greedy evaluation slips, and neither receives shaping reward. A result reported only on shaped return fails the measurement contract because the candidate is scored with a bonus the baseline never receives.

Use a sign-reversed potential only as an exploratory stress test after confirmatory analysis. It may reveal sensitivity to learning dynamics, but it can't retroactively redefine the primary hypothesis.

Plan the ablation matrix

An ablation removes or changes one component to identify which mechanism drives a result. Change one factor per row and preserve all other settings. That makes the comparison answerable when a result moves.

Don't call an unplanned hyperparameter sweep an ablation after choosing its best row. Preserve all runs and label post-result searches exploratory.

Define metrics and uncertainty

Choose the estimator before seeing results. For seed $i$, let $p^{shape}_{i,30}$ and $p^{sparse}_{i,30}$ be the fractions of the first 30 training episodes that reach the goal. The paired effect is:

Primary estimate averages all 24 differences:

That number answers a bounded question about mean early success under the declared seed-generating procedure. It doesn't prove every seed improves. Report all $d_i$ values and count regressions.

Choose uncertainty at the seed-pair level

Why use a bootstrap interval rather than a paired t interval?

Exact small-sample t inference assumes independent normally distributed differences. A t interval can also be a useful approximation without exact normality; bounded or discrete observations don't automatically make it invalid. Here, success fractions occur in increments of $1/30$ and paired differences lie in $[-1,+1]$. With only 24 pairs, inspect skew, extremes, and sensitivity rather than claiming that either method is automatically reliable.

The percentile bootstrap avoids specifying a normal model, but it isn't assumption-free. We treat seed pairs as independent draws from a fixed trial-generating procedure and use their empirical distribution to approximate its variation. A small sample can miss rare outcomes, and resampling it can't discover them. Agarwal and colleagues motivate interval reporting for finite-run RL comparisons; this one-corridor interval isn't their multi-task benchmark methodology.^{[8]Reference 8Deep Reinforcement Learning at the Edge of the Statistical Precipicehttps://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html}

We treat the 24 observed paired differences $\{d_0, d_1, \dots, d_{23}\}$ as an empirical distribution $\hat{F}_n$. We draw $B = 10{,}000$ bootstrap resamples with replacement:

For each resample $b$, compute the replicate sample mean:

Take the 5th and 95th percentiles of the $10{,}000$ replicate means for the worked 90% interval. The script explicitly uses NumPy's method="linear": it interpolates between sorted values at zero-based position $q(B-1)$, rather than taking replicate 500 or 9500.^{[9]Reference 9numpy.quantilehttps://numpy.org/doc/stable/reference/generated/numpy.quantile.html} Other interval methods, including basic and bias-corrected and accelerated (BCa), make different adjustments. SciPy exposes these choices and paired resampling. None guarantees good coverage for every small sample.^{[10]Reference 10scipy.stats.bootstraphttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html}

Use fixed bootstrap seed 20260731. The interval estimates sampling uncertainty across seed pairs under this trial procedure. The 30 episodes within a run share an evolving learner, so don't treat them as 30 independent analysis units. A 90% confidence procedure targets repeated-sampling coverage; this approximate interval isn't a 90% probability that the theorem or hypothesis is true. It also doesn't account for choosing a protocol after inspecting results. Keep those limits beside the estimate.

Keep a final-policy guardrail

After 80 training episodes, evaluate each greedy policy for 200 episodes on original reward. Pair evaluation slip tapes by seed. Compute the shaped-minus-sparse final-success difference exactly as above.

Require the observed mean and its 90% paired bootstrap interval to lie within [-0.02, +0.02]. This illustrative equivalence-style guardrail supports a narrow comparison on the observed distribution. A zero-width interval when every observed pair matches still leaves rare failures and unseen states untested. This check and equal means say nothing about high absolute competence.

Also inspect per-seed final differences. An average near zero can hide equal and opposite failures if candidate helps one region and harms another.

Plan the experiment table before running

A compact experiment table stops notebook state from becoming methodology. Each row names the exact config, sample, output, and decision role. Read it as a promise about what will be run, not as a retrospective summary.

Store the table in study_plan.md and mirror machine-readable fields in configs/study.json. Freeze both in the same commit as the hypothesis.

Use one status vocabulary: planned, running, complete, mechanical_failure, or excluded_by_locked_rule. "Bad result" isn't an exclusion reason.

Package an exact repository

A reviewer shouldn't have to guess where the hypothesis ends and the analysis begins. Keep source, config, immutable run outputs, report, and talk in distinct paths.

The tree shows the repository you should produce for the capstone. This page ships a runnable miniature of the experiment core; the final submission also needs the plan history, packaging, traces, report, and independent review.

reproducible-reward-shaping/
├── README.md
├── LICENSE
├── CITATION.cff
├── pyproject.toml
├── uv.lock
├── study_plan.md
├── paper_triage.md
├── configs/
│   └── study.json
├── src/
│   └── corridor_study/
│       ├── __init__.py
│       ├── environment.py
│       ├── train.py
│       ├── analyze.py
│       └── manifest.py
├── tests/
│   ├── test_environment.py
│   ├── test_zero_potential.py
│   └── test_manifest.py
├── data/
│   └── generated/
│       └── README.md
├── artifacts/
│   ├── manifest.json
│   ├── runs/
│   │   ├── primary_seed_rows.csv
│   │   └── bootstrap_means.npy
│   └── checksums.sha256
├── report/
│   ├── report.md
│   ├── figures/
│   │   └── paired_seed_delta.svg
│   └── tables/
│       └── primary_result.csv
└── talk/
    ├── outline.md
    └── q-and-a.md

Pin environment and entrypoint

Use pyproject.toml plus uv.lock to pin NumPy and development tools. README should name supported Python version and one clean command such as uv run corridor-study --config configs/study.json.

Run from a fresh clone or container. Record operating system, architecture, Python, NumPy, commit, config hash, and wall-clock duration in manifest. The machine identity may remain coarse; hostnames and usernames don't belong in public output.

The Machine Learning Reproducibility Checklist calls for clear model, hyperparameter, seed, environment, and result reporting.^{[11]Reference 11The Machine Learning Reproducibility Checklisthttps://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf} A lockfile helps, but complete execution instructions and immutable inputs still matter.

Record data provenance

Synthetic data still needs provenance. Save generator version, seed range, transition rule, slip probability, state count, step cap, and environment source hash.

Don't commit one unexplained CSV and call it generated. data/generated/README.md should state exact command that reconstructs any derived rows and whether tracked output is canonical or disposable.

If you adapt capstone to public dataset later, record original URL, license, version, retrieval date, checksum, preprocessing command, and excluded rows. Never replace original identifier with local filename alone.

Create an artifact manifest

A manifest records the files and environment associated with a reported run. A digest tells a reviewer whether bytes match a reference. Execution, scientific validity, and producer authority each need separate evidence.

For the larger submission, record these six kinds of evidence:

1. Commit: git rev-parse HEAD identifies a tracked tree. It doesn't include dirty or untracked files; separately record and review the actual working-tree state.
2. Executed source: hash the source snapshot using a documented filename order and encoding. Compare it with the retained expected snapshot; a self-reported source digest alone doesn't exclude edits.
3. Dependencies and interpreter: retain uv.lock and its digest, plus the exact interpreter and package/build versions used. ==3.12.* permits multiple Python patch versions; a lockfile doesn't pin every property of the host runtime.
4. Platform: record OS and CPU architecture, and additional numerical-library or accelerator details where relevant. Matching these coarse fields still isn't a universal bitwise-repeatability guarantee.
5. Configuration: hash the retained configuration bytes and link them to the execution record. A freshly recomputed hash cannot by itself show when a configuration changed.
6. Outputs: hash the seed table and derived data, and preserve the original reference independently of a rerun that could overwrite it.

View larger

This illustrative schema outlines the locked fields before the run; a completed run fills in the exact hashes:

{
  "study_id": "corridor-reward-shaping-v1",
  "git_commit": "<full-40-char-commit-hash>",
  "config_sha256": "<sha256-of-configs-study-json>",
  "source_sha256": "<sha256-of-src-tree>",
  "python": "3.12.x",
  "numpy": "2.2.6",
  "os": "Darwin",
  "architecture": "arm64",
  "confirmatory_seeds": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
  "bootstrap_seed": 20260731,
  "outputs": {
    "primary_seed_rows.csv": "<sha256-hash>",
    "study_results.json": "<sha256-hash>"
  },
  "status": "worked-run-not-preregistered"
}

Specify what comparison should pass. This fixture aims to reproduce the seed-table and result-JSON bytes under the pinned setup. A different platform can legitimately change the environment manifest, and different serialization can change bytes without changing numerical values. Investigate any unexpected mismatch before reporting agreement. For a numerical-tolerance policy, declare the tolerances and fields in advance and preserve both outputs; don't widen tolerances after seeing an inconvenient result.

The included illustrations/run_study.py reruns the named experiment fence and exports CSV, figure JSON, and a manifest. It records source/exporter/config/output hashes and coarse runtime fields. It doesn't verify an independent reference, retain an earlier run, pin a full environment lock, authenticate execution, or implement the proposed release pipeline. Run it in a separate checkout when comparing with the retained artifacts.

Here is why the distinction matters. Equivalent JSON values needn't have the same bytes, and a fabricated result can be given a perfectly consistent checksum:

import hashlib
import json

row = {"seed": 0, "success": 0.5}
compact = json.dumps(row, sort_keys=True, separators=(",", ":")).encode()
pretty = json.dumps(row, sort_keys=True, indent=2).encode()
digest = lambda data: hashlib.sha256(data).hexdigest()
print("Equal decoded values:", json.loads(compact) == json.loads(pretty))
print("Equal byte digests:", digest(compact) == digest(pretty))

invented_output = b'{"success": 1.0}'
self_reported_manifest = {"output_sha256": digest(invented_output)}
print("Invented output matches its own manifest:",
      digest(invented_output) == self_reported_manifest["output_sha256"])

Equal decoded values: True
Equal byte digests: False
Invented output matches its own manifest: True

A reviewer needs retained expected bytes or an explicit value-comparison rule, plus evidence of the run that produced them. Hash consistency alone supplies neither.

CITATION.cff gives the software both human-readable and machine-readable citation metadata. GitHub renders that file directly in the repository's Cite panel.^{[12]Reference 12About CITATION fileshttps://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-citation-files} After archiving a release to Zenodo or an institutional repository, add a persistent DOI.

Run the miniature study

The script generates paired random tapes, trains both arms, executes the zero-potential control, evaluates on original reward, and prints the decision from all gates. NumPy is pinned here for the worked fixture; a random seed alone doesn't pin a library, build, platform, or generator's call sequence.^{[13]Reference 13Random number compatibility policyhttps://numpy.org/doc/stable/reference/random/compatibility.html}

A Python random port isn't a drop-in substitute: it generates different random tapes and can change the measured results. Sample size and the decision-rule definition would stay the same. Document such a port as a changed implementation rather than expecting this fixture's exact rows.

Read the control points before running. make_tape freezes randomness by episode and step; train changes only the reward; evaluate never receives shaping reward; the bootstrap samples paired differences.

Run it with uv run reward_shaping_study.py. Expected output follows the code.

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = "==3.12.*"
# dependencies = ["numpy==2.2.6"]
# ///

import numpy as np

N_STATES = 7
GOAL = N_STATES - 1
GAMMA = 0.95
ALPHA = 0.35
EPSILON = 0.20
TRAIN_EPISODES = 80
MAX_STEPS = 12
EARLY_EPISODES = 30
EVAL_EPISODES = 200
SLIP_PROBABILITY = 0.15
BOOTSTRAP_REPLICATES = 10_000
SEEDS = tuple(range(24))

def potential(state: int) -> float:
    return -(GOAL - state) / GOAL

def make_tape(seed: int) -> dict[str, np.ndarray]:
    rng = np.random.default_rng(seed)
    shape = (TRAIN_EPISODES, MAX_STEPS)
    return {
        "explore": rng.random(shape),
        "action": rng.integers(0, 2, shape),
        "tie": rng.integers(0, 2, shape),
        "slip": rng.random(shape),
    }

def q_target(reward: float, next_state: int, q: np.ndarray) -> float:
    # A rollout timeout is not a terminal state in the underlying task.
    return reward if next_state == GOAL else reward + GAMMA * q[next_state].max()

def train(tape: dict[str, np.ndarray], shaped: bool, phi=potential) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
    q = np.zeros((N_STATES, 2), dtype=float)
    solved: list[int] = []
    steps: list[int] = []

    for episode in range(TRAIN_EPISODES):
        state = 0
        reached_goal = False

        for step in range(MAX_STEPS):
            if tape["explore"][episode, step] < EPSILON:
                action = int(tape["action"][episode, step])
            else:
                best = np.flatnonzero(q[state] == q[state].max())
                action = int(best[tape["tie"][episode, step] % len(best)])

            move = -1 if action == 0 else 1
            if tape["slip"][episode, step] < SLIP_PROBABILITY:
                move = -move

            next_state = int(np.clip(state + move, 0, GOAL))
            environment_reward = float(next_state == GOAL)
            shaping_reward = (
                GAMMA * phi(next_state) - phi(state)
                if shaped
                else 0.0
            )
            training_reward = environment_reward + shaping_reward
            target = q_target(training_reward, next_state, q)
            q[state, action] += ALPHA * (target - q[state, action])
            state = next_state

            if state == GOAL:
                reached_goal = True
                break

        solved.append(int(reached_goal))
        steps.append(step + 1)

    return q, np.asarray(solved), np.asarray(steps)

def evaluate(q: np.ndarray, seed: int, episodes: int = EVAL_EPISODES) -> float:
    slips = np.random.default_rng(10_000 + seed).random((episodes, MAX_STEPS))
    successes: list[bool] = []

    for episode in range(episodes):
        state = 0
        for step in range(MAX_STEPS):
            action = int(np.argmax(q[state]))
            move = -1 if action == 0 else 1
            if slips[episode, step] < SLIP_PROBABILITY:
                move = -move
            state = int(np.clip(state + move, 0, GOAL))
            if state == GOAL:
                break
        successes.append(state == GOAL)

    return float(np.mean(successes))

rows: list[tuple] = []
for seed in SEEDS:
    tape = make_tape(seed)
    sparse_q, sparse_train, sparse_steps = train(tape, shaped=False)
    shaped_q, shaped_train, shaped_steps = train(tape, shaped=True)
    zero_q, zero_train, zero_steps = train(tape, shaped=True, phi=lambda state: 0.0)
    np.testing.assert_array_equal(zero_q, sparse_q)
    np.testing.assert_array_equal(zero_train, sparse_train)
    np.testing.assert_array_equal(zero_steps, sparse_steps)
    rows.append(
        (
            seed,
            float(sparse_train[:EARLY_EPISODES].mean()),
            float(shaped_train[:EARLY_EPISODES].mean()),
            evaluate(sparse_q, seed),
            evaluate(shaped_q, seed),
            int(sparse_steps.sum()),
            int(shaped_steps.sum()),
        )
    )

results = np.asarray(rows, dtype=float)
early_delta = results[:, 2] - results[:, 1]
final_delta = results[:, 4] - results[:, 3]

bootstrap_rng = np.random.default_rng(20_260_731)
bootstrap_means = np.asarray(
    [
        early_delta[
            bootstrap_rng.integers(0, len(early_delta), len(early_delta))
        ].mean()
        for _ in range(BOOTSTRAP_REPLICATES)
    ]
)
interval_low, interval_high = np.quantile(bootstrap_means, [0.05, 0.95], method="linear")

final_rng = np.random.default_rng(20_260_732)
final_bootstrap = final_delta[final_rng.integers(0, len(rows), (BOOTSTRAP_REPLICATES, len(rows)))].mean(axis=1)
final_low, final_high = np.quantile(final_bootstrap, [0.05, 0.95], method="linear")
# Sensitivity only: independent arm resampling removes the observed pairing.
unpaired_rng = np.random.default_rng(20_260_733)
shape_index = unpaired_rng.integers(0, len(rows), (BOOTSTRAP_REPLICATES, len(rows)))
sparse_index = unpaired_rng.integers(0, len(rows), (BOOTSTRAP_REPLICATES, len(rows)))
unpaired_means = results[shape_index, 2].mean(axis=1) - results[sparse_index, 1].mean(axis=1)
unpaired_low, unpaired_high = np.quantile(unpaired_means, [0.05, 0.95], method="linear")

primary_supported = early_delta.mean() >= 0.10 and interval_low > 0.0
guardrail_passed = abs(final_delta.mean()) <= 0.02 and final_low >= -0.02 and final_high <= 0.02
supported = primary_supported and guardrail_passed

print(f"paired seeds: {len(SEEDS)}")
print("zero-potential control: exact equality on all 24 seeds")
print(f"early sparse mean: {results[:, 1].mean():.3f}")
print(f"early shaped mean: {results[:, 2].mean():.3f}")
print(f"mean paired delta: {early_delta.mean():+.3f}")
print(f"90% paired bootstrap CI: [{interval_low:+.3f}, {interval_high:+.3f}]")
print(
    "seeds improved / tied / regressed: "
    f"{int(np.sum(early_delta > 0))} / "
    f"{int(np.sum(early_delta == 0))} / "
    f"{int(np.sum(early_delta < 0))}"
)
print(f"final sparse mean: {results[:, 3].mean():.3f}")
print(f"final shaped mean: {results[:, 4].mean():.3f}")
print(f"mean final delta: {final_delta.mean():+.3f}")
print(f"90% final-difference interval: [{final_low:+.3f}, {final_high:+.3f}]")
print(f"unpaired sensitivity interval: [{unpaired_low:+.3f}, {unpaired_high:+.3f}]")
print(f"mean training transitions sparse / shaped: {results[:, 5].mean():.1f} / {results[:, 6].mean():.1f}")
print(f"decision: {'SUPPORT UNDER RULE' if supported else 'WITHHOLD SUPPORT'}")
print(f"guardrail: {'PASS' if guardrail_passed else 'FAIL'}")

paired seeds: 24
zero-potential control: exact equality on all 24 seeds
early sparse mean: 0.439
early shaped mean: 0.582
mean paired delta: +0.143
90% paired bootstrap CI: [+0.061, +0.225]
seeds improved / tied / regressed: 19 / 0 / 5
final sparse mean: 0.911
final shaped mean: 0.911
mean final delta: +0.000
90% final-difference interval: [+0.000, +0.000]
unpaired sensitivity interval: [+0.046, +0.235]
mean training transitions sparse / shaped: 773.2 / 754.7
decision: SUPPORT UNDER RULE
guardrail: PASS

Read the result without hiding variance

Before reading the headline mean, inspect the seed distribution. The declared rule needs a mean of at least +0.10 and an interval above zero, but neither tells you whether the candidate improved every seed.

View larger

The worked mean is $+0.143$ (+14.3 percentage points), above the declared $+0.10$ point-estimate threshold. Its approximate 90% paired percentile interval is $[+0.061,+0.225]$: above zero, but including effects smaller than ten points. This run passes the displayed rule. A claim of at least ten points at 90% confidence needs a lower endpoint above $+0.10$. Post-result protocol choices and the absence of independent preregistration also limit the interpretation.

Independent arm resampling gives $[+0.046,+0.235]$, about 15% wider than the paired interval in this run. The observed early-score covariance is positive, approximately $0.009525$, consistent with the variance reduction in the formula. This sensitivity analysis resamples the same completed rows. Retraining agents or assessing shared-tape precision requires separate runs.

The five regressions and exploratory traps

Five out of 24 seeds regressed ($\Delta_i<0$): 6, 11, 17, 19, and 23. Keep those points beside the mean. Seeds 9, 13, and 21 actually improved; identifying regressions requires reading the rows, not supplying a plausible list.

Why would a potential that points toward the goal hurt early learning?

One possible influence is the learning signal after a slip. A realized move from state 0 to state 1 adds $+0.208$. A realized return from state 1 to state 0 adds:

Which Q entry changes? The entry for the starting state and intended action. If the agent intended right at state 1 and slipped back to 0, it updates $Q(1,\text{right})$, not $Q(0,\text{right})$. Also, the target includes the next-state bootstrap value; a negative shaping term alone doesn't determine the update's sign.

Check the mechanics with zero initial Q-values. This constructed transition is not a trace from one of the five regressions:

gamma, alpha, goal = 0.95, 0.35, 6
phi = lambda s: -(goal - s) / goal
q = [[0.0, 0.0] for _ in range(goal + 1)]
state, intended_action, next_state = 1, 1, 0
bonus = gamma * phi(next_state) - phi(state)
target = bonus + gamma * max(q[next_state])
q[state][intended_action] += alpha * (target - q[state][intended_action])
assert q[0][1] == 0.0 and q[1][1] < 0.0
print(f"Q(1, right) after slipped action: {q[1][1]:+.3f}")
print(f"Q(0, right) remains: {q[0][1]:+.3f}")

Q(1, right) after slipped action: -0.041
Q(0, right) remains: +0.000

Shaping changes finite learning dynamics, but the row table doesn't reveal which transitions caused each regression. Preserve diagnostic traces and test a proposed mechanism in a new study. Before the first goal, sparse zero-initialized Q-values stay zero; that fact alone doesn't explain why sparse wins a particular seed.

Dropping the five negative differences would raise the mean from $+0.143$ to approximately $+0.239$. That is selection on the outcome, not a valid exclusion rule. Report all 24 pairs and label explanations exploratory.

Assess competence separately from equivalence

Seed 18 offers a different reality check. Both the sparse baseline and the shaped candidate achieved only 45.5% greedy success (91 out of 200 evaluation episodes).

This pair's difference is $+0.000$ ($0.455-0.455$), consistent with the aggregate guardrail. Both policies remain weak on this seed, and the result says nothing about convergence. Inspecting the frozen Q-tables shows both greedy policies choose left at state 0 and right at states 1 through 5. At the start, intended left is clamped back to 0 unless a slip reverses it, so the agent relies on a slip to leave.

The difference guardrail checks observed final success similarity. Policy invariance and objective equivalence are separate theorem questions; absolute competence requires an appropriate task metric. The shaping theorem addresses optimal policies in the specified MDP, while the finite-run guardrail addresses measured outcomes of these learned policies.

Reconstruct both observations from the executed study rather than typing them into the report:

regressed_seeds = results[early_delta < 0, 0].astype(int).tolist()
print("Regressed seeds:", regressed_seeds)
print(f"Mean using all seeds: {early_delta.mean():+.3f}")
print(f"Mean after improper outcome exclusion: {early_delta[early_delta > 0].mean():+.3f}")
for shaped in (False, True):
    frozen_q, _, _ = train(make_tape(18), shaped=shaped)
    policy = np.argmax(frozen_q[:GOAL], axis=1).tolist()
    score = evaluate(frozen_q, seed=18)
    arm = "shaped" if shaped else "sparse"
    print(f"Seed 18 {arm}: actions={policy}, success={score:.3f}")

Regressed seeds: [6, 11, 17, 19, 23]
Mean using all seeds: +0.143
Mean after improper outcome exclusion: +0.239
Seed 18 sparse: actions=[0, 1, 1, 1, 1, 1], success=0.455
Seed 18 shaped: actions=[0, 1, 1, 1, 1, 1], success=0.455

Both arms received 80 training episodes, not an identical number of environment steps: means were 773.2 transitions for sparse and 754.7 for shaped. Reaching the goal terminates an episode early, so successful agents consume fewer steps. This design measures early success per episode, not wall-clock efficiency or success at an identical step budget.

Separate experimental evidence from theory

Potential-based shaping theory motivates candidate design. This experiment checks implementation behavior under one finite setup; the theorem states its own assumptions and result.

A bootstrap interval summarizes sampling uncertainty under this study's design. External validity depends on how new environments, algorithms, reward scales, horizons, and hyperparameters change learning dynamics.

Use precise verbs: observed, estimated, passed the declared rule, and didn't detect. A fresh confirmatory study can say supported under the preregistered rule when that history exists. Avoid proved, guaranteed, always, and works in production.

Preserve the complete seed table

Report seed, sparse early rate, shaped early rate, paired difference, sparse final rate, shaped final rate, and run status. Sort by seed, not performance.

Keep failed mechanical runs in the ledger with their errors. If the locked rerun rule permits replacement, preserve the first failed attempt and link the replacement rather than overwriting it.

The aggregate table should point to the row-table checksum. A screenshot alone can't support reanalysis.

Treat negative results as artifacts

The capstone succeeds when it identifies which locked claim the evidence supports. A strong artifact makes that result easy to understand, rerun, and extend.

Keep the report structure unchanged when the claim fails. Replace celebratory language with the declared decision, the uncertainty, the likely explanations, and the next discriminating study. Don't quietly change the metric or the seed set.

Use this diagnosis table after locked analysis:

A negative result can still narrow future work. For example, a wide interval may justify more seeds; a stable null effect may rule out one potential; guardrail failure may expose incorrect terminal shaping.

Mark post-result hypotheses exploratory. "Early slips explain seed 6's regression" is a hypothesis for diagnostic traces and a new registered study, not a mechanism established by this aggregate row table.

Produce a figure, table, and report

A research artifact needs three aligned views. The table preserves exact values, the figure makes the distribution visible, and the report connects evidence to the claim. All three must derive from the same immutable rows, or a polished report can drift away from the run it describes.

Generate the table and figure from artifacts/runs/primary_seed_rows.csv. Don't type result numbers into plot code or the report by hand. Check that derived rows and aggregate values match the run before recording file checksums. Matching hashes establishes byte identity, not scientific validity.

This lesson's bundle includes illustrations/run_study.py: it executes the named experiment fence and exports primary_seed_rows.csv, study_results.json, and artifact_manifest.json. The result figure reads that JSON rather than a second list of numbers. From the repository root, regenerate those artifacts with:

uv run web/src/content/foundations/capstone-reproducible-research-study/illustrations/run_study.py

The manifest records the executed code hash, exporter hash, config, runtime, and output hashes. The run is labeled as worked evidence because it was not independently preregistered. A repeat run should reproduce the row and JSON hashes under the same recorded environment; an independently implemented simulator is a stronger cross-check against shared bugs.

Use accessible colors plus sign and position. Negative seed differences need a label or distinct shape so interpretation doesn't depend on red-green perception.

Required table

report/tables/primary_result.csv should contain one row per arm plus paired delta summary:

Required figure

Show every seed difference on a common x-axis with a visible zero line. Add the paired mean and interval. The figure above is that view: the scatter carries the five regressions, while the mean and interval together show why this run passes the declared rule.

Write a caption that states the inference, not a restatement of the title. Mention the five negative seeds. Avoid a bar chart that hides the seed distribution.

Export SVG for the report and PNG for slides. Save the plotting command and source hash in the manifest.

Required report

Keep the report short enough to review in one sitting:

1. Abstract with scope and decision.
2. Paper claim and study adaptation.
3. Locked hypothesis, expected result, and rejection rule.
4. Environment, baseline, candidate, controls, and seed plan.
5. Primary table, figure, uncertainty, and negative seeds.
6. Ablations and mechanical checks.
7. Limitations, negative findings, and next study.
8. Reproduction command, manifest, license, and citation.

State deviations from plan in their own table. none is acceptable only after checking commit diff between locked plan and final config.

Write limitations that constrain the claim

Limitations aren't a ceremonial paragraph. Each should identify an unsupported inference and, when possible, the experiment that would test it. Treat the table as a map of where the current evidence stops.

This capstone has at least six:

Also state the theorem's boundary. Potential-based shaping preserves optimal policies under the assumptions in the source formulation. A finite implementation can still be wrong, undertrained, or measured on the wrong reward.

The public-safe synthetic setup is easy to audit. Claims about user behavior need representative user data.

Stage the evidence for review

Each stage ends with a reviewable evidence package before the next begins.

Apply these stages to a new study with untouched confirmatory seeds. The worked rows already demonstrate analysis; a new seed set collected after the plan is frozen supplies confirmatory history.

Keep reviewer feedback in reviews/ or an issue tracker. When feedback changes the locked plan, record an amendment before the confirmatory run.

The four stages move from claim selection to a defensible public artifact:

Milestone 1: paper and study contract

Submit paper_triage.md, study_plan.md, source links, one-paragraph scope, falsifiable hypothesis, expected result, rejection rule, public-safety boundary, and limitations forecast.

At this point, a reviewer should trace the shaping equation to the paper and distinguish paper claims from study adaptations. No confirmatory results should exist yet.

Move on when another engineer can predict which output supports, rejects, or leaves the claim inconclusive.

Milestone 2: simulator and baseline

Submit the exact repository skeleton, environment tests, sparse Q-learning baseline, pilot-only run, zero-potential equality test, config schema, dependency lock, and clean-run command.

The reviewer runs seed 900 and inspects one episode trace, confirming goal reward, slip behavior, terminal handling, and the 12-step cap.

Move on when baseline and zero-potential control are bit-for-bit equal under the same tape.

Milestone 3: confirmatory evidence

Submit the retained row table for all planned seed pairs, paired bootstrap output, experiment table, manifest, checksums, primary figure, planned ablations, and deviation log. Use storage or release history that preserves earlier outputs; a CSV is editable by itself.

The reviewer reconstructs headline values from the row table and reruns at least one seed pair. Mechanical failures remain visible.

Move on when the decision follows the locked rule without manual number edits.

Milestone 4: public artifact and defense

Submit the final report, exact reproduction instructions, license, CITATION.cff, archived release identifier if available, ten-minute talk outline, Q&A sheet, and artifact review checklist.

The reviewer starts from a clean clone and produces the primary table. They also identify one unsupported claim and check that the report limits its conclusions accordingly.

The artifact is ready when an independent reviewer can exercise it and explain its evidence, limits, and next study.

Prepare a ten-minute talk

A short technical talk tests whether your research decisions form a coherent story. Ten minutes leaves no room for every implementation detail, so use an evidence hierarchy rather than a file tour.

Practice with a timer and one backup slide per likely challenge. The main deck should fit seven slides and reserve at least two minutes for questions.

Use this pacing:

Build Q&A from attack surfaces

Write the answers before the presentation. Each answer should point to an evidence file or admit unresolved uncertainty. That keeps Q&A from becoming a second, improvised results section.

Don't answer every challenge with "more data." Some failures demand corrected control, better estimand, or narrower claim.

Review the artifact like an outsider

Under ACM's current terminology, repeatability is the same team obtaining a result with the same setup. Reproducibility is a different team obtaining it with the authors' artifacts, and replicability is a different team obtaining it with independently developed artifacts.^{[14]Reference 14Artifact Review and Badginghttps://www.acm.org/publications/policies/artifact-review-and-badging-current}

This capstone targets repeatability first and provides enough documentation for an independent reproducibility attempt.

ACM's functional artifact criteria emphasize documented, consistent, complete, exercisable artifacts with verification and validation evidence.^{[14]Reference 14Artifact Review and Badginghttps://www.acm.org/publications/policies/artifact-review-and-badging-current} Use those words as review questions and support any claim of compliance with the corresponding evidence.

FAIR principles add findability, accessibility, interoperability, and reuse to data stewardship.^{[15]Reference 15The FAIR Guiding Principles for scientific data management and stewardshiphttps://www.nature.com/articles/sdata201618} Accessible data can require authentication and authorization; FAIR doesn't require unrestricted public access. This synthetic study can be public. Open formats, machine-readable metadata, a license, and a persistent identifier help, but don't replace executable validation.

Outsider checklist

• Can a reviewer find the hypothesis and locked commit?
• Can a reviewer install the environment without global packages?
• Can a reviewer regenerate synthetic rows from config and seeds?
• Can a reviewer verify source, config, and result checksums?
• Can a reviewer recompute the table and figure from the same rows?
• Can a reviewer see failed runs and plan deviations?
• Can a reviewer distinguish confirmatory from exploratory work?
• Can a reviewer state exactly where the claim stops?
• Can a reviewer cite the release and understand the license?
• Can a reviewer reproduce the primary decision from a clean clone?

If any answer is "no," the artifact isn't ready. Fix the package before polishing the talk.

Defend bounded evidence

One paper claim has to be a question a seed table can contradict, and the writeup has to say where that evidence stops. Write the expected result and the rejection rule before confirmatory output. Then keep the negative seeds and the failed hypotheses.

Paired tapes separate an algorithm change from some random noise. Bootstrap uncertainty exposes sampling sensitivity. Original-reward evaluation keeps the task fixed. Those checks transfer to model training, retrieval, evaluators, and agent benchmarks.

What you hand a reviewer is the source, lockfile, config, seed table, manifest, figure, report, limitations, talk, and Q&A. The package is credible when the claim stays smaller than that evidence.