Capstone: Reproducible ML Study

A research demo can look convincing and still collapse when someone asks for the exact command, seed, data, or baseline. One chart isn't evidence if nobody can tell which choices produced it.

This capstone turns a small idea into a research artifact another engineer can inspect and rerun. You'll read one paper, lock a falsifiable claim, run a controlled offline study, report uncertainty and failures, then defend the result in ten minutes.

The running study asks whether a safe form of reward shaping helps a tiny agent learn faster. Everything runs locally with synthetic data and NumPy. No private logs, live users, paid APIs, or specialized hardware enter the study.

Triage one paper before building

Start with Ng, Harada, and Russell's paper on policy invariance under reward transformations.^{[1]Reference 1Policy Invariance Under Reward Transformations: Theory and Application to Reward Shapinghttps://wordpress.andrewng.org/index.php/publication/policy-invariance-under-reward-transformations-theory-and-application-to-reward-shaping/} Its main claim is precise enough to test in a small simulator: a potential-based shaping reward can guide learning while preserving optimal policies under the theorem's assumptions.

Don't begin by reproducing every table or proof. First decide whether the paper contains one claim that fits your time, compute, and safety constraints. Keshav's three-pass reading method provides a practical sequence: classify the paper, inspect its evidence, then reconstruct the part you plan to test.^{[2]Reference 2How to Read a Paperhttps://dl.acm.org/doi/10.1145/1273445.1273458}

Use this triage card while reading. Fill it with page or section pointers so a reviewer can trace your interpretation back to source text.

First pass: classify the claim

Read title, abstract, introduction, headings, figures, conclusion, and references. Write one sentence for problem, method, comparison, and claimed result. If those sentences stay vague, the study isn't ready to design.

For this paper, the method isn't "add helpful rewards." Instead, the method derives a transition reward from a potential function. That distinction separates the policy-invariance theorem from arbitrary bonuses that may change the task.

Stop the first pass with a scope decision: reproduce, extend, or decline. This capstone performs a small empirical check of one implication and adds an uncertainty-focused artifact. It doesn't attempt a full paper reproduction.

Second pass: inspect evidence and assumptions

Study the equation, experimental setup, plots, and failure caveats. Write down what stays constant, what changes, and what the paper measures. Missing implementation detail belongs in an assumptions log, not in a silent guess.

Pay close attention to evaluation reward. A candidate trained with shaped reward can look better if its score includes the extra training bonus. Compare both learned policies using the original environment reward or you have changed the measurement.

Record every adaptation. The corridor, Q-learning hyperparameters, action-slip probability, seed budget, and bootstrap rule are capstone choices rather than claims copied from the paper.

Third pass: reconstruct the test

Explain the shaping equation without looking. Rebuild the smallest environment where moving toward a goal changes potential. Predict what should happen to early learning and final policy before running code.

Then challenge the design. Could the candidate receive more steps, a friendlier random stream, or an easier evaluation? Could a single lucky seed drive the mean? Every plausible shortcut becomes a control, ablation, or stated limitation.

Finish paper_triage.md with three labels: supported by source, study adaptation, and unresolved. That file prevents a plausible implementation detail from turning into a misattributed paper claim.

Turn curiosity into a falsifiable contract

"Does reward shaping help agents learn?" is too broad. It doesn't identify an agent, environment, shaping rule, metric, sample, or outcome that would count against the idea.

A falsifiable hypothesis names an observation that can contradict it. Narrow scope makes the capstone stronger because every noun can become a file, parameter, table column, or assertion.

Use this research question:

In a stochastic seven-state corridor, does potential-based shaping increase tabular Q-learning's first-30-episode goal-reaching rate versus sparse reward, while keeping final greedy-policy success unchanged under original reward?

Lock the contract before inspecting confirmatory results:

State the expected result

Prediction comes before output. Expect shaping to improve early goal-reaching because each move changes a distance-based potential. Expect final greedy success to match because the shaping term has the potential-based form studied by Ng and colleagues.^{[1]Reference 1Policy Invariance Under Reward Transformations: Theory and Application to Reward Shapinghttps://wordpress.andrewng.org/index.php/publication/policy-invariance-under-reward-transformations-theory-and-application-to-reward-shaping/}

That expectation isn't permission to hide a miss. It gives reviewers a record of what you believed before seeing confirmatory values. Exploratory explanations can follow, but they need an explicit label.

Write this expected pattern in study_plan.md before running seeds 0 through 23:

• positive early paired difference
• interval above zero
• near-zero final difference
• timestamp or immutable commit for locked plan

A commit makes later edits visible, but it isn't equivalent to independent preregistration. For formal work, register hypotheses, variables, tests, decision criteria, and exclusions in a timestamped read-only system before collecting confirmatory results.^{[3]Reference 3Welcome to Registrationshttps://help.osf.io/article/330-welcome-to-registrations}

Write the rejection rule

Support the capstone's narrow claim only if the mean early difference is at least +0.10 and the 90% interval's lower bound is above zero. Also require the final-policy guardrail.

Reject the release claim when either requirement fails. A mean of +0.09 is below the declared effect threshold even if its interval is positive. A mean of +0.20 also fails if final original-reward performance shifts beyond the guardrail.

"Inconclusive" can be accurate when interval width leaves multiple practical interpretations. Keep that label distinct from "no effect" and from "the candidate is worse."

Keep the study public-safe

Small research work still needs a data and authority boundary. Synthetic data removes many privacy risks, but a public repository can still leak credentials, copied code, or outputs from proprietary environments.

The corridor generator owns every transition. It doesn't imitate a customer, scrape a service, call an external model, or exercise a live agent. Runs can be shared because their complete input is a tiny configuration plus pseudorandom seed.

Add this boundary to the README before code:

Run a secret scanner before publishing. Review generated files for machine paths and usernames. Use an open license only for code and assets you have authority to release.

Build the smallest controlled environment

The corridor has states 0 through 6. Every episode starts at 0; reaching 6 yields environment reward 1; every other transition yields 0. An intended action reverses with probability 0.15, and an episode ends after 12 steps.

Write the environment contract before writing the agent. This prevents reward, reset, or ending-condition changes from entering one experimental arm unnoticed.

terminated means task reached goal. truncated means measurement budget ended episode. Keep distinction in run rows because treating timeout as terminal success or bootstrapping through true terminal changes learning target.

Tabular Q-learning stores one value for each state-action pair. During training, it updates the chosen action toward observed reward plus the best estimated next-state value. Sutton and Barto provide the broader reinforcement-learning foundation behind this update.^{[4]Reference 4Reinforcement Learning: An Introductionhttp://incompleteideas.net/book/the-book-2nd.html}

The first diagram follows one environment transition. Both arms then read the same environment reward; only the candidate receives a training-only potential difference before its update. Final evaluation reads the original reward branch for both arms.

Define distance potential as:

$$\Phi(s) = -\frac{6-s}{6}.$$

States nearer the goal have a larger potential. Candidate training adds:

$$F(s,a,s') = \gamma \Phi(s') - \Phi(s).$$

With discount 0.95, a move toward the goal usually produces a positive shaping signal. A move away usually produces a negative one. The environment reward hasn't changed; only candidate training receives $F$.

Lock controls, seeds, and ablations

Reinforcement-learning measurements can vary with random seeds, environment randomness, codebase, and hyperparameters. Henderson and colleagues show why a single run or loosely standardized baseline can mislead.^{[5]Reference 5Deep Reinforcement Learning That Mattershttps://ojs.aaai.org/index.php/AAAI/article/view/11694}

Pair each baseline run with candidate run under one pre-generated random tape. Tape stores exploration draws, random actions, tie-break draws, and action slips for every episode and step. Indexing by episode and step prevents early episode termination from shifting all later randomness.

Pairing reduces irrelevant seed noise in the difference estimator. It doesn't erase algorithm variability: policies visit different states, so identical tape entries can affect the arms differently.

Separate pilot and confirmatory seeds

Use seeds 900, 901, and 902 to debug code, plots, and manifests. Never include them in confirmatory metrics because you have already observed their behavior.

Use integers 0 through 23 for the locked study. Twenty-four paired seeds fit a laptop capstone and expose heterogeneity, but they aren't a universal sample-size recommendation.

Run every pair. Stopping after ten favorable seeds would make sample size depend on observed outcome. If a run fails mechanically, record failure and rerun policy before looking at aggregate results.

Add controls before interpretation

Zero-potential control sets $\Phi(s)=0$. Under the same tape, it must match sparse training exactly. If it doesn't, candidate code changed more than reward.

Original-reward evaluation is another control. Both arms receive identical greedy evaluation slips, and neither receives shaping reward. A result reported only on shaped return fails the measurement contract.

Use a sign-reversed potential only as an exploratory stress test after confirmatory analysis. It may reveal sensitivity to learning dynamics, but it can't retroactively redefine the primary hypothesis.

Plan the ablation matrix

An ablation removes or changes one component to identify which mechanism drives a result. Change one factor per row and preserve all other settings.

Don't call an unplanned hyperparameter sweep an ablation after choosing its best row. Preserve all runs and label post-result searches exploratory.

Define metrics and uncertainty

For seed $i$, let $p^{shape}_{i,30}$ and $p^{sparse}_{i,30}$ be fractions of first 30 training episodes that reach the goal. The paired effect is:

$$d_i = p^{shape}_{i,30} - p^{sparse}_{i,30}.$$

Primary estimate averages all 24 differences:

$$\bar{d} = \frac{1}{24}\sum_{i=1}^{24}d_i.$$

That number answers a bounded question about mean early success under the declared seed-generating procedure. It doesn't prove every seed improves, so report all $d_i$ values and count regressions.

Bootstrap paired seed differences

A bootstrap resamples the 24 paired differences with replacement and recomputes their mean. Repeating that process 10,000 times gives a distribution of plausible mean estimates under the empirical seed sample.

Use fixed bootstrap seed 20260731. Report 5th and 95th percentiles as a 90% interval. Agarwal and colleagues emphasize that finite-run reinforcement-learning comparisons need uncertainty views rather than point estimates alone.^{[6]Reference 6Deep Reinforcement Learning at the Edge of the Statistical Precipicehttps://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html}

The interval isn't a probability that theorem is true, and its coverage doesn't extend automatically beyond this simulator. It quantifies sampling uncertainty over declared seed pairs, conditional on code and study design.

Keep a final-policy guardrail

After 80 training episodes, evaluate each greedy policy for 200 episodes on original reward. Pair evaluation slip tapes by seed. Compute shaped-minus-sparse final-success difference exactly as above.

Require absolute mean final difference at most 0.02. Equal final means don't prove policy invariance generally; they show this finite implementation didn't detect a final-performance change at declared resolution.

Also inspect per-seed final differences. An average near zero can hide equal and opposite failures if candidate helps one region and harms another.

Plan the experiment table before running

A compact experiment table stops notebook state from becoming methodology. Each row names exact config, sample, output, and decision role.

Store the table in study_plan.md and mirror machine-readable fields in configs/study.json. Freeze both in the same commit as the hypothesis.

Use one status vocabulary: planned, running, complete, mechanical_failure, or excluded_by_locked_rule. "Bad result" isn't an exclusion reason.

Package an exact repository

A reviewer shouldn't have to guess where hypothesis ends and analysis begins. Keep source, config, immutable run outputs, report, and talk in distinct paths.

This tree is small enough for a public repository while still representing a credible research artifact:

Pin environment and entrypoint

Use pyproject.toml plus uv.lock to pin NumPy and development tools. README should name supported Python version and one clean command such as uv run corridor-study --config configs/study.json.

Run from a fresh clone or container. Record operating system, architecture, Python, NumPy, commit, config hash, and wall-clock duration in manifest. The machine identity may remain coarse; hostnames and usernames don't belong in public output.

The Machine Learning Reproducibility Checklist calls for clear model, hyperparameter, seed, environment, and result reporting.^{[7]Reference 7The Machine Learning Reproducibility Checklisthttps://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf} A lockfile helps, but complete execution instructions and immutable inputs still matter.

Record data provenance

Synthetic data still needs provenance. Save generator version, seed range, transition rule, slip probability, state count, step cap, and environment source hash.

Don't commit one unexplained CSV and call it generated. data/generated/README.md should state exact command that reconstructs any derived rows and whether tracked output is canonical or disposable.

If you adapt capstone to public dataset later, record original URL, license, version, retrieval date, checksum, preprocessing command, and excluded rows. Never replace original identifier with local filename alone.

Create an artifact manifest

The manifest connects the claim to exact files. JSON stays easy to inspect and hash.

The sample below names identities rather than mutable labels such as latest:

Add SHA-256 checksums after results become immutable. Verify the checksums in the clean-run script before rendering the report.

CITATION.cff gives software human-readable and machine-readable citation metadata. GitHub can render that file as citation formats for repository visitors.^{[8]Reference 8About CITATION fileshttps://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-citation-files} Use release version or persistent identifier after archive.

Run the miniature study

The complete script below generates paired random tapes, trains both arms, evaluates on original reward, bootstraps paired differences, and prints decision. Its only dependency is NumPy.

Read control points before running: make_tape freezes randomness by episode and step; train changes only reward; evaluate never receives shaping reward; bootstrap samples paired differences.

Run it with uv run reward_shaping_study.py. Expected output follows code.

Read the result without hiding variance

The primary estimate clears the locked +0.10 threshold, and the 90% interval stays above zero. The final original-reward mean is identical at printed precision. Under the declared rule, the fixture supports H1 and passes the guardrail.

Five of 24 seeds regress. That observation belongs next to headline mean because it shows shaping isn't uniformly better. A reviewer can inspect those rows, rerun them, and ask whether one potential interacts poorly with early exploration.

The result supports one sentence: "In this corridor fixture, potential-based shaping increased mean first-30-episode success by 0.143 across 24 paired seeds, with a 90% paired bootstrap interval of [0.061, 0.225], while mean final original-reward success was unchanged."

Separate support from proof

Potential-based shaping theory motivates candidate design, but a tiny experiment doesn't prove the theorem. It verifies implementation behavior under one finite setup.

Likewise, a bootstrap interval isn't a certificate of external validity. New environments, algorithms, reward scales, horizons, and hyperparameters can change learning dynamics.

Use precise verbs: observed, estimated, supported under the locked rule, and didn't detect. Avoid proved, guaranteed, always, and works in production.

Preserve the complete seed table

Report seed, sparse early rate, shaped early rate, paired difference, sparse final rate, shaped final rate, and run status. Sort by seed, not performance.

Keep failed mechanical runs in the ledger with their errors. If the locked rerun rule permits replacement, preserve the first failed attempt and link the replacement rather than overwriting it.

The aggregate table should point to the row-table checksum. A screenshot alone can't support reanalysis.

Treat negative results as artifacts

A rejected hypothesis isn't failed capstone. Hidden rejection is. Strong artifact makes negative result easy to understand, rerun, and extend.

Keep report structure unchanged when claim fails. Replace celebratory language with declared decision, uncertainty, likely explanations, and next discriminating study. Don't quietly change metric or seed set.

Use this diagnosis table after locked analysis:

A negative result can still narrow future work. For example, a wide interval may justify more seeds; a stable null effect may rule out one potential; guardrail failure may expose incorrect terminal shaping.

Mark post-result hypotheses exploratory. "Seed 6 regressed because early slips trapped policy" can become next registered study, but it isn't confirmed by same observed seed trace.

Produce a figure, table, and report

A research artifact needs three aligned views. The table preserves exact values, the figure makes the distribution visible, and the report connects evidence to the claim. All three must derive from the same immutable rows.

Generate the table and figure from artifacts/runs/primary_seed_rows.csv. Don't type result numbers into plot code or the report by hand. Add a test that the report-table hash matches the manifest.

Use accessible colors plus sign and position. Negative seed differences need a label or distinct shape so interpretation doesn't depend on red-green perception.

Required table

report/tables/primary_result.csv should contain one row per arm plus paired delta summary:

Required figure

Show every seed difference on common x-axis with visible zero line. Add paired mean and interval, then align early and final arm means on same zero-to-one scale.

Caption should state inference rather than repeat title. Mention five negative seeds and equal final means. Avoid a bar chart that hides seed distribution.

Export SVG for report and PNG for slides. Save plotting command and source hash in manifest.

Required report

Keep report concise enough to review in one sitting:

1. Abstract with scope and decision.
2. Paper claim and study adaptation.
3. Locked hypothesis, expected result, and rejection rule.
4. Environment, baseline, candidate, controls, and seed plan.
5. Primary table, figure, uncertainty, and negative seeds.
6. Ablations and mechanical checks.
7. Limitations, negative findings, and next study.
8. Reproduction command, manifest, license, and citation.

State deviations from plan in their own table. none is acceptable only after checking commit diff between locked plan and final config.

Write limitations that constrain the claim

Limitations aren't a ceremonial paragraph. Each should identify unsupported inference and, when possible, experiment that would test it.

This capstone has at least six:

Also state the theorem's boundary. Potential-based shaping preserves optimal policies under the assumptions in the source formulation; a finite implementation can still be wrong, undertrained, or measured on the wrong reward.

Don't describe public-safe synthetic setup as representative of users. Its strength is auditability, not realism.

Submit four milestones

Each submission should be reviewable before the next stage begins. A milestone is an evidence package, not a calendar date.

Keep reviewer feedback in reviews/ or issue tracker. When feedback changes locked plan, record amendment before confirmatory run.

Use these four submissions:

Milestone 1: paper and study contract

Submit paper_triage.md, study_plan.md, source links, one-paragraph scope, falsifiable hypothesis, expected result, rejection rule, public-safety boundary, and limitations forecast.

Reviewer should trace shaping equation to paper and distinguish paper claims from study adaptations. No confirmatory results should exist yet.

Exit criterion: another engineer can predict which output supports, rejects, or leaves claim inconclusive.

Milestone 2: simulator and baseline

Submit exact repository skeleton, environment tests, sparse Q-learning baseline, pilot-only run, zero-potential equality test, config schema, dependency lock, and clean-run command.

Reviewer should run seed 900 and inspect one episode trace. They should confirm goal reward, slip behavior, terminal handling, and 12-step cap.

Exit criterion: baseline and zero-potential control are bit-for-bit equal under same tape.

Milestone 3: confirmatory evidence

Submit immutable 24-seed row table, paired bootstrap output, experiment table, manifest, checksums, primary figure, planned ablations, and deviation log.

Reviewer should reconstruct headline values from row table and rerun at least one seed pair. Mechanical failures must remain visible.

Exit criterion: decision follows locked rule without manual number edits.

Milestone 4: public artifact and defense

Submit final report, exact reproduction instructions, license, CITATION.cff, archived release identifier if available, ten-minute talk outline, Q&A sheet, and artifact review checklist.

Reviewer should start from clean clone and produce primary table. They should also identify one unsupported claim and verify report doesn't make it.

Exit criterion: independent reviewer can exercise artifact and explain evidence, limits, and next study.

Prepare a ten-minute talk

A short technical talk tests whether your research decisions form a coherent story. Ten minutes leaves no room for every implementation detail, so use an evidence hierarchy rather than a file tour.

Practice with a timer and one backup slide per likely challenge. The main deck should fit seven slides and reserve at least two minutes for questions.

Use this pacing:

Build Q&A from attack surfaces

Write the answers before the presentation. Each answer should point to an evidence file or admit unresolved uncertainty.

Don't answer every challenge with "more data." Some failures demand corrected control, better estimand, or narrower claim.

Review artifact like an outsider

Reproducibility asks whether another team can obtain result using same artifacts and setup; replicability asks about independent artifacts under ACM's current terminology.^{[9]Reference 9Artifact Review and Badginghttps://www.acm.org/publications/policies/artifact-review-and-badging-current} This capstone targets repeatability first and provides enough documentation for an independent reproducibility attempt.

ACM's functional artifact criteria emphasize documented, consistent, complete, exercisable artifacts with verification and validation evidence.^{[9]Reference 9Artifact Review and Badginghttps://www.acm.org/publications/policies/artifact-review-and-badging-current} Use those words as review questions, not as a badge claim.

FAIR principles add findability, accessibility, interoperability, and reuse to data stewardship.^{[10]Reference 10The FAIR Guiding Principles for scientific data management and stewardshiphttps://www.nature.com/articles/sdata201618} A public repository, open formats, machine-readable manifest, license, and citation metadata help. They don't replace executable validation.

Outsider checklist

• Can reviewer find hypothesis and locked commit?
• Can reviewer install environment without global packages?
• Can reviewer regenerate synthetic rows from config and seeds?
• Can reviewer verify source, config, and result checksums?
• Can reviewer recompute table and figure from same rows?
• Can reviewer see failed runs and plan deviations?
• Can reviewer distinguish confirmatory from exploratory work?
• Can reviewer state exactly where claim stops?
• Can reviewer cite release and understand license?
• Can reviewer reproduce primary decision from clean clone?

If any answer is "no," the artifact isn't ready. Fix the package before polishing the talk.

Defend bounded evidence

You can move from one paper claim to a bounded empirical question without pretending a small study reproduces an entire field. You can state expected result and rejection rule before confirmatory output, then preserve negative seeds and failed hypotheses.

You can also separate algorithm change from random noise through paired tapes, report bootstrap uncertainty, and keep final evaluation on original task. Those habits apply to model training, retrieval, evaluators, and agent benchmarks.

Ship evidence another engineer can inspect: source, lockfile, config, seed table, manifest, figure, report, limitations, talk, and Q&A. The artifact is credible because the claim remains smaller than the evidence.