Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
Suppose a payment provider sends the same billing event twice. The application should create one invoice, not two. You hand billing_event_78291 to a coding agent, and it returns a confident pull request (PR) with a debounce on PayButton.tsx, a changed lockfile, and a green suite whose new test creates an invoice once. The duplicate still appears when the provider retries the webhook, because the server never enforces the rule that one event ID maps to one invoice.
The patch is valid TypeScript, and the suite is green. The workflow still failed because no change fixed the server behavior, no boundary kept unrelated files out, and no test replayed the same event. A reliable coding-agent workflow turns each requirement into evidence a reviewer can inspect.
Earlier lessons established this split. Code Generation & Sandboxing showed how to isolate generated code while it runs. Human-in-the-Loop Agent Architecture showed the same boundary for a risky write: a release assistant named Vega could propose promote_model, but a trusted host owned authorization.
A merge into main is that gated write again. In this workflow, the agent may prepare a local patch; publishing a branch or PR and merging it are separately authorized actions. Keep one question in view: can a reviewer point to a bounded change and replay its evidence before allowing it into main? The running example uses a fictional billing service whose internal handler returns an invoice ID. Replaying billing_event_78291 must return the original ID instead of creating another invoice. A real webhook may return only an acknowledgment; inspect persistent invoice state too.
Treat the agent as a worker inside a system
An inline coding assistant suggests text near your cursor. A coding agent combines a large language model (LLM) with tools that can inspect files, choose actions, edit the working tree, and run commands. Product labels overlap, so classify the enabled capabilities rather than the name. More reach means the workflow must define what the agent may touch, what it must prove, and who can release the result.
As checked on September 2, 2026, products expose that action surface in different ways. GitHub Copilot's cloud agent researches a repository, plans, edits a branch, and runs tests in an ephemeral GitHub Actions environment.[1] Claude Code reads a codebase, edits files, runs commands, and works across terminal, editor, desktop, and browser surfaces.[2] Their surfaces will change, but four responsibilities stay stable:
| Responsibility | Owner |
|---|---|
| Propose files, commands, and a patch | Coding agent |
| Admit filesystem, command, network, credential, and tool capabilities | Trusted runner or host |
| Decide whether evidence satisfies acceptance criteria | Tests, continuous integration (CI), reviewer, and product checks |
| Authorize merge or another external effect | Human or policy system outside the model |
Research often describes a tool-using loop with the ReAct pattern: interleave reasoning about the next step with actions and observations.[3] For repository work, hidden reasoning isn't the review artifact. A reviewer can inspect the files read, diff produced, commands run, failures observed, and final evidence packet.
SWE-agent calls the tool layer an agent-computer interface (ACI) and shows that repository navigation, editing, and execution tools change software-engineering results.[4] Clear tool names, structured arguments, bounded output, and fast checks give the agent a better working environment. They improve execution; they don't grant authority.
An autonomous software-engineering agent moves through five distinct lifecycle phases:
- Contract scoping: Pin down the behavior invariant, owned files, non-goals, and reproduction verifier before any code changes.
- Repository exploration: Trace call paths and locate definitions using indexed search tools rather than flooding the prompt context window.
- Reproduction test creation: Write a failing test that reproduces the bug against the unpatched base code, establishing the red baseline.
- Surgical patch iteration: Make minimal, targeted code edits with search-and-replace tools rather than rewriting whole files.
- Verification and handoff: Execute the reproduction verifier to prove the red-to-green transition, run regression checks, and package cryptographic commit-bound receipts for human review.
Tool design directly determines whether the agent succeeds during exploration and editing. In a repository with 50,000 lines of code across hundreds of files, stuffing entire directories into the context window burns hundreds of thousands of tokens, runs up API costs, and causes the model to overlook critical lines due to attention degradation. An effective ACI provides targeted exploration tools instead: ripgrep for literal string and route matches, Language Server Protocol (LSP) queries (find_definition, find_references) to trace symbol graphs in milliseconds, and Abstract Syntax Tree (AST) pattern searches to inspect method signatures without loading unreferenced code. This approach narrows the agent's working context down to a few thousand tokens of high-signal code.
The editing interface needs the same precision. When an agent rewrites an entire 800-line file to adjust four lines, it frequently drops helper methods, strips license headers, scrambles whitespace, or truncates mid-file when hitting output token limits. Fine-grained search-and-replace tools avoid this trap. The agent specifies the exact contiguous target_content to match and the replacement content to insert. If the target string isn't unique in the file, the tool rejects the operation immediately instead of corrupting source code. That constraint produces minimal Git diffs, saves output tokens, and makes human code review straightforward.
What changes when an inline coding assistant becomes a coding agent?
Answer
The system gains a larger action surface. The agent may inspect many files, edit the working tree, and run commands, so the host must control capabilities, collect evidence, and keep merge authority outside the model.
Write a task contract before code changes
Once the agent's role is clear, give it a contract before it edits. “Fix duplicate invoices” names a symptom, not an acceptance condition. Without more detail, the agent may propose changes to the frontend, payment configuration, dependencies, or deployment code. An underspecified request isn't permission for all those actions.
Write the contract around behavior, ownership, exclusions, checks, and execution policy:
Investigate duplicate invoice creation when
billing_event_78291is replayed.Owned files
web/src/billing/webhook.tsweb/src/billing/webhook.test.tsNon-goals
- Don't edit payment-provider configuration, auth, checkout UI, dependencies, or deployment files.
- Don't call production services or use production credentials.
Acceptance evidence
- Add a reproduction that replays one event ID twice.
- Show it failing before the patch and passing after the patch.
- Preserve normal invoice creation for a new event ID.
- Run
pnpm test --run billingandpnpm lint.Allowed tools and checks
- Repository read/search tools; patch tool restricted to the two owned files.
pnpm test --run billingandpnpm lintthrough the approved runner.- Host-controlled status and diff inspection, with external diff helpers disabled.
- No dependency installation, shell setup changes, commits, pushes, PR creation, or merge.
Delivery
- The host provides branch
agent/billing-event-78291and records its base revision in the restricted project runner.- Report changed files, exact commands, outputs, and remaining risk.
Stop condition
- After three failed repair attempts after the baseline reproduction, escalate with the last failure, suspected missing context, and a proposed rescope. Stop sooner for denied capability, scope expansion, or suspected secret exposure.
The pnpm commands assume this example project's manifest defines a billing test selector and a non-fixing lint script. Inspect the actual scripts and test collection before adopting them elsewhere. The executable examples below are standalone Python checks; they don't run a TypeScript service or start a coding-agent product.
Read each field as a separate guardrail. The written contract guides the agent and reviewer; host enforcement is what blocks actions. Each field answers a different question:
| Contract field | Question it answers |
|---|---|
| Behavior | What must become true? |
| Owned files | Where may the patch write? |
| Non-goals | Which tempting adjacent changes stay out? |
| Acceptance evidence | Which observations would support the invariant? |
| Allowed commands and tools | What may execute while producing the patch? |
| Delivery | Which review artifact must return? |
| Stop condition | When should the agent escalate instead of guessing again? |
File ownership is a review boundary, not a prediction that the bug must live in those files. If diagnosis shows that the invariant belongs in another module, the agent pauses and requests a rescope before editing it. That pause keeps a valid diagnosis from silently becoming a larger diff.
Why should an agent request a rescope instead of silently editing a newly discovered file?
Answer
The new file may change the task's blast radius, reviewer needs, or security risk. A rescope makes that decision explicit before the diff expands and preserves clear ownership.
Separate execution from integration
The task contract limits the intended patch. It doesn't yet limit what commands can do while the patch is being produced. Developers often conflate three distinct mechanisms here: a Git branch, a Git worktree, and an execution sandbox.
A Git branch is an integration artifact: a named pointer to a commit. It provides zero process isolation. When an agent runs pnpm test on a feature branch, that process runs with the full ambient authority of the host machine. It can inspect ~/.ssh or ~/.aws, query internal network services, or download untrusted dependencies before any commit exists. A branch can't stop an untrusted command from calling a production endpoint or modifying cloud infrastructure.
Git worktrees solve a different problem: local filesystem collisions. Using git worktree add <path> <branch> gives the agent a separate working directory linked to the same local repository. The agent inspects and edits files in that path without stashing your uncommitted changes, switching your active checkout, or thrashing the working tree. Multiple concurrent agents can each operate in dedicated worktrees without colliding on Git index locks.
The execution sandbox enforces the runtime boundary. Ephemeral containers, microVMs, or sandboxed runner environments provide kernel-level process isolation that keeps commands from affecting external systems.
For the billing task, the runner needs source files, fixtures, and named checks. It doesn't need production tokens, package publication, Terraform, email, or arbitrary outbound network access.
| Execution surface | Default for this task |
|---|---|
| Filesystem | Read repository; write owned paths and designated disposable test-output/cache directories in an isolated worktree. No host home directory or container-control socket. |
| Credentials | No production credentials; only fixtures or scoped test identity. |
| Network | Deny by default; admit a named endpoint only when a check requires it. |
| Commands | Admit named checks and read/diff tools from the contract. Validate arguments and working directory. |
| External effects | Block deploys, releases, messages, payments, and infrastructure changes. |
| Lifetime | Bound wall time, resource use, and repair attempts; remove runner state after approved artifacts are collected. |
An allowed command string isn't an execution sandbox. pnpm test runs a manifest-defined script, which can load project code and dependencies; a lint script may even use --fix.[5] Review the scripts, pin dependencies in a trusted setup phase, and keep credentials, filesystem restrictions, and network policy in force for the check and its child processes. A modified test file can execute code too. Prefer host-selected executables and argument arrays over a model-built shell string, but don't confuse shell-injection prevention with safe execution of the program itself.
Enforce integration separately. On GitHub, configure required reviews and status checks on the protected target branch, and review bypass permissions rather than assuming they don't exist. Dismiss stale approvals or require approval of the latest push according to team policy.[6] The runner identity shouldn't be able to change those settings or approve its own integration. A denied push is a boundary working, not a reason to obtain broader credentials.
The boundary must include tools, not only shell commands. Some agents use the Model Context Protocol (MCP) to reach repository services, documentation, or other tools.[7] An MCP server can still expose credentials or side effects, so official MCP security guidance recommends least-privilege scopes and process isolation for sensitive integrations.[8]
Resource text may inform a patch; it can't authorize a new command, widen file ownership, or grant production access. Apply the same rule to README.md, issue comments, fixtures, and generated logs. Explicitly admitted project instructions can guide style and checks within the task, but repository content can't grant itself higher privilege. OWASP identifies prompt injection and excessive agency as separate risks in LLM applications; a coding workflow needs both instruction isolation and limited capabilities.[9]
An agent works only on a feature branch, but its runner has a production token and unrestricted network access. Which boundary is missing?
Answer
The execution boundary is missing. The branch limits source integration, but it doesn't prevent commands from using the token or causing external effects while the patch is being produced.
Close the loop with executable evidence
The agent now has a bounded task and a safe place to work. A plausible patch can still be wrong, so it needs feedback tied to the behavior the task names.
A verifier is a check whose output bears directly on the requested behavior. For billing_event_78291, a useful cheap verifier replays the same event twice, checks the returned identity, and counts stored invoices. Lint is useful, but it can't establish idempotency; a broad green suite may never exercise the reported retry.
Feedback only helps when the retry loop has an end. Without a budget, an agent can churn through speculative edits, inflate the diff, and spend compute while learning little. Here the expected baseline failure doesn't consume a repair attempt. Each patch-and-check cycle does. A timeout or missing dependency is an environment problem, not evidence of the reported bug; preserve that result and request help instead of rewriting code until any command turns green.
Prove the invariant before and after
The foundational rule of coding agents is straightforward: if you haven't seen a test fail on unpatched code, you haven't proven that it tests the defect. A patch accompanied only by a green test suite is suspect because agents easily stumble into four common false-positive traps:
- Tautological assertions: The agent writes
expect(response).toBeDefined()or checks for HTTP status 200. Both webhook deliveries return 200 OK, but the server minted two distinct invoice IDs in the database. The assertion never verifies identity or state. - Mock masking: The agent mocks the database with an in-memory dictionary or stubs the payment client so aggressively that the deduplication query is completely bypassed.
- Selector zero-matches: The agent runs
pnpm test --run billing-webhook, but the test file is namedwebhook.test.ts. Test runners exit with code 0 when zero tests match a pattern unless configured with strict empty-suite flags. The agent sees exit code 0 and reports success without executing a single assertion. - Setup errors masquerading as reproductions: The test fails on the base revision with exit code 1, but inspecting the logs reveals a syntax error or missing import rather than an assertion failure on duplicate invoices. When the patch fixes the import, the test passes, but the bug was never reproduced.
A reliable reproduction contract enforces a strict red-to-green transition:
- Red on base: The test runs against the unpatched base commit and fails specifically on the business invariant (
retry != first), returning non-zero with the failing assertion visible in stdout. - Green on head: The exact same test file runs against the candidate commit and passes without any modifications to test assertions.
- Regression checks: Existing billing suites remain green on the candidate commit to guarantee adjacent behaviors weren't damaged.
Hold the input event constant so the behavior change is visible. The executable example runs one verifier against two in-memory implementations, then prints invoice IDs instead of a generic “tests pass.” A second event checks the neighboring contract: a genuinely new event still creates a new invoice. Checking stored count also catches a handler that returns the old ID while secretly creating another invoice. This is a sequential demonstration, not a checkout of two real service revisions.
1from typing import Protocol
2
3EVENT_ID = "billing_event_78291"
4NEW_EVENT_ID = "billing_event_99002"
5
6class InvoiceStore(Protocol):
7 def create_for(self, event_id: str) -> str: ...
8 def invoice_count(self) -> int: ...
9
10class BuggyInvoiceStore:
11 def __init__(self) -> None:
12 self.invoices: list[str] = []
13
14 def create_for(self, event_id: str) -> str:
15 invoice_id = f"inv_{len(self.invoices) + 1}" # ignores event_id
16 self.invoices.append(invoice_id)
17 return invoice_id
18
19 def invoice_count(self) -> int:
20 return len(self.invoices)
21
22class FixedInvoiceStore:
23 def __init__(self) -> None:
24 self.by_event: dict[str, str] = {}
25
26 def create_for(self, event_id: str) -> str:
27 if event_id not in self.by_event:
28 self.by_event[event_id] = f"inv_{len(self.by_event) + 1}"
29 return self.by_event[event_id]
30
31 def invoice_count(self) -> int:
32 return len(self.by_event)
33
34def replay(store: InvoiceStore) -> tuple[str, str, str]:
35 first = store.create_for(EVENT_ID)
36 retry = store.create_for(EVENT_ID)
37 other = store.create_for(NEW_EVENT_ID)
38 return first, retry, other
39
40def run_receipt(label: str, store: InvoiceStore) -> bool:
41 first, retry, other = replay(store)
42 if retry != first:
43 print(f"{label}: FAIL (retry {retry} != {first})")
44 return False
45 if other == first:
46 print(f"{label}: FAIL (new event reused {first})")
47 return False
48 if store.invoice_count() != 2:
49 print(f"{label}: FAIL (expected 2 stored invoices, got {store.invoice_count()})")
50 return False
51 print(f"{label}: PASS (retry kept {first}; new event {other}; stored 2)")
52 return True
53
54assert not run_receipt("before patch", BuggyInvoiceStore())
55assert run_receipt("after patch", FixedInvoiceStore())
56
57first, retry, other = replay(FixedInvoiceStore())
58assert first == retry == "inv_1"
59assert other == "inv_2"1before patch: FAIL (retry inv_2 != inv_1)
2after patch: PASS (retry kept inv_1; new event inv_2; stored 2)The dictionary isn't a production fix. It disappears on restart, isn't shared by multiple workers, and its check-then-insert isn't a storage transaction. Two workers can both observe an absent event and each create an invoice. A real implementation usually needs a durable uniqueness rule and an atomic operation over the deduplication key and invoice creation. If another service performs the side effect, a local database transaction alone can't make that remote call exactly once.
Choose the key from the actual provider contract, including account or tenant scope when needed. Event-ID deduplication covers delivery retries of the same event; distinct event IDs referring to the same business operation may require an additional business key. For the real patch, test concurrency, restart, and the crash between invoice creation and recording completion where those paths exist. None of those properties follows from the sequential example.
Before-and-after evidence must also identify what ran. Keep the new reproduction unchanged while running it against the base implementation and the candidate. Confirm the baseline fails on the duplicate-invoice assertion, not import, setup, or unrelated failures. Record collected test IDs, result, command arguments, environment, source revision or tree digest, and the verifier version. A command that exits zero after collecting no relevant tests proves nothing about retries.
The replay check is the strongest receipt for this bug, but other changes need different evidence:
| Change | Strong minimum receipt |
|---|---|
| Bug fix | Same reproduction fails before and passes after. |
| Feature | Acceptance path plus meaningful error or boundary path. |
| Refactor | Behavior-focused suite stays green and diff explains preserved contract. |
| UI | Focused test plus rendered screenshot or browser smoke when visual behavior matters. |
| API | Contract test with request, response, and error behavior. |
| Security | Threat-specific test plus focused human review. |
Read coding benchmarks as context, not local proof
The replay receipt answers a local question: did this patch fix this behavior? Coding benchmarks answer a broader question, and they shouldn't replace the local check.
SWE-bench introduced repository-level tasks drawn from real GitHub issues and pull requests.[10] SWE-bench Pro later added longer-horizon tasks across public, held-out, and proprietary repositories.[11] These benchmarks can compare a model plus its agent scaffold under a stated protocol. They don't prove that one patch fixed your webhook.
Benchmark interpretation also changes as suites and agent setups change. In February 2026, OpenAI reported test and task-description defects in an audit of 138 hard SWE-bench Verified problems, plus evidence that frontier models could reproduce gold patches or problem specifics, a sign of training-set contamination.[12] The hard-problem sample wasn't a random audit of the full 500-task suite. OpenAI stopped reporting Verified scores and recommended SWE-bench Pro in that report. That's a provider's evaluation decision, not proof that another benchmark is contamination-free. Record the split, benchmark version, scaffold, tools, and attempt budget, then demand repository-specific evidence for each merge.
Why is a high coding-benchmark score weaker than a failing-then-passing webhook replay test for this task?
Answer
The benchmark measures performance over another task set and agent setup. The replay test exercises the exact invariant, repository state, and failure the patch claims to fix.
Review scope before implementation detail
At this point, the agent can return a branch and receipts. Review still starts from the task contract and artifacts, not from the agent's summary:
- Compare the complete candidate diff with owned paths, including deletions, both sides of renames, file-mode changes, and untracked files in a local handoff.
- Check that before-and-after evidence exercises acceptance criteria.
- Inspect sensitive surfaces such as auth, billing, dependencies, shell execution, workflows, and deployment.
- Read implementation for correctness, failure handling, observability, and maintainability.
- Confirm CI and any rendered or integration behavior that local checks couldn't prove, against the exact candidate being reviewed.
Scope comes first because a correct four-line fix can hide inside an unrelated eighteen-file patch. The helper below turns ownership and receipt metadata into early findings. Its inputs come from the host: the complete changed-path set and results from the test runner. Try the fixture's older-candidate receipt: a passing check on that revision can't validate the current head. This checker screens metadata; it doesn't authenticate logs, inspect code, or approve a merge.
1from dataclasses import dataclass, replace
2from pathlib import PurePosixPath
3
4OWNED = {
5 "web/src/billing/webhook.ts",
6 "web/src/billing/webhook.test.ts",
7}
8REQUIRED_TESTS = frozenset({"same_event_replay", "new_event_creates_invoice"})
9# Fixture identities, not hashes collected from a real repository.
10BASE = "a" * 40
11HEAD = "b" * 40
12VERIFIER = "d" * 64
13
14@dataclass(frozen=True)
15class Receipt:
16 revision: str
17 verifier_digest: str
18 collected: frozenset[str]
19 failed: frozenset[str]
20 exit_code: int | None # None means interrupted or no terminal result.
21
22def canonical_path(path: str) -> bool:
23 parsed = PurePosixPath(path)
24 return (bool(path) and not parsed.is_absolute() and ".." not in parsed.parts
25 and "\\" not in path and "\x00" not in path and str(parsed) == path)
26
27def needs_focused_review(path: str) -> bool:
28 parts = PurePosixPath(path).parts
29 return (bool({"auth", "billing", "deploy"}.intersection(parts))
30 or parts[:2] == (".github", "workflows")
31 or PurePosixPath(path).name in {"pnpm-lock.yaml", "package.json"})
32
33def screen_pull_request(
34 changed_files: set[str],
35 *,
36 before: Receipt,
37 after: Receipt,
38 base: str,
39 head: str,
40 verifier_digest: str,
41) -> list[str]:
42 findings: list[str] = []
43
44 if not changed_files:
45 findings.append("no candidate changes supplied")
46 if any(not canonical_path(path) for path in changed_files):
47 findings.append("invalid repository-relative path")
48 outside_scope = sorted(changed_files - OWNED)
49 if outside_scope:
50 findings.append(f"scope drift: {', '.join(outside_scope)}")
51
52 sensitive = sorted(path for path in changed_files if needs_focused_review(path))
53 if sensitive:
54 findings.append(f"focused review: {', '.join(sensitive)}")
55
56 if before.revision != base or after.revision != head:
57 findings.append("receipt revision differs from reviewed base or head")
58 if any(receipt.verifier_digest != verifier_digest for receipt in (before, after)):
59 findings.append("receipt uses a different verifier")
60 if any(receipt.collected != REQUIRED_TESTS for receipt in (before, after)):
61 findings.append("expected test collection differs")
62 if before.exit_code != 1 or before.failed != frozenset({"same_event_replay"}):
63 findings.append("baseline did not show the expected behavior failure")
64 if after.exit_code != 0 or after.failed:
65 findings.append("candidate did not finish with both checks passing")
66
67 return findings
68
69before = Receipt(BASE, VERIFIER, REQUIRED_TESTS, frozenset({"same_event_replay"}), 1)
70after = Receipt(HEAD, VERIFIER, REQUIRED_TESTS, frozenset(), 0)
71changed = {
72 "web/src/billing/webhook.ts",
73 "web/src/checkout/PayButton.tsx",
74 "pnpm-lock.yaml",
75}
76stale_after = replace(after, revision="c" * 40)
77for finding in screen_pull_request(changed, before=before, after=stale_after,
78 base=BASE, head=HEAD, verifier_digest=VERIFIER):
79 print(f"REVIEW: {finding}")
80
81clean_findings = screen_pull_request(OWNED, before=before, after=after,
82 base=BASE, head=HEAD, verifier_digest=VERIFIER)
83print("scoped current packet:", clean_findings)
84print("merge authorized:", False)1REVIEW: scope drift: pnpm-lock.yaml, web/src/checkout/PayButton.tsx
2REVIEW: focused review: pnpm-lock.yaml, web/src/billing/webhook.ts
3REVIEW: receipt revision differs from reviewed base or head
4scoped current packet: ['focused review: web/src/billing/webhook.test.ts, web/src/billing/webhook.ts']
5merge authorized: FalseHere, scope and evidence freshness need different responses. A button debounce is out of scope and doesn't prove server idempotency. The lockfile changes the dependency surface without justification. The old-head receipt needs a rerun on the reviewed candidate, not a rewritten revision field. Even the scoped, current packet still flags billing for focused review. An empty findings list from an automated screen would mean only that its checks found nothing, not that integration is authorized.
A reliable review packet binds four cryptographic roots:
- Base revision (
base_commit): The Git commit SHA of the unpatched base branch where the reproduction test failed. - Candidate revision (
head_commit): The Git commit SHA of the proposed patch. If the agent later commits a tiny typo fix or formatting change to the branch, all earlier test receipts are instantly void. - Tree digest (
tree_digest): The SHA of the Git tree object (git rev-parse HEAD^{tree}). A commit hash pins commit metadata, but uncommitted modifications, untracked files, or gitignore anomalies in a dirty worktree can alter test execution. The tree hash guarantees that the exact filesystem state evaluated by the runner matches what reviewers inspect. - Verifier digest (
verifier_digest): The SHA-256 checksum of the reproduction test file itself. This prevents an agent from passing the test by surreptitiously weakening assertion tolerances or deleting failing assertions during patch iteration.
In a real deployment, a trusted runner records these digests directly from immutable container snapshots, test runner event streams, terminal process exit codes, and retained output artifacts. An exit_code=1 is meaningful only with the expected failing test assertion visible in stdout; other runners may use different exit conventions. The model mustn't generate or self-sign these fields. Path screening provides early triage, but branch protection rules enforce the final gate.
Before integration, re-read the current head, required checks, reviews, and target-branch state. A later edit invalidates earlier patch evidence even if the branch name is unchanged. Test the merge result or use the repository's merge-queue policy when integration with a moving base matters. Keep the final authorization check and write bound to the reviewed revision so a new push can't slip between review and merge. Preserve the human merge and deploy boundary: automated agents can prepare review packets, but human code owners retain exclusive authority to merge into main and trigger production releases.
Diagnose common workflow failures
| Symptom | Likely cause | Correct response |
|---|---|---|
| Green suite, original bug still reproduces | Existing tests never encoded reported invariant. | Add smallest reproduction and rerun before and after. |
| Diff spreads into adjacent modules | Ownership was vague or diagnosis changed. | Stop, explain new dependency, and rescope before editing. |
| Agent keeps making speculative patches | Verifier is weak or retry budget is missing. | Improve check, preserve failure receipts, and escalate after bounded attempts. |
| Green receipt belongs to yesterday's head | Branch name was treated as an immutable version. | Rerun the relevant checks on the final snapshot and refresh review state. |
| Check exits zero but replay never ran | Test selector, skip logic, or runner configuration changed. | Inspect collected test IDs and require the behavior assertion to execute. |
| Branch is clean but external state changed | Branch was mistaken for execution isolation. | Revoke capability, inspect audit logs, and rebuild runner policy. |
| Review log contains a token | Evidence capture lacked redaction and secret scanning. | Revoke exposed token, restrict artifact access, preserve required incident evidence, and fix capture before rerun. |
| Agent follows a command from issue text | Untrusted repository content gained authority. | Block effect, restore task contract as authority, and inspect for prompt injection. |
| Two workers edit one shared file | Ownership split ignored shared state. | Choose one owner or sequence work before combining patches. |
An agent PR includes a webhook fix plus PayButton.tsx and the lockfile. What should the reviewer do first?
Answer
Stop deep review and flag scope drift. Ask for a billing-only patch with before-and-after evidence. The debounce isn't proof of server idempotency, and the lockfile needs its own justification, owner, and focused review.
Use roles as handoffs, not as authority
Once a patch passes the first scope check, roles make the next handoff explicit. One agent can work in several passes, or several agents can own separate artifacts. Either way, each role should name the output it owes the next role:
| Role | Input | Required output | Must not do |
|---|---|---|---|
| Explorer | Issue plus read scope | Relevant files, observed behavior, open questions | Edit code or widen authority. |
| Worker | Approved plan and owned paths | Small patch plus local receipts | Change architecture or shared files silently. |
| Verifier | Diff and acceptance criteria | Commands, outputs, uncovered risks | Treat command success as product proof. |
| Reviewer | Contract, diff, and receipts | Findings ordered by behavior and risk | Rubber-stamp author summary. |
A worker should run checks while coding. A later verifier pass still adds value because it begins from acceptance criteria and the final diff, not from the worker's edit path. Fresh context, different instructions, CI, or a human reviewer can make that pass more independent; calling the same model twice doesn't create independence by itself.
Self-review can catch omissions before handoff, but it isn't external verification. Reflexion is a more specific research pattern: evaluated trials produce verbal feedback that enters episodic memory for later attempts.[13] A single critique of the draft in front of the same agent is better described as self-review.
Parallel agents also need exclusive file ownership. If two tasks discover a shared file, pause integration and either assign one owner or sequence the changes. Separate worktrees isolate source edits, but tests can still collide through a shared database, port, cache, or fixture directory. Isolate those resources or serialize their use, then verify the combined patch. A clean textual merge doesn't prove that two semantic changes compose correctly.
Start coding agents where feedback is local and judgment is bounded: narrow bug fixes, focused tests, one-module migrations, documentation synced from source, and contained refactors with strong coverage. Keep humans close to auth, billing, dependency policy, migrations, security boundaries, and architectural choices.
Why doesn't a separate verifier label guarantee independent verification?
Answer
The verifier may share the worker's assumptions, context, model, or tools. Independence comes from a distinct evidence path such as acceptance criteria, fresh context, CI, another implementation-aware reviewer, or direct product behavior.
Assemble the merge packet
Return to billing_event_78291. A reviewable completion isn't a claim that the agent feels confident. It's a compact chain of artifacts:
| Artifact | Expected value |
|---|---|
| Task contract | Billing handler and test owned; checkout UI, auth, dependencies, deploy, and production calls excluded. |
| Before receipt | Replaying billing_event_78291 returns two invoice IDs; focused test fails. |
| Patch | Durable, atomic event-to-invoice operation appropriate to the real storage and side-effect boundary. |
| After receipt | Same replay returns original invoice without extra stored invoices; focused test passes on the reviewed candidate. |
| Regression receipt | New event still creates one new invoice; relevant suite and lint pass. |
| Diff receipt | Changed paths match ownership; no secret, dependency, workflow, or deploy drift. |
| Risk note | Concurrency and persistence behavior reviewed against real storage transaction. |
| Decision | Human reviewer approves, requests changes, or rejects. |
The packet doesn't need a transcript of every agent thought. It needs enough evidence to reproduce behavior, plus enough provenance to identify the code, command, environment, verifier, and final revision behind each result. Redact secrets and cap noisy output before attaching logs. “Not run,” “timed out,” “failed,” and “passed on an older revision” are different states; retain them rather than collapsing all four into “checked.”
If the team later auto-merges a narrow class of low-risk generated patches, keep policy outside the model: explicit repository class, owned paths, required checks, rollback plan, audit trail, and a fast way to disable automation. “Agent confidence” isn't a merge policy.
Use the packet to adjudicate this final case:
Agent says: “Fixed duplicate invoices. All tests pass.” Diff changes the billing handler,
PayButton.tsx, auth middleware, and the lockfile. A new test calls invoice creation once.
What should the review response request, and why?
Answer
Request a smaller billing-only patch. Require a test that replays same event and fails on old code, then passes on fix. Ask agent to explain any newly discovered file before rescoping. Reject frontend debounce as proof of server idempotency, and move auth or dependency work into separately owned reviews.