[svozza]

What changes

One sentence of SECOND_PASS_NOTE in src/aceiro/review_passes.py, the harness-authored instruction given to the opt-in second review pass (ADR-0022).

Before: "look for confirmed defects that pass missed, starting with changed files and behaviours it did not touch."

After: "look for confirmed defects that pass missed. Start with the kinds of defect it did not report (error paths, edge cases, compatibility for existing callers, concurrency, tests that cannot fail) and with changed files and behaviours it did not touch."

The main review prompt is untouched. The second pass stays off by default.

Evidence

Opus 5.5 at high effort on prompt v6, source-only tools. Both arms of a comparison are shown the same first review. Blind judge (Fable 5) reading source. The figure is new supported defect families the second pass adds beyond the first review; paired by PR, 90% bootstrap interval over PRs.

Set	Runs per PR per arm	Current note	New note	Change	Interval	PRs up / down
18 PRs, 250-1,500 lines, batch 1	2	1.25	1.56	+0.31	-0.08 to +0.67	9 / 3
18 PRs, 250-1,500 lines, batch 2 (a different first review)	4	1.50	1.76	+0.26	-0.01 to +0.57	8 / 4
8 held-out PRs, 2,300-3,900 lines	6	2.69	3.46	+0.77	+0.48 to +1.10	7 / 0

On the large PRs:

	Current note	New note
First review + second pass, families	9.38	10.15
Findings supported	148 of 148	181 of 183
Cost per second pass	$1.93	$1.98
Elapsed	152s	168s
Delivered	48 of 48	48 of 48

• Mid-size PRs: 192 of 192 findings from the new note supported; cost $0.60 per pass in both arms; 108 of 108 delivered.
• The two unsupported findings came from one session on one PR: the same mistaken claim at two sites, both low severity (a removed fallback called a regression where the old path never worked).
• A second variant, appending a harness-computed list of changed files the first pass reported nothing on, measured +0.17 (0.00 to +0.33) on the mid-size set and is not included.

Checks

• 2,426 unit tests pass; type check clean.
• The repo's model eval scenarios were not run: they do not exercise the second pass, and the review prompt is unchanged.

Not measured

• Severity and stated impact of the extra findings; the judge audits only whether the defect is real.
• The mid-size gain on its own is marginal; the case for the change rests on the large-PR result.

🤖 Generated with Claude Code