[svozza]

What changes

One sentence of prompts/ai-pr-review.md, and the version marker (5 -> 6).

Before: "If you suspect a problem but cannot confirm it with the tools available, put one short note in residual_risk instead of a finding."

After: "If you suspect a problem, do not drop it: read the code that settles it, and report it as a finding once it is confirmed. Only what the tools available cannot settle goes in residual_risk, as one short note."

The "confirmed defects only" mandate is unchanged.

Evidence

18 public PRs of 250-1,500 changed lines, Opus 5.5 at high effort, 3 runs per PR per arm, each batch against its own concurrent control. Blind judge (Fable 5) reading source. Paired by PR, 90% bootstrap interval over PRs.

Batch	Control	With the edit	Change	Interval	PRs up / down
p1 (control 6 runs per PR)	3.23	3.56	+0.32	+0.18 to +0.48	11 / 1
p2	3.19	3.67	+0.48	+0.11 to +0.94	10 / 4

Figures are supported defect families per review.

• Findings supported: 406 of 410 (4 judged generic, 0 unsupported). Control: 542 of 543.
• Cost per review: $0.56 -> $0.59. Delivered: 108 of 108.
• residual_risk length: about 610 -> 380 characters.
• Submissions per review: 1.41 -> 1.49.

Two other single edits were tested and are not included: replacing the "handful of tool calls" sentence (+0.19, interval includes zero) and a pre-submit sweep of unexamined files (+0.16, interval includes zero). Stacked on this edit they added -0.06 and -0.02.

Held-out confirmation on large PRs

8 public PRs of 2,300-3,900 changed lines, not used while choosing the edit. 6 runs per PR per arm, concurrent control, same judge.

	Control	With the edit
Supported families per review	6.06	6.71
Findings supported	316 of 322	350 of 356
Cost per review	$1.88	$1.89
Submissions per review	1.67	1.46
Delivered	48 of 48	48 of 48
residual_risk characters	770	409

Change: +0.65 families (90% interval +0.17 to +1.12); up on 6 PRs, level on 1, down on 1.

Repo evals (ADR-0008)

• Review scenarios, one run: 41 of 42. Plan scenarios: 6 of 6.
• The failure was grouped_cross_file_defect, and it is caused by this prompt: 0 of 8 runs failed on v5, 3 of 9 on v6. The v6 reviewer also reports the removed constant as a breaking change for outside importers, in a group of its own. A trial variant of the prompt with an extra grouping example failed 3 of 12 the same way, once with a fourth finding (a test gap), and was dropped. The grader required any finding on that line to share the callers' group and capped the scenario at 3 findings.
• Second commit: the optional finding may join the callers' group or keep its own, and the ceiling is 4. Still asserted: both callers share one group, nothing unrelated joins it, an optional finding cannot replace a caller or bridge split groups. 12 of 12 runs pass on v6 with the change.
• The full 42 were not rerun after the grader change; it touches only that scenario.
• 2,426 unit tests pass; type check clean.

Not measured

• Severity and stated impact of the extra findings; the judge audits only whether the defect is real.

🤖 Generated with Claude Code