Skip to content

docs: findings 0005 correction and ADR-0022 addendum, coverage re-measured with search working - #90

Merged
svozza merged 3 commits into
mainfrom
prompt-coverage
Nov 11, 2026
Merged

svozza merged 3 commits into
mainfrom
prompt-coverage

Conversation

@svozza

@svozza svozza commented Nov 11, 2026

Copy link
Copy Markdown
Owner

What this records

Every Grep call failed in the 2026-11-10 large-PR study: 1,456 of 1,456, in all 325 sessions, in the reviewer arms and the plain Claude Code arm alike. The experiment's source-tool shim looked up ripgrep with shutil.which('rg') and no rg binary was on the session PATH. Production uses Claude Code's built-in Grep and was not affected; the October studies were not affected.

The coverage arms were re-run the same day with Grep working (202 sessions, all delivered) and re-judged.

Changes

  • docs/findings/0005-large-pr-delivery-and-coverage.md: a Correction section with the cause, the scope, and the re-measured coverage table beside the original figures.
  • docs/adr/0022-addendum-coverage-re-measured-with-search-working.md: ADR-0022 is left as written; the addendum gives the re-measured figures and what does and does not stand.

What stands and what does not

  • Stands: coverage, not precision, is the large-PR problem (6.1 of 17.1 families per review; 98.6-100% of findings supported). A second pass adds 2.6 families against 1.5 for an independent rerun (difference 1.1, standard error 0.3, 7 of 8 PRs). The skip after an empty first pass holds.
  • Does not: the second pass is not separated from two half-file reviewers (0.75 apart, standard error 1.2), and plain Claude Code is not separated from the reviewer (+0.8, standard error 0.65).
  • The delivery results do not depend on Grep.

Docs only; no code or prompt change.

🤖 Generated with Claude Code

@svozza
svozza deployed to ai-pr-review-runtime November 11, 2026 01:36 — with GitHub Actions Active
@svozza
svozza merged commit feb4372 into main Nov 11, 2026
8 checks passed
@svozza
svozza deleted the prompt-coverage branch November 11, 2026 13:47

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — c9ccbfe5 Deployed Nov 11, 2026 by svozza via evals #207
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant