Record the 2026-10-11 hard-corpus model and prompt evaluation

[mattgodbolt]

Summary

Adds one bullet to the "Anthropic API gotchas" section of CLAUDE.md recording the follow-up evaluation on the 31 reviewable hard_* cases, so the results and the dead ends are in permanent context.

• Sonnet 5.5 with the bullet-cap prompt stays: 85% pooled before the cap, 94% for the exact merged prompt; configs differing only in input compaction scored up to 9 points apart, so gaps under ~8 points are noise.
• Opus 5.5 low is the only plausible upgrade: 93% pooled over 12 runs, 2x cost, ~3s slower, one 33s outlier in 512 calls. Not adopted.
• Fable 5.1 on the merged prompt matches Sonnet's accuracy (45/47, 29/31 on hard cases) but takes 17s on average and over 30s on 5 of 53 calls.
• Tried with no effect: dropping the "compare with other optimisation levels" focus bullet, a mechanical control-flow digest, input compaction.
• Fast mode returns a 429 for this org.

Documentation only; pre-commit and the 224 tests pass.

🤖 Generated with Claude Code