[Daniel Peng (original4422)]

Summary

Claude replay now selects an isolated mode without excluding supported subscription/OAuth setups:

 API key present
   use --bare
 No API key
-  refuse replay
+  probe and cache successful claude --help capability
+  use --safe-mode when advertised; otherwise refuse

Both plain and tool-aware replay stay fail-closed. Per-night claude_isolation_modes diagnostics and report notes record the selected mode, including split optimizer/target roles. Admin-managed policy settings still apply in safe mode. No dependency or credential-discovery change.

Closes #335

Evidence

• New original-source regressions initially failed for supported safe mode and mode reporting. The final full suite passes: 2,000 passed, 24 skipped, 4 xfailed; 381 subtests passed, with eight existing Qwen warnings. The focused 169-test run is included in that total.
• Real Claude Code 2.1.283, isolated synthetic HOME/config and local HTTP provider fixture: normal mode sent the synthetic user-level CLAUDE.md marker and ran its UserPromptSubmit hook; --safe-mode and --bare excluded both. All three calls received fixture responses and exited successfully. The modified SkillOpt backend also completed an actual no-API-key safe-mode call against the fixture using synthetic bearer authentication.
• This measures actual CLI request construction and hook execution. No real personal Claude configuration, credentials, user OAuth session, live provider or model was accessed; it is not a live OAuth-login test.
• python -m mkdocs build --strict and git diff --check pass. Ruff reports eight violations already present in the original head, independently matched by source line/code/message; no new violation. Docs retain two existing unnavlisted-page notices and the Material lifecycle warning.
• Hosted CI is separate; only the existing license/cla success was observed before this update.

Merge Danger

Door: two-way. Blast Radius: replay-authentication.

OAuth requires a CLI advertising --safe-mode. Missing, unsuccessful or timed-out help probes never permit an unisolated replay; the capability result is cached per backend instance.

AI disclosure: developed and validated with OpenAI Codex as the contributor's authorized proxy.

[Daniel Peng (original4422)]

Hi Yifan Yang (@Yif-Yang), could you review this when convenient? This refuses Claude OAuth replays that cannot be isolated and keeps API-key replays in --bare mode. The Codex-assisted change has historical offline test and strict docs-build evidence in the PR; no live Claude authentication or user configuration was exercised. Thanks!

[Yifan Yang (Yif-Yang)]

Thanks for picking this up, and for keeping API-key replays in --bare. The underlying problem in #335 is real, and I agree that SkillOpt should not report a score it cannot control.

Before we make OAuth a hard refusal, there is one option that may keep subscription users working. Current Claude Code has a --safe-mode flag. According to claude --help, it disables CLAUDE.md, skills, plugins, hooks, MCP servers and other customisations, while auth and model selection "work normally". I checked it locally on Claude Code 2.1.312 with OAuth auth:

• With a project CLAUDE.md instructing a marker token, a plain -p call appended the marker, while the same call with --safe-mode did not.
• The same held for a UserPromptSubmit hook.

Could you rework the change along these lines?

• Use --bare when ANTHROPIC_API_KEY is set, as today.
• Otherwise use --safe-mode when the installed CLI supports it (detect it from claude --help or the version, and cache the result).
• Refuse with your current error only when neither is available.
• Record which isolation mode was used in the night's diagnostics/report.
• Document that admin-managed (policy) settings still apply in safe mode.

Please also add a test that the OAuth path passes --safe-mode and never reaches an unisolated call. My local check covered a project-level CLAUDE.md and a project hook only. Before relying on safe mode, please also confirm that it excludes a user-level ~/.claude/CLAUDE.md, which is the case #335 reports.

[Daniel Peng (original4422)]

Implemented in 82a24a1:

• API-key calls retain --bare; without a key, both replay paths use --safe-mode only after a successful, cached --help capability probe. Unsupported/failed/timed-out probes refuse replay.
• Diagnostics and report notes record per-night isolation modes, including optimizer/target roles. The docs state that admin-managed policy settings still apply.

I checked the real installed Claude Code 2.1.283 using an entirely synthetic HOME/config and a loopback HTTP provider fixture. The normal-mode positive control sent the user-level ~/.claude/CLAUDE.md marker and executed its UserPromptSubmit hook. Safe mode excluded both, received the fixture response and exited successfully. The modified SkillOpt backend also completed a no-API-key safe-mode call using synthetic local bearer authentication. This verifies CLI request construction/hook behavior; no real personal configuration, credentials, user OAuth login, live provider or model was used.

Current full validation: 2,000 passed, 24 skipped, 4 xfailed; 381 subtests passed, plus strict MkDocs and diff checks. Eight Ruff findings match the original head's source exactly; no new lint violation. The PR body records the warnings and validation limits. Developed and tested with Codex.