[lakhansamani]

Supersedes #821, #822 and #823 — one PR, no merge-order choreography. Four commits, each reviewable on its own.

Started from one question: had we ever run the storage suite against real Apache Cassandra? No. cassandradb was missing from the suite's backend list, so the provider that serves both cassandradb and scylladb was only ever exercised through Scylla. Pointing it at cassandra:4.1 failed on the first run, at subtest 9, with a panic.

---

1. chore(proto): regenerate stubs after a protovalidate dep bump

buf dep update resolves a newer buf.build/bufbuild/protovalidate, which moved its documentation URLs:

- * [see our documentation](https://buf.build/docs/protovalidate/schemas/custom-rules/)
+ * [see our documentation](https://protovalidate.com/schemas/custom-rules/)

Seven doc comments in gen/ts/buf/validate/validate_pb.ts plus the proto/buf.lock digest. No API, field or wire change.

Latent on main: buf.yml triggers on proto/**, gen/** or Makefile, so the staleness gate stayed quiet until a PR touched one of those — commit 2 adds a test target to the Makefile — and then failed pointing at the wrong change. Included here so this PR's CI is green on its own merits.

2. fix(storage): make the cassandradb provider work on apache cassandra

Three queries use CQL LIKE. Apache Cassandra accepts LIKE only against a SASI index and rejects it against the plain secondary index this provider creates:

InvalidRequest: LIKE restriction is only supported on properly indexed columns.
event_name LIKE 'test_event%' is not valid.

ScyllaDB accepts the same statements, which is exactly why this survived.

Method	Effect on Apache Cassandra
GetWebhookByEventName	errors — webhook dispatch by event name never fired
DeleteAllSessionTokensByUserID	errors — session revocation fails
DeleteSessionTokensByNamespace	errors — same

All three already carried ALLOW FILTERING, i.e. a full table scan, so the index was never doing any work. Moving the comparison into Go changes nothing but where it runs — no new scans, no behaviour change on Scylla.

A second bug, live on both engines. DeleteAllSessionTokensByUserID matched "%" + userId + "%" — unanchored. A delete for 42 also took out tokens stored for 142 or provider:427. The SQL provider was anchored long ago, with a comment explaining precisely this; the Cassandra provider kept the defect because the assertion covering it sat inside testSQLCRUDCorrectnessFixes, which only runs for SQL backends. It is now Anchored Session Token Delete, running on every backend. Mutation-verified: restoring the unanchored match fails it on ScyllaDB.

Also: Webhook_Operations indexed fetchedByEventName[0] after only an assert.NotNil. An empty-but-non-nil slice panics there, and a panic in one backend's subtest aborts the run for every backend queued behind it — which is how the first Cassandra run ended, hiding everything after it. Now require.NotEmpty.

Coverage: cassandradb added to allDBTypes and getTestDBConfig on port 9142 — deliberately not 9042, so TEST_DBS=scylladb,cassandradb cannot bind both types to one engine and silently test Scylla twice. Plus make test-cassandradb, and cassandradb in test-all-db, test-docker-up, test-cleanup, wait-for-test-dbs.sh and the AGENTS.md table (7 DBs → 8).

3. test(storage): stop the user suite depending on an almost-empty table

Chasing #813's remaining failure, I looped the scylladb storage suite and hit 15 consecutive full-suite failures — on main and on pre-#818. Neither was a storage bug; both were the test's own fault, and both produce exactly the "flaky ScyllaDB storage tests" symptom #813 reports.

1. ListUsers first-page assumption. User_Operations asserted the just-created user appears in ListUsers(Limit: 10, Offset: 0) unfiltered. That only holds while the table is nearly empty. At 463 accumulated rows it failed every run with created user should appear in ListUsers — indistinguishable from a write that never landed. Now filtered by the user's own email, which keeps both assertions and makes them independent of how much data is already there.
2. Cascading pollution. That failure aborts before the subtest's cleanup, leaving a fixed +1234567891 literal in the table. Every later run then failed earlier, with user with given phone number already exists — one failure mutating into a different, misleading one. Both phone numbers are unique per run now, like the emails already were.

Invisible in CI, which gets a fresh container per run. Deterministic locally.

4. fix(storage): put every cassandra table in one timestamp domain

Closes out the allow-list #818 deliberately left behind.

A single-use session token could be claimed twice. DeleteSessionTokenByUserIDAndKey consumes a token with DELETE … IF EXISTS and treats the applied flag as the exactly-once arbiter behind refresh-token rotation — the comment in the code says so explicitly. That LWT is stamped from the Paxos ballot, i.e. the server clock. AddSessionToken's plain insert carried gocql's client timestamp. When the client clock runs ahead, the tombstone lands below the insert: the row survives, but Paxos saw a live row and returned applied=true. The next claim of the same token also returns applied=true.

Measured on scylladb/scylla, client clock emulated 10 ms ahead with WithTimestamp, 20 insert-then-two-LWT-deletes cycles:

insert	double-claimed	row survived both deletes
plain, client clock +10ms	20/20	20/20
plain, no skew	0/20	0/20
DefaultTimestamp(false)	0/20	0/20

oauth_states has the identical shape for single-use authorization codes. authenticators mixes an LWT insert and two conditional updates with a plain update and a plain delete.

The fix is DefaultTimestamp(false) on the plain writes to those three tables: gocql sends no timestamp and the coordinator assigns one — the same clock the LWT already uses.

• Per query, never on the cluster. gocql round-robins coordinators, so cl.DefaultTimestamp = false would push the 20-odd tables that never touch an LWT off a single monotonic client clock onto several skewed node clocks. Strictly worse than today.
• No Paxos round-trip. That is what makes this routine rather than the performance decision flagged on fix(storage): stop mixing LWT and plain writes on cassandra partitions #818: converging the other way means lifting AddSessionToken to IF NOT EXISTS, i.e. a Paxos round on every login.

Re-measured on both engines after review, LWT insert then delete on one partition, 50 pairs per cell:

delete	cassandra 4.1.12	scylladb
plain, ambient clocks	0/50	0/50 *
plain, client stamp forced −10ms	50/50	48/50
… IF EXISTS (#818's fix), forced −10ms	0/50	0/50
server-stamped plain, forced −10ms	0/50	0/50

* an earlier ScyllaDB container, with a different instantaneous client-vs-coordinator clock relationship, dropped 48/50 on that same ambient row. Whether it fires without forcing depends entirely on which way the two clocks happen to sit — which is what made #813 look flaky.

This corrects something I said earlier in this PR. I had written that Apache Cassandra escapes the drop because its ballot timestamp is truncated to whole milliseconds. It does not: with the client stamp forced back 10 ms, Cassandra drops 50/50, and an ambient run with the engine clock 15 ms ahead still dropped 0/50, which the truncation story cannot explain either. The exposure is not engine-specific and I no longer offer a mechanism for the ambient behaviour — only the measurement.

The guard is reframed from "LWT vs plain" to the invariant that actually matters — server-stamped vs client-stamped — and recognises DefaultTimestamp(false) as server-domain. knownMixedTables is empty.

It also had a blind spot worth naming: the statement regex matched "…" but not raw strings, so it never saw AddSessionToken's insert — one of the two writes it was written to watch. Both quote forms now, and the five tables it covers are asserted by name so a regex drift fails loudly instead of silently losing coverage.

SessionToken insert            guard caught it: YES
OAuthState insert              guard caught it: YES
SessionToken plain delete      guard caught it: YES
Authenticators plain update    guard caught it: YES
Authenticators plain delete    guard caught it: YES

---

Verification, all on this branch

Check	Result
Storage suite, cassandra:4.1	26/26 subtests (was: panic at subtest 9)
Storage suite, scylladb/scylla	26/26 subtests
Full storage suite × 12, populated keyspace	0/12 failed (was 15/15)
User_SCIM_Fields / Org_Domain_Operations / Session_Token_Operations × 100	0 failed
TEST_DBS=sqlite go test -p 1 ./...	pass
make proto-check, make proto-check-clients, make proto-lint	pass
go build ./..., go vet ./...	clean

gofmt -l flags internal/http_handlers/oauth_sso_verify_test.go, which is pre-existing on main and not touched by this PR.

What this does not fix

• Flaky ScyllaDB storage tests: Org_Domain_Operations and User_SCIM_Fields fail on read-after-write #813's User_SCIM_Fields never reproduced — 0 failures in 100 targeted runs, before and after fix(storage): stop mixing LWT and plain writes on cassandra partitions #818. The 15/15 "flaky" failures were commit 3's test pollution, which is a different thing. If it resurfaces after this lands it deserves a fresh issue.
• An unexplained asymmetry. A plain UPDATE after an LWT insert loses 0/50, in the identical loop shape where a plain DELETE drops 48/50. The drop and the fixes are measured; why cell overwrites escape it is not established, and I would rather flag that than invent a mechanism.

Refs #813

---

Added after review

A security second-pass (security-engineer, as AGENTS.md asks for on auth-sensitive storage changes) found a merge blocker I had missed, plus several smaller things. Three more commits.

5. test(storage): probe CQL readiness, not the docker port proxy

wait-for-test-dbs.sh checks with nc -z, which on Docker Desktop connects to the published-port proxy rather than to the server — it succeeds the instant docker run -p returns. Measured: nc -z on a fresh cassandra:4.1 succeeded at t=1s while cqlsh inside the same container still failed at t=3s. The gate that script exists to provide was absent for exactly the two slowest-booting backends.

It stayed hidden because go test ./... spends minutes on other packages before reaching internal/storage, which is long enough for the database to boot. A narrower -run does not.

wait-for-cql.sh asks the server from inside the container, which a proxy cannot answer for. Cold runs with no container present:

make test-cassandradb   nc: ready after 0s, CQL after 50s, suite green
make test-scylladb      nc: ready after 0s, CQL after  4s, suite green

6. fix(storage): stamp the user-delete cascade for the table it hits

Found re-reviewing commit 4 — a hole in my own fix. Converting session_tokens, oauth_states and authenticators to the server clock covered their direct call sites and missed the one that is not a direct call site.

DeleteUser cascades over schemas.UserOwnedCollections with a single generic DELETE, and three of those seven tables are the ones just converted. That delete stayed client-stamped, so those tables were still mixed: a session token inserted under the server clock and deleted under the client clock is dropped silently when the client runs behind — the token outlives the hard delete of its own user.

The guard could not see it: the statement names its table through a loop variable, so a source scan cannot classify it. It now (a) requires any write whose table is a variable to consult serverDomainTables, and (b) cross-checks that map against the domains it detected, in both directions. Mutation-verified — dropping either table from the map, adding a client-domain table to it, or removing the cascade's domain check entirely, each fails it with its own diagnostic.

7. fix(storage): anchor the session-token delete on the four remaining backends

The blocker. Commit 2 promoted the anchored-delete assertion to every backend, but only four of eight were anchored. MongoDB, ArangoDB, DynamoDB and Couchbase still matched an unanchored substring, so the new test failed on them. I had verified on sqlite, scylladb and cassandradb and stopped — exactly the gap AGENTS.md's "also run a non-SQL backend" rule exists to close.

backend	before
mongodb/session_token.go:66	bson.M{"$regex": userId} — unanchored and unescaped
arangodb/session_token.go:96	FILTER CONTAINS(d.user_id, @user_id)
dynamodb/session_token.go:91	strings.Contains(t.UserID, userId)
couchbase/session_token.go:103	WHERE CONTAINS(user_id, $1)

A delete for 42 took out tokens for 142 or provider:427. Harmless in practice — every user id is a UUIDv4 — but it is a live over-delete primitive, and the SQL provider removed it long ago.

• mongodb — anchored regex plus regexp.QuoteMeta. The old form interpolated the id straight into $regex, so an id containing regex metacharacters was evaluated as a pattern rather than matched literally.
• arangodb — d.user_id == @user_id OR RIGHT(d.user_id, LENGTH(@suffix)) == @suffix.
• dynamodb — HasSuffix instead of Contains.
• couchbase — CONTAINS stays as the server-side narrowing filter; the anchored predicate runs in Go. Doing the suffix test in N1QL means LIKE '%:'||$1, whose wildcards would need escaping to stay literal. Its comment claimed the substring match mirrored the SQL provider — untrue since SQL was anchored.

Also folded in from the review:

• The test now covers the bare <userID> stored form as well as <loginMethod>:<userID>, so the equality branch of the predicate is exercised. It never was, on any backend.
• AddOAuthState documents why its delete-then-insert is safe (fresh state.ID per call, so different partitions) and what would break it — a caller supplying a stable id turns it into a same-partition pair where two coordinator timestamps can drop the insert.
• The Cassandra authorizer_webhook_event_name index has no consumer left after commit 2. Said so in place rather than leaving it looking load-bearing; dropping it is a schema migration and belongs to a change that owns one. (authorizer_email_template_event_name is not dead — email_template.go still selects on it with =.)
• WithContext(ctx) on the two new full scans, so they are cancellable.

Backward compatibility

• No schema or data migration anywhere in this PR. The one index change is a comment.
• The anchored matches are strictly tighter, never looser. Every value written to session_tokens.user_id is <userID> or <loginMethod>:<userID> (memory_store SetUserSession, plus service_account:<id>), so the user id is always the final segment and == userId || HasSuffix(":"+userId) is exhaustive. Every caller passes a bare user.ID. The refresh_family_<id> reuse-detection entries live under the same session key and are still swept.
• DefaultTimestamp(false) and already-stored rows. A row written by the old binary carries Tc = t_insert + skew, where skew is app-host clock minus cluster clock. A new server-stamped delete at Ts = t_delete is dropped only if t_delete − t_insert < skew. If skew ≤ 0 there is no new exposure at all; if skew > 0 the window is skew measured from each row's own insert, not "until the old rows expire". The revocation paths that matter fire seconds to days after login. Practical advice: sanity-check NTP on app hosts before upgrading — a host minutes off NTP was already losing LWT deletes on the old binary, which is the bug being fixed.
• Rolling upgrade adds the mirror case (old client-stamped delete vs new server-stamped insert), bounded identically by |skew|. No new failure class, same magnitude as the one removed.
• One premise correction worth recording: gocql's client timestamp is not monotonic. gocql@v1.6.0/frame.go:1533 is a bare time.Now().UnixNano()/1000 with no per-session counter, so client stamping never gave a same-process ordering guarantee either. The change swaps one unsynchronised clock for a set of loosely-synchronised ones — better, not a proof.

Verification after review

Check	Result
make test-all-db — full storage suite, all 8 backends	exit 0, 0 failures (couchbase, postgres, sqlite, mongodb, arangodb, scylladb, cassandradb, dynamodb)
Anchored_Session_Token_Delete on all 8 backends	pass
Mutation test: restore the substring match on mongodb / dynamodb / arangodb	fails on each
Mutation test: 9 variants against the timestamp-domain guard	caught on all 9
Storage suite, cassandra:4.1 and scylladb	26/26 each
make test-cassandradb / make test-scylladb, cold	green
go build ./..., go vet ./...	clean

Known gaps, stated rather than papered over

• DeleteSessionTokensByNamespace is still unanchored-by-CONTAINS on the non-Cassandra backends. It has no live caller: its only consumers are in internal/graphql/_deprecated_update_env.go, and a leading _ excludes the file from the build. Left alone deliberately.
• Flaky ScyllaDB storage tests: Org_Domain_Operations and User_SCIM_Fields fail on read-after-write #813's User_SCIM_Fields never reproduced — 0 failures in 100 targeted runs, before and after fix(storage): stop mixing LWT and plain writes on cassandra partitions #818.
• An unexplained asymmetry. A plain UPDATE after an LWT insert loses 0/50, in the identical loop shape where a plain DELETE drops 48/50. The drop and the fixes are measured; why cell overwrites escape it is not established, and I would rather flag that than invent a mechanism.

---

Second review round

An independent principal-engineer pass over commits 5–7 (the ones written in response to the first review, which had had no outside eyes). Verdict was "no blockers", but it found four real things and three demonstrated false passes in the guard — all reproduced before and after. Two more commits.

MongoDB was the one backend whose predicate was not equivalent — and it diverged both ways

$regex is PCRE: . does not cross a newline, and $ also matches before a trailing newline. Verified live on mongo:4.4.15 against the SQL semantics:

stored user_id	SQL / Dynamo / Couchbase / Arango	MongoDB (before)
"T", "p:T", "a:b:T"	deleted	deleted
"xT", "p:xT"	kept	kept
"a\nb:T"	deleted	survives — missed revocation
"p:T\n"	kept	deleted — over-delete

The second row is the exact primitive this PR exists to remove, alive on one backend. regexp.QuoteMeta only ever made the needle literal — it never made . and $ agree with the other five. Now (?s)^(.*:)? + QuoteMeta(id) + \z.

Neither edge is reachable from application writes (login methods come from a fixed set, ids are UUIDv4) — but "unreachable today" is precisely the standard the substring match was removed under, so it gets the same treatment. The test now carries the newline shape; mutation-verified that reverting the pattern fails it on MongoDB and passes the other seven.

Arango, DynamoDB and Couchbase were confirmed genuinely equivalent, Arango live against arangodb:3.10.3 — including a stored value shorter than the suffix, and null/missing/numeric user_id, all with no AQL warnings.

The guard had three false passes

Each was demonstrated, then re-run after the fix:

mutation	before	after
invert if serverDomainTables[table] to if !…	passed	caught — negates the serverDomainTables lookup
client-stamped write placed one line above a server-stamped one	passed	caught — SessionToken mixes timestamp domains
replace the branch with a comment mentioning serverDomainTables	passed	caught — without consulting serverDomainTables

The inversion is the one that matters: one flipped ! client-stamps the server-domain tables and server-stamps the client-domain ones, and the guard said nothing. It was checking presence, not correctness.

Fixes: classification regions are now cut at the next write statement or the end of the enclosing function (so a neighbour's flag can't classify this statement), comment lines are dropped from regions, and the variable-table check requires DefaultTimestamp(false) in the same region and rejects a negated lookup. All 9 earlier mutations still caught — 12 of 12 now.

wait-for-cql.sh and a container leak

• It counted sleep 2 iterations, not wall time, so TIMEOUT=180 really meant "90 attempts" and the true ceiling was open-ended — and the "CQL after 50s" figure was measured the same way. Now a date-based deadline.
• No fail-fast: a missing or OOM-killed container burned the full timeout in silence, which at MAX_HEAP_SIZE=512M is plausible. Now docker inspect the state first and tail the container log on failure. Both paths verified: missing container → no such container, exited container → is 'exited', not running, each immediately.
• Each attempt is bounded by cqlsh's own --connect-timeout/--request-timeout rather than timeout(1), which is not present on macOS — where this runs most.
• Its knob is now TEST_CQL_WAIT_TIMEOUT. Overloading TEST_DB_WAIT_TIMEOUT (which means "wait for a port" and is routinely set low) would have made TEST_DB_WAIT_TIMEOUT=30 make test-cassandradb fail against this PR's own measured 50s cold boot.
• The leak: as standalone lines in the test-docker-up prerequisite, a probe failure aborted make before $(MAKE) test-cleanup in test-all-db's recipe ever ran — leaking all eight containers and every port, the exact failure that target's comment says was fixed.

Fixing the leak then broke the other half, which I caught by hitting it: with the waits moved wholly into test-all-db, a standalone make test-docker-up && go test ./internal/storage/ started against a still-booting Cassandra, and cassandradb failed the anchored-delete run. So: an advisory copy in test-docker-up (prefixed -, so a slow boot can't abort make) and the authoritative copy inside test-all-db's && chain. Re-running it there is free — 0s once CQL answers.

Verified on a faithful model of the recipe:

path	tests	cleanup	exit
probe fails	skipped	ran	non-zero
tests fail (7)	ran	ran	7 — the tests' status, not the teardown's
all pass	ran	ran	0

Smaller corrections

• The "three of seven" count was wrong — it is two. oauth_states is in serverDomainTables but is not user-owned, so the cascade never reaches it. Two server / five client. Corrected in all three places.
• The cascade comment overstated its reach. All backends cascade on the bare user_id, while memory_store stores session tokens as <loginMethod>:<id> — so that select finds no namespaced session-token rows on any backend. The live beneficiary is authenticators, whose user_id is bare; stamping the session-token half is correct and defensive rather than currently active.
• WithContext(ctx) on the cascade select and the final user-row delete, in the same function the previous commit edited.
• The sibling DeleteSessionTokensByNamespace carried the same two defects one function below the one just fixed: raw $regex interpolation on MongoDB, and an unescaped N1QL LIKE on Couchbase where _ is a single-char wildcard and every recipe name contains one (basic_auth, magic_link_login, service_account). Both fixed. It has no live caller — its only consumers are in _deprecated_update_env.go, which the leading _ excludes from the build (go list confirms zero matching build files) — but it is the same family one function away.

Final verification

Check	Result
make test-all-db	exit 0, 0 failures, all 8 backends; no containers left behind
CQL probe, cold	Cassandra 50s, Scylla 0–1s
Anchored_Session_Token_Delete, all 8 backends incl. the newline shape	pass
Guard mutations	12 of 12 caught
Mutation: revert the Mongo pattern	fails on MongoDB, passes the other 7
go build ./..., go vet ./...	clean

Found but deliberately not fixed here

The service-layer DeleteUser does not revoke namespaced session tokens. admin_users.go:406 calls StorageProvider.DeleteUser (cascade on bare user_id) and purges FGA tuples, but never DeleteAllUserSessions — and the cascade cannot see <loginMethod>:<id> rows. So a deleted user's session-token rows survive on every backend. Pre-existing, uniform, and testDeleteUserCascade misses it because it stores tokens with a bare id. I verified the code path, not the live behaviour. It wants its own issue and its own test rather than being folded in here.