Skip to content

fix(storage): put every cassandra table in one timestamp domain - #822

Closed
lakhansamani wants to merge 1 commit into
test/cassandradb-coveragefrom
fix/cassandra-single-use-claim
Closed

lakhansamani wants to merge 1 commit into
test/cassandradb-coveragefrom
fix/cassandra-single-use-claim

Conversation

@lakhansamani

@lakhansamani lakhansamani commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Closes out the allow-list #818 deliberately left behind. Based on #821 (shares session_token.go); rebase onto main once that merges.

The bug: a single-use session token could be claimed twice

DeleteSessionTokenByUserIDAndKey consumes a token with DELETE … IF EXISTS and treats the applied flag as the exactly-once arbiter behind refresh-token rotation — the comment in the code says so explicitly. That LWT is stamped from the Paxos ballot, i.e. the server clock. AddSessionToken's plain insert carried gocql's client timestamp.

When the client clock runs ahead of the server, the tombstone lands below the insert. The row survives — but Paxos saw a live row and returned applied=true. The next claim of the same token sees the row still there and also returns applied=true.

Measured on scylladb/scylla, client clock emulated 10ms ahead with WithTimestamp, 20 insert-then-two-LWT-deletes cycles:

insert double-claimed row survived both deletes
plain, client clock +10ms 20/20 20/20
plain, no skew 0/20 0/20
DefaultTimestamp(false) 0/20 0/20

oauth_states has the identical shape for single-use authorization codes. authenticators mixes an LWT insert and two conditional updates with a plain update and a plain delete.

The fix, and why not the other one

DefaultTimestamp(false) on the plain writes to those three tables. gocql then sends no timestamp and the coordinator assigns one — the same clock the LWT already uses.

Per query, never on the cluster. gocql round-robins coordinators, so cl.DefaultTimestamp = false would push the 20-odd tables that never touch an LWT off a single monotonic client clock and onto several skewed node clocks. Strictly worse than today.

It adds no Paxos round-trip. That is what makes this a routine change rather than the performance decision I flagged on #818: converging the other way means lifting AddSessionToken to IF NOT EXISTS, i.e. a Paxos round on every login.

For completeness, both directions were measured on the org_domain shape (LWT insert, then delete), 50 pairs each:

delete dropped
plain 48/50 ambient, 50/50 with forced skew
… IF EXISTS (#818's fix) 0/50
server-stamped plain 0/50

The guard

Reframed from "LWT vs plain" to the invariant that actually matters — server-stamped vs client-stamped — and it now recognises DefaultTimestamp(false) as server-domain. knownMixedTables is empty.

It also had a blind spot worth naming: the statement regex matched "…" but not raw strings, so it never saw AddSessionToken's insert — one of the two writes it was written to watch. Both quote forms now, and the five tables it covers are asserted by name, so a regex drift fails loudly instead of silently losing coverage.

Mutation-verified, all five server-stamps:

SessionToken insert            guard caught it: YES
OAuthState insert              guard caught it: YES
SessionToken plain delete      guard caught it: YES
Authenticators plain update    guard caught it: YES
Authenticators plain delete    guard caught it: YES

Verification

  • Full storage suite, 26/26 subtests on scylladb and on cassandra:4.1.
  • TEST_DBS=sqlite go test -p 1 ./...: pass.
  • go build ./..., go vet ./..., gofmt clean.

A single-use session token could be claimed twice.

DeleteSessionTokenByUserIDAndKey consumes a token with DELETE ... IF
EXISTS and treats the applied flag as the exactly-once arbiter behind
refresh-token rotation. That LWT is stamped from the Paxos ballot, i.e.
the server clock, while AddSessionToken's plain insert carried gocql's
client-side timestamp. With the client clock ahead, the tombstone lands
BELOW the insert: the row survives, but Paxos saw it and returned
applied=true -- so the next claim of the same token also returns
applied=true. Measured on scylladb/scylla with the client clock 10ms
ahead: 20 of 20 tokens claimed twice, 0 of 20 after the fix.
oauth_states has the identical shape for single-use authorization codes.

The fix is DefaultTimestamp(false) on the plain writes to the three
tables that mixed domains, which hands the timestamp to the coordinator
-- the same clock the LWT already uses. Per query, never on the cluster:
gocql round-robins coordinators, so a global switch would expose the
20-odd tables that never touch LWT to inter-node skew instead. It adds
no Paxos round-trip, so the hot login path is untouched; converging the
other way (lifting the inserts to LWT) would have cost one per login.

The guard is reframed from "LWT vs plain" to the invariant that actually
matters, server-stamped vs client-stamped, and recognises
DefaultTimestamp(false) as server-domain. Its allow-list is now empty.

It also had a blind spot worth naming: the statement regex matched only
"..." and not raw strings, so it never saw AddSessionToken's insert --
one of the two writes it was written to watch. Both quote forms now.
Mutation-verified: removing any one of the five server-stamps fails it.

Full storage suite green on scylladb and cassandra:4.1, 26 subtests each.

Refs #813
@lakhansamani

Copy link
Copy Markdown
Contributor Author

Superseded by #824, which carries this change unmodified alongside the other three commits in one PR — no merge-order dependency between them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant