Skip to content

DDIR: project values in the first phase of join, half join and reduce - #1058

Merged
frankmcsherry merged 4 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-project-first
Oct 31, 2026
Merged

frankmcsherry merged 4 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-project-first

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

Join, half join and reduce now project their inputs to the fields they read before assigning the identifiers the proxy tactics work with, so updates that differ only in unread fields share an identifier and cancel before any matching. This protects them against noisy input arrangements: churn in columns nobody reads used to cost full matching work.

Three commits:

  • Join. Values that agree on the fields the join reads share one token. Whole values and leading leaf projections keep the existing merge; other projections sort within each real key, and single rows skip the sort. Collision checks still use the original keys.
  • Half join. Both inputs project: the arranged side reuses the join's projected staging, and released stream rows consolidate on their projected values within each block, keeping the real key, the update time and the carried data time (all three affect half-join results).
  • Reduce. Projection is sound only for some reducers. This ports the compiled analysis for a diff-weighted map followed by a total sum (which covers count and weighted sums), and requires that a group whose projected values cancel would also produce nothing from the original reducer. Other reducers keep whole values. min and max cannot be projected: DDIR allows signed collections, and for ((1,a),+1), ((1,b),−1), ((2,c),+1) the minimum first field is 1 before projecting and 2 after.

Join and half join are bilinear, so equal projected values can share a representative and sum their diffs.

Measured (server CPU per tick, one worker, three alternating trials, medians). 100,000 people with an unread last_seen field replaced every tick; consumers read only id and name:

Operator Unread replacements per tick Before (ms) After (ms)
join 100,000 58.3 27.7 (−52%)
delta join (half joins) 100,000 62.2 17.3 (−72%)
half join probing an arrangement 100,000 88.7 39.9 (−55%)
reduce (weighted sum) 100,000 34.4 31.6 (−8%)
any of these 0 within noise within noise

Under unread churn, the join's and delta join's retained tokens and matches drop to zero. CPU doesn't go flat: ingesting and staging the input remain, and reduce still visits the changed keys.

On today's workloads it costs a little: LDBC SF1 one-off replay +1.1% CPU (41.2 → 41.7 s), SCC initial computation +1.3%, reachability and SCC churn within noise. Answers are identical everywhere: LDBC answer digests at all 500 merged ticks, graph snapshots, and the noisy programs.

Checks: the DD, dogs3, interactive and ddir-server suites, and 2,000 difftest cases with every default oracle, including the delta-join one. New tests cover cancellation before join matching, both half-join stream times surviving consolidation, and the boundaries of the reduce analysis. No runtime switches.

Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex), who recommended landing as is, and by Claude.

🤖 Generated with Claude Code

https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o

frankmcsherry and others added 4 commits October 30, 2026 12:01
Consolidate values that agree on the fields the join reads before matching.
Keep the existing merge for whole values and leading leaf projections; sort
other projections within each real key. Unread values share one token.
Collision checks retain the original keys. Adapt the half join's staging API
without changing its demand yet.

Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex) and Claude.

Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
Use the join's projected-value staging for the arranged side. Consolidate
released stream rows on their projected values within each block, preserving
real keys and both update and data times. Original coordinates still serve
output gathering and collision checks. Keep the whole-read stream path.

Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex) and Claude.

Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
Port the compiled-graph analysis for a diff-weighted map and total sum,
requiring empty-group suppression independent of the key. Use projected
columns in input presentation only with this fact and without a value limit.
Keep output identity and raw novel time seeds. Other reducers retain whole
values: in particular, signed projected extrema can change when values cancel.

Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex) and Claude.

Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
- The reduce analysis evaluates the empty case directly. Its argument has the
  compiled input's shape and corgi's ops are total, so the evaluation cannot
  fail on data; a panic would be a corgi bug, and is no longer caught.
- Half-join needles come from the consolidated stream, so identifiers whose
  rows all cancel are neither probed nor staged. The collision walk over the
  block's rows skips rows whose identifier is not a needle.
- The join's ungrouped path reuses its row-index scratch across keys.
- The join module's token documentation is rewrapped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
@frankmcsherry
frankmcsherry merged commit 219c93c into TimelyDataflow:master-next Oct 31, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant