Repository navigation
DDIR: project values in the first phase of join, half join and reduce - #1058
Merged
frankmcsherry merged 4 commits intoOct 31, 2026
Merged
Conversation
Consolidate values that agree on the fields the join reads before matching. Keep the existing merge for whole values and leading leaf projections; sort other projections within each real key. Unread values share one token. Collision checks retain the original keys. Adapt the half join's staging API without changing its demand yet. Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex) and Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
Use the join's projected-value staging for the arranged side. Consolidate released stream rows on their projected values within each block, preserving real keys and both update and data times. Original coordinates still serve output gathering and collision checks. Keep the whole-read stream path. Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex) and Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
Port the compiled-graph analysis for a diff-weighted map and total sum, requiring empty-group suppression independent of the key. Use projected columns in input presentation only with this fact and without a value limit. Keep output identity and raw novel time seeds. Other reducers retain whole values: in particular, signed projected extrema can change when values cancel. Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex) and Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
- The reduce analysis evaluates the empty case directly. Its argument has the compiled input's shape and corgi's ops are total, so the evaluation cannot fail on data; a panic would be a corgi bug, and is no longer caught. - Half-join needles come from the consolidated stream, so identifiers whose rows all cancel are neither probed nor staged. The collision walk over the block's rows skips rows whose identifier is not a needle. - The join's ungrouped path reuses its row-index scratch across keys. - The join module's token documentation is rewrapped. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Join, half join and reduce now project their inputs to the fields they read before assigning the identifiers the proxy tactics work with, so updates that differ only in unread fields share an identifier and cancel before any matching. This protects them against noisy input arrangements: churn in columns nobody reads used to cost full matching work.
Three commits:
countand weighted sums), and requires that a group whose projected values cancel would also produce nothing from the original reducer. Other reducers keep whole values.minandmaxcannot be projected: DDIR allows signed collections, and for((1,a),+1), ((1,b),−1), ((2,c),+1)the minimum first field is 1 before projecting and 2 after.Join and half join are bilinear, so equal projected values can share a representative and sum their diffs.
Measured (server CPU per tick, one worker, three alternating trials, medians). 100,000 people with an unread
last_seenfield replaced every tick; consumers read only id and name:Under unread churn, the join's and delta join's retained tokens and matches drop to zero. CPU doesn't go flat: ingesting and staging the input remain, and reduce still visits the changed keys.
On today's workloads it costs a little: LDBC SF1 one-off replay +1.1% CPU (41.2 → 41.7 s), SCC initial computation +1.3%, reachability and SCC churn within noise. Answers are identical everywhere: LDBC answer digests at all 500 merged ticks, graph snapshots, and the noisy programs.
Checks: the DD, dogs3,
interactiveandddir-serversuites, and 2,000 difftest cases with every default oracle, including the delta-join one. New tests cover cancellation before join matching, both half-join stream times surviving consolidation, and the boundaries of the reduce analysis. No runtime switches.Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex), who recommended landing as is, and by Claude.
🤖 Generated with Claude Code
https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o