Repository navigation
DDIR reduce: integer values are their own ids - #1068
Closed
frankmcsherry wants to merge 1 commit into
Closed
frankmcsherry wants to merge 1 commit into
frankmcsherry wants to merge 1 commit into
Conversation
A reduce's value ids named rows of a pool, numbered per window by sorting each key's values with corgi's segmented sort. A value that is a primitive integer (perhaps in one-field products) now names itself: its bits with the sign bit flipped, so ids ascend in value order and equal values share one without a sort. The reduction reads such values from their ids, corrections are a consolidation of ids, and emission writes the ids back as the value column. The input decides by the values `present` compares: their projection, where a weighted sum is proven to merge equal projections, else the whole value. A pair read at one integer field projects to `(int)`, and names itself. Its integers are read from the projected columns, where they are held as `i64`s, else gathered. The output decides by the reduction's output value shape. Both decide per window: only where each identifier holds one real key, always for integer keys and for hashed keys unless the window has a collision, where two real keys under one identifier could hold equal values. Such windows keep pool ids, and a batch built from both kinds of window gathers from the pool and the values together. `present` now builds each window's labels and rows at their final size. Grown by doubling, they cost a presentation up to half again its time, depending on the allocator's state: a projected pair sum ran its presentations at 3.1 or 2.1 ms, and its retires at 6.4 or 2.6 ms, by the launch. Reserved, it runs at 2.0 and 2.5. Implemented and measured by a Claude subagent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A reduce's value ids named rows of a pool, numbered per window by sorting each key's values with corgi's segmented sort. A value that is a primitive integer, perhaps inside one-field products, now names itself: its bits with the sign bit flipped, so ids ascend in value order and equal values share an id without a sort. One commit, in
interactive/src/runtime/reduce.rs(+429 −47, about half tests).What changes on the fast path.
i64lanes. There is no gather, nosort_blocks, and no pool entry per value. A key held by several chunks gets a stable sort of its ids.When it applies.
presentcompares, which since DDIR: project values in the first phase of join, half join and reduce #1058 is their projection where a reduction is proven to merge equal projections. A pair read at one integer field projects to(int)and names itself.countprojects to a unit and keeps row ids. Projected fields held as bytes are gathered before their integers are read.Measured with a probe of the reduce backend, each scenario in its own process, base and branch alternating, median of five launches:
min, 1% churnmin, 1% churnmin, 10% churndistinct(int, int)value summed at its first fieldThe
countscenarios present one row per key under #1058 (1,000 rows from 126k history rows), and the base's timings for them swing between 1.65 and 3.0 ms from launch to launch, so they show no real change either way.One shared change.
presentnow builds each window'slabelsandrowsat their final size, for both kinds of id. Grown by doubling, they cost a presentation up to half again its time depending on the allocator's state: the projected pair sum ran at 3.1 or 2.1 ms by the launch, and at 2.0 ms with the reservation in 8 of 8 launches.The join is unchanged: it already uses integer values as their own tokens, and it numbers no values with a segmented sort.
Checks:
cargo test -p differential-dataflow -p differential-dogs3 -p interactive -p ddir-server(490 passed) and the 1,000-case release difftest. The new end-to-end test runscount,distinct,min,collectand a weighted sum over bare, one-field-product and pair values, with ties, retractions andi64::MIN/i64::MAX, and requires each to equal the run where every identifier collides, which forces pool ids everywhere.Implemented and measured by a Claude subagent.
🤖 Generated with Claude Code
https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o