[frankmcsherry]

A new module, runtime::pages, encodes a chunk's keys, values, times and diffs as pages of about 16 KiB of uncompressed bytes, and decodes a run of pages back into corgi columns equal to the original rows, for all columns or a chosen subset.
Nothing in the engine uses it yet; it is the first step toward chunks compressed at settle, restrict (#1072) decoding only the pages it needs, and a pager.
On the SF1 chunks after the LDBC replay, 3,345.4 MB become 647.0 MB of pages plus 16.1 MB of index.

What changes

• Pages::encode(keys, vals, times, diffs, target) cuts rows into pages and codes each column of each page on its own.
  • A page never splits a row, so a row's list elements stay in its page.
  • A page ends where the identifier changes (the integer key, or the hash lane of a compound key), when there is such a place in the last quarter of the target.
• A resident index holds each page's first identifier, first row and byte start, and where each of its columns starts.
• Pages::decode(pages, &Select) decodes a run of pages into one set of columns.
  • Select takes a corgi Demand for the keys and for the values, plus whether to decode times and diffs; the result equals Demand::project on those rows.
  • Columns not demanded are never read.
• Pages::pages_for(id) names the pages that may hold an identifier.
• Shapes become columns recursively:
  • a leaf is one column;
  • a product is its fields' columns;
  • a list is a column of lengths, then its elements' columns;
  • a sum is a column of tags, then each variant's rows for the page;
  • a unit has no column;
  • a reference to list rows is carried by value, and decodes to a reference into an arena of just those rows.
• Each column of a page is coded by the smallest of:
  • raw;
  • frame of reference with bit-packing (zero bits for a constant column);
  • a per-page dictionary;
  • for eight-byte values, the high and low 32 bits coded apart.
• Signed integers are coded as x ^ 2^63, and floats as their total-order keys.
• Bit-unpacking is scalar safe Rust; there is no unsafe and no new dependency.

Why

A chunk is one set of contiguous columns, so a reader of a few keys pays for all of them, and nothing can be compressed or paged out.
Pages that decode independently let a reader bring in only the pages its keys can be in, and only the columns it reads.
The codecs are lightweight only, pure Rust with no new dependencies; general-purpose compressors are a separate, later decision.
Splitting 64-bit integers into halves matters on LDBC, whose identifiers carry few distinct high halves.

Measurements

The 580 chunks of every live arrangement after the 808-tick LDBC SF1 replay, captured by the compressed-pages spike; one thread, mini-00.

	raw MB	pages MB	spike, light codecs MB
key identifiers (integer keys / hash lanes)	557.0 / 83.7	47.2 / 69.2	40.9 / 61.3
other key integers	172.6	68.0	116.8
value integers	927.5	314.9	405.1
text bytes	158.8	132.6	132.4
list lengths	94.6	6.5	6.2
times and diffs	1,350.4	8.2	7.6
total, with index	3,345.4	663.0	777.0

• Encoding takes 4.53 s (0.74 GB/s of raw bytes); decoding every page into corgi columns takes 0.70 s (4.80 GB/s).
• One page takes 2.73 µs to decode in full (16.0 KB), and 0.49 µs for its identifier column alone.
• Every chunk decodes back to its original columns, at the same leaf storages.

Open questions

• Deltas for sorted identifier lanes would recover most of the 14 MB the identifier lanes lose to the spike, about 2% of the total.
• Index entries are usize; u32 would take the index from 16.1 to 8.9 MB.
• Text barely compresses under these codecs (0.835 of raw) and is 20% of the pages; that belongs to the later decision on a general-purpose compressor.
• corgi's public surface at 2751e04 made the encoder take long ways round:
  • Value::as_u8 is crate-private, so a byte leaf is read through as_i64, a copy widened to i64s;
  • floats are read and rebuilt through as_f64 and Value::f64, a pass and a copy each way;
  • Prim isn't exported, so a leaf's storage is inferred from its shape;
  • Bounds::len, span and total are crate-private, and sums can be built only from usize tags.

Tests

• New: round trips over every shape with edge values, each codec on runs it is smallest for, runs of pages against the gathered rows, and 32 key and value demand pairs against Demand::project.
• cargo test -p interactive: 298 passed, 0 failed, 9 ignored; no warnings.

Implemented and measured by a Claude Opus subagent; reviewed and opened by Claude Code.

🤖 Generated with Claude Code

https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL