[frankmcsherry]

DDIR's chunks now count themselves: each chunk's contents log their updates and bytes when they are made and again when they drop, against the account of the operator that made them, on DDIR's own logging stream. DDIR has a diag.arrangements view that reads "this arrangement's chunks hold N updates in B bytes", wherever the chunks are: its trace, its batcher, a merge in progress, or a batch a reader still holds. On reachability over a 1M-node, 4M-edge graph, the chunks hold 256.9 MiB, 78% of the live heap the program's drop releases; the rest is mostly capacity never written, which is not counted by design.

DD: no change. The first commit's Chunk::heap_bytes, SpineBatch::heap_bytes and byte fields on the batch, merge and drop events are reverted.

DDIR

• A ticket in CorgiChunk's shared contents. Made with the contents, it logs +(updates, bytes) against the account current at that moment and remembers it; its drop logs the same amounts back. Clones of a chunk share the ticket and log nothing; a copy of the contents gets its own ticket in the same account. Bytes are the key and value columns, the time lanes and the diffs, by stored length rather than capacity. Every chunk with contents is built by one constructor, so none escapes a ticket.

• An account is an operator's id with a handle to the "ddir/chunks" logger, and one is current per worker (runtime/account.rs). Accounts are entered in three places:

  • the arrange operator's batcher is wrapped (AccountedBatcher, passed to arrange_core) to enter its account around insert and extract, which covers the chunker's flushes and the batcher's merges;
  • the reduce enters its own account when it writes output chunks (it learns its operator id with peek_identifier just before reduce_with_tactic builds the operator, as nothing in between allocates one);
  • CorgiChunk's merge, extract, advance and settle make their outputs under their inputs' account, which covers the batch builder and the spine's merges wherever they run, with no change to DD.

  Chunks made with no operator's account current go to the worker's unattributed account. With no logger registered there are no accounts, and tickets measure nothing.

• diag.records and diag.bytes come from these events instead of DD's batch, merge and drop events, keyed by (worker, operator) as before. Views diag.arrangements (program, statement, operator, address → updates, bytes, summed over workers) and diag.arrangements_by_program. Unattributed holdings are their own row, program (unattributed); an operator that shuts down while its chunks are still alive hands what they hold to that row, where their drops are counted.

Validated against the allocator, with a server built on the system allocator and live heap read from vmmap:

	reach, 1M nodes / 4M edges	SCC, 300K nodes / 1M edges
Reported	8.42M updates, 256.9 MiB	9.18M updates, 347.3 MiB
Unattributed	0	0
Live heap: idle → loaded → after drop	0.7 → 330.8 → 0.9 MiB	0.7 → 462.4 → 1.0 MiB
Share of what drop releases	78%	75%

The allocator counts capacity. The first commit counted it too, and found 40 MiB (reach) and 64 MiB (SCC) of capacity beyond length, mostly time lanes a merge reserves for both inputs and diffs grown by doubling; with it the shares would be 90% and 89%. On SCC the chunks hold 40 updates the spine's events did not show: the batchers of the two labels arrangements hold them for later iterations.

Cost: median ms of CPU per tick, reachability on a 10K-node, 40K-edge random graph replacing 10 edges a tick, five rounds of builds and settings alternating.

	diagnostics off	diagnostics, no views	diagnostics and shipped views
master-next	0.408	0.439	0.597
first commit	0.414	0.444	0.835
this PR	0.414	0.455	0.848

Tickets cost nothing measurable with diagnostics off, and the events under 0.02 ms per tick with them on. The two new views add 0.24 ms per tick, as in the first commit.

What it does not count, largest first:

1. Capacity allocated and never written, by design: 40 MiB on reach.
2. Working state outside chunks: the reduce's from its last pass (16 MiB on reach), the half join's own queue of stream updates, which is not a CorgiChunk, and the chunker's blocks between a batcher insert and its next flush.
3. Containers in flight between operators, exchange buffers, feed staging and peek snapshots, which are CorgiContainers. Containers made from chunks (as_collection, peeks) share the chunks' key and value columns, which are counted once, in the chunk.

Checks: cargo test -p differential-dataflow -p differential-dogs3 -p interactive -p ddir-server (485 passed) and the 1,000-case release difftest. A new test shows a merge's output counted in its inputs' account, and an input a reader keeps after the merge counted until the reader drops it. tests/diag.rs checks that every arrangement holds at least 16 bytes per update, that the per-program view equals the sum of the per-arrangement view, and that after 30 install/drop cycles nothing is left in the dropped program's accounts or the unattributed one.

Implemented and validated by a Claude subagent.

🤖 Generated with Claude Code

https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o