Skip to content

SOW-0122: fast --list-invocations and stock IDX alignment; track remaining CLI gaps - #21

Open
ktsaou wants to merge 26 commits into
sow-0160/xz-dictionaryfrom
sow-0122/journalctl-perf
Open

ktsaou wants to merge 26 commits into
sow-0160/xz-dictionaryfrom
sow-0122/journalctl-perf

Conversation

@ktsaou

@ktsaou ktsaou commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Stacked on #20. Completes SOW-0122 (.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md) and opens SOW-0166, SOW-0167 and SOW-0168 for the remaining measured gaps.

Decisions for review (agent-chosen, pending your review)

  1. --list-invocations prefilter limited to 32 IDs.
    • SOW-0121 added a prefilter, an OR of every invocation-ID value, to keep sparse-ID journals cheap.
    • Every step evaluates every term, so with ~10,000 IDs the command was 240-590x slower than stock.
    • It now applies only up to 32 IDs (ELIGIBILITY_MAX_TERMS / invocationEligibilityMaxTerms). Above that, the walk matches stock's unit/boot walk.
    • Rows without an ID are skipped either way, so output is unchanged.
  2. IDX column sized like stock: the column is as wide as the widest index, at least 3 when the header shows. Previously, at 100+ rows stock printed IDX / -99 and we printed IDX / -99.

Measurements

Fixtures: 1.5M entries, as one 931 MB file and as 300 files. Full table: .local/sow0122/report.md (local, not committed), summarized in the SOW.

--list-invocations -u before after stock v262
Rust, file / dir 27.7 s / 13.0 s 0.23 s / 0.67 s 0.06 s / 1.03 s
Go, file / dir 14.5 s / 15.4 s 0.66 s / 1.81 s

Output is byte-identical to stock for --list-invocations (with and without -q) and --list-boots, in both languages.

Rust is otherwise at or faster than stock: full output is 2-4x faster. The new pending SOWs:

SOW Gap Measured
SOW-0166 Directory --since seek 17x slower than stock
SOW-0167 Every Go open loads the whole entry array 0.27 s / 160 MB floor per file
SOW-0168 --follow polls and reopens instead of waiting idle CPU and latency far above stock; needs your decision (inotify vs cheaper polling)

Validation

  • Query matrix vs v262 oracle: 3587/3587 PASS.
  • Parser parity: 134/134 in each language.
  • go test ./cmd/journalctl and cargo test -p journalctl: pass.
  • New paired tests for the IDX width rule.

Summary by cubic

Completes SOW-0122 by fixing --list-invocations performance and IDX column alignment in both CLIs, and opens SOW-0166, SOW-0167, and SOW-0168 for the remaining measured gaps. Output is now byte-identical to stock for --list-invocations and --list-boots; Rust is otherwise at or faster than stock. Carries the sow-0160/xz-dictionary base via merges, with its SOWs recorded in the ledger.

Two agent-chosen decisions, both approved by the user on 2026-11-03:

  • The valid-ID prefilter (an OR of every invocation ID) is now bounded to 32 IDs. Above that, the walk matches stock's unit/boot walk; rows without a valid ID are skipped either way, so output is unchanged. This takes --list-invocations -u from 240-590x slower than stock to near parity (Rust 0.23 s file / 0.67 s dir vs stock 0.06 s / 1.03 s).
  • The IDX column is sized to the widest index, at least 3 when the header is shown, matching stock's table formatting.

Residual gaps tracked as new SOWs

  • SOW-0166: directory --since seek at 17x slower than stock; the SOW records the reproduction procedure.
  • SOW-0167: every Go open loads the full entry array (0.27 s and 160 MB floor per file).
  • SOW-0168: --follow polls and reopens instead of waiting; decided 2026-11-03 to make portable polling cheap (reuse readers, check header counters per poll, back off while idle) — inotify needs a separate policy decision.

Validation: 3587/3587 query matrix PASS against the v262 oracle, 134/134 parser parity in each language, paired IDX-width tests added, and both go test ./cmd/journalctl and cargo test -p journalctl pass.

Written for commit cffcb3d. Summary will update on new commits.

View guided diff Turn on auto-fix

--list-invocations evaluated an OR of every invocation ID on each step,
which made many-ID journals hundreds of times slower than stock. The
prefilter now applies only to a few IDs. Tables size the IDX column to the
widest index, as stock does. Records the SOW-0122 measurements and tracks
the remaining gaps in SOW-0166, SOW-0167 and SOW-0168.
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f7cb65b8-c2d8-49de-9653-21d7c724e370

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codacy-production

codacy-production Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics -8 complexity · 0 duplication

Metric Results
Complexity -8
Duplication 0

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

@codacy-production codacy-production Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

✅ No blocking issues noticed

No blocking findings were identified. The change lacks committed regression tests for the large-ID prefilter behavior, sparse-ID behavior, output preservation, and Rust/Go parity; these remain important validation gaps. The pending performance gaps are documented but not covered by automated tests.

A documentation formatting issue was also reported in the follow-mode tracking note.

About this PR

  • The central bounded-prefilter behavior and cross-language parity are not covered by committed regression tests. Add automated coverage for these cases and for the documented remaining CLI gaps.

Test suggestions

  • Verify both CLIs disable the eligibility prefilter above 32 IDs and preserve invocation results.
  • Verify sparse journals with 32 or fewer IDs retain eligibility filtering and correct results.
  • Verify IDX width for negative, multi-digit, and empty index sets in quiet and non-quiet modes.
  • Verify boot and invocation output formatting matches stock for wide indexes in both languages.
  • Verify cross-language output parity for the bounded-prefilter behavior.
  • Add automated regression coverage for the documented directory-seek, Go eager-loading, and follow-mode gaps.
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Verify both CLIs disable the eligibility prefilter above 32 IDs and preserve invocation results.
2. Verify sparse journals with 32 or fewer IDs retain eligibility filtering and correct results.
3. Verify boot and invocation output formatting matches stock for wide indexes in both languages.
4. Verify cross-language output parity for the bounded-prefilter behavior.
5. Add automated regression coverage for the documented directory-seek, Go eager-loading, and follow-mode gaps.

TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread SOW-status.md Resolved
ktsaou added 22 commits November 1, 2026 21:09
…tighten SOW-0166 gate and SOW-0168 policy note

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .agents/sow/SOW-status.md Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant