From 993649bbbc98f85f5e3fd943850cdd9cc71c3344 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sun, 1 Nov 2026 21:03:01 +0000 Subject: [PATCH 1/8] Bound the invocation prefilter and align table IDX like stock --list-invocations evaluated an OR of every invocation ID on each step, which made many-ID journals hundreds of times slower than stock. The prefilter now applies only to a few IDs. Tables size the IDX column to the widest index, as stock does. Records the SOW-0122 measurements and tracks the remaining gaps in SOW-0166, SOW-0167 and SOW-0168. --- .agents/sow/SOW-status.md | 13 ++- ...260621-journalctl-performance-followups.md | 96 +++++++++++++++++-- ...6-20261101-directory-realtime-seek-cost.md | 83 ++++++++++++++++ ...67-20261101-go-reader-eager-entry-array.md | 84 ++++++++++++++++ ...journalctl-follow-wait-and-error-parity.md | 89 +++++++++++++++++ SOW-status.md | 12 +-- go/cmd/journalctl/colors.go | 9 +- go/cmd/journalctl/invocation.go | 10 ++ go/cmd/journalctl/main.go | 49 +++++----- go/cmd/journalctl/table_json_test.go | 22 +++++ rust/src/cmd/journalctl/colors.rs | 10 +- rust/src/cmd/journalctl/invocation.rs | 11 +++ rust/src/cmd/journalctl/main.rs | 54 +++++++---- 13 files changed, 475 insertions(+), 67 deletions(-) rename .agents/sow/{pending => done}/SOW-0122-20260621-journalctl-performance-followups.md (62%) create mode 100644 .agents/sow/pending/SOW-0166-20261101-directory-realtime-seek-cost.md create mode 100644 .agents/sow/pending/SOW-0167-20261101-go-reader-eager-entry-array.md create mode 100644 .agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md diff --git a/.agents/sow/SOW-status.md b/.agents/sow/SOW-status.md index a992db10..44ee8da3 100644 --- a/.agents/sow/SOW-status.md +++ b/.agents/sow/SOW-status.md @@ -11,7 +11,8 @@ order; the user may reorder it. 1. SOW-0156 - Fixed-header consistency audit (includes merged SOW-0155). 2. SOW-0161 - File indexer source posting membership (completed 2026-11-01). 3. SOW-0160 - Go XZ reader dictionary allocation (analysis done; needs user decision D1). -4. SOW-0122 - journalctl performance follow-ups. +4. SOW-0122 - journalctl performance follow-ups (completed 2026-11-01; + gaps tracked by SOW-0166, SOW-0167, SOW-0168). 5. SOW-0097 - Go duplication debt. 6. SOW-0098 - Rust legacy core / `jf` duplication debt (feeds SOW-0162). 7. SOW-0153 - Consumer docs release clarity. @@ -35,8 +36,12 @@ order; the user may reorder it. line numbers need refreshing at activation. - SOW-0098 - Rust Legacy Core Duplication Debt: real `jf` duplication; feeds SOW-0162. -- SOW-0122 - Journalctl Performance Followups: unmeasured large-input and - `--follow` scalability work for the CLIs. +- SOW-0166 - Directory Realtime Seek Cost: directory `--since` 17x slower than + stock (SOW-0122 measurement). +- SOW-0167 - Go Reader Eager Entry Array Load: every Go open loads the full + entry array. +- SOW-0168 - Journalctl Follow Wait And Error Parity: polling/reopen follow + cost; needs a user decision. - SOW-0125 - Netdata Filter Operators Gap Analysis: feature analysis; needs semantic decisions before implementation. - SOW-0153 - Consumer Documentation Release Clarity: stale `go/v0.3.0` text and @@ -52,6 +57,8 @@ order; the user may reorder it. ## Recently Closed Or Completed +- SOW-0122 - Journalctl Performance Followups: completed 2026-11-01; + `--list-invocations` 8-120x faster and IDX alignment matches stock. - SOW-0161 - File Indexer Source Posting Membership: completed 2026-11-01; source histograms count the captured ENTRY population once per entry. - SOW-0137 - Rust Writer Residual Array-Open Performance: completed 2026-11-01. diff --git a/.agents/sow/pending/SOW-0122-20260621-journalctl-performance-followups.md b/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md similarity index 62% rename from .agents/sow/pending/SOW-0122-20260621-journalctl-performance-followups.md rename to .agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md index 00cb17d3..f105a8a6 100644 --- a/.agents/sow/pending/SOW-0122-20260621-journalctl-performance-followups.md +++ b/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md @@ -2,11 +2,12 @@ ## Status -Status: open +Status: completed -Sub-state: pending activation; prerequisite SOW-0121 is completed, including vacuum ordering. This separate -scalability opportunity does not replace mandatory performance preservation in -the user-requested SDK/journalctl workstreams prepared on 2026-10-14. +Sub-state: completed 2026-11-01 in the autonomous backlog run. Measured the +envelope, fixed the `--list-invocations` cost and IDX alignment parity in both +CLIs; the three other measured gaps are tracked by SOW-0166, SOW-0167 and +SOW-0168. Scope choices are agent-chosen, pending user review (PR body). ## Requirements @@ -91,7 +92,8 @@ Risks: ## Pre-Implementation Gate -Status: pending activation +Status: passed 2026-11-01 (measurement below; the original text is kept as +the activation record). Problem / root-cause model: @@ -169,7 +171,45 @@ Open decisions: ## Implications And Decisions -Pending activation. +Measurement (2026-11-01, release builds at commit 96eecf5, stock v262 built +locally, synthetic fixtures from the Go writer: one 931 MB file and a +300-file 1.8 GB directory, 1.5M entries, 6 boots, 45 active units with 221 +invocations each; 3 runs, median; host load 16-66 on 112 CPUs, so only gaps +of 5x or more are claimed). Full table and commands: +`.local/sow0122/report.md`, `.local/sow0122/run-all.sh` (local only). + +- Rust is at or faster than stock on full forward/reverse output, `-u`, + `-b -5`, field matches, `-n` and single-file `--since` (full output is 2x + faster than stock on the file and 4x on the directory). +- Gaps of 5x or more: + 1. `--list-invocations -u`: Rust 29.4 s / Go 12.1 s vs stock 0.05 s on the + file; 11.9 s / 16.1 s vs 0.96 s on the directory. Fixed here. + 2. Directory `--since` seek: 3.5 s vs 0.21 s (up to 17 s vs 0.16 s near the + tail). Tracked by SOW-0166. + 3. Go opens load the complete entry array (0.27 s and 160 MB floor per + file). Tracked by SOW-0167. + 4. `--follow` polls and reopens the input (Go 21-30% idle CPU, Rust 2-6%, + stock 0%; latency 0.2-0.8 s vs 2-4 ms). Tracked by SOW-0168 (needs a + user decision). +- Output: default output byte-identical across all three except + `--list-invocations`, where stock right-aligns a wider IDX column once + indexes reach three digits plus sign. Fixed here for invocations and boots. + The `Error: ` stderr prefix difference is tracked by SOW-0168. + +Decisions (agent-chosen, pending user review): + +1. Root cause of gap 1: SOW-0121 added a valid-ID eligibility prefilter (an + OR of every invocation ID value) to keep sparse-ID journals cheap. Each + step evaluates every term, so with about 10,000 IDs it is far slower than + stock's unit/boot walk (logs-show.c `discover_next_id`, + systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492, + src/shared/logs-show.c:1978-2183). The prefilter is now kept only for at + most 32 IDs (Rust `ELIGIBILITY_MAX_TERMS`, Go + `invocationEligibilityMaxTerms`); with zero IDs the walk still ends at + once. Rows without a valid ID are skipped by the walker either way, so + output is unchanged. +2. IDX width: both tables size IDX to the widest index, at least 3 with the + header, and pad the header (`table_index_width` / `tableIndexWidth`). ## Plan @@ -231,16 +271,52 @@ Failure handling: ## Validation -Pending activation. +- `--list-invocations -u` after the fix (same fixtures, single runs, load + about 36): Rust 0.23 s file / 0.67 s directory (was 27.7 / 13.0); Go + 0.66 s / 1.81 s (was 14.5 / 15.4); stock 0.06 s / 1.03 s. Output of the + old and new binaries is identical before the IDX fix; after it, output is + byte-identical to stock for `--list-invocations` with and without `-q`, and + for `--list-boots`, on the file and the directory, in both languages. +- New tests: Rust `table_index_width_follows_widest_index`, Go + `TestTableIndexWidthFollowsWidestIndex`. +- `go test ./cmd/journalctl` and `cargo test -p journalctl` pass. +- `tests/parser-parity/check_v260_manifest.py`: OK. + `run_parser_parity.py`: 134/134 in each language. +- `tests/interoperability/run_journalctl_query_matrix.py --journalctl` + (v262 oracle): 3587/3587 PASS. +- Same-failure search: the eligibility OR is the only value-list prefilter in + either CLI (`grep -n eligibility`); both tables share the width helper. +- Real-use path: the CLIs were run on the synthetic fixtures above. +- Runtime purity: CLI-only changes; no new native code. + +Sensitive data gate: + +- Synthetic fixtures and identities only; fixtures and raw results stay in + ignored `.local/sow0122/`. + +Artifact maintenance gate: + +- AGENTS.md and project skills: no new rule. +- Specs: no spec states the prefilter or IDX width; the parity matrix spec + is unaffected (output now closer to stock). +- End-user docs: `docs/Journalctl-CLI.md` makes no claim about these paths; + no change. +- Status indexes: updated. ## Outcome -Pending. +Completed. `--list-invocations` is 40-120x faster in Rust and 8-22x in Go on +many-ID journals, faster than stock on directories in Rust, and its table now +matches stock. The remaining measured gaps are tracked by SOW-0166, SOW-0167 +and SOW-0168. ## Lessons Extracted -Pending activation. +A prefilter tuned for a sparse case needs a bound for the dense case; measure +both shapes before keeping it. ## Followup -Pending activation. +- SOW-0166: directory realtime seek cost. +- SOW-0167: Go reader eager entry-array load. +- SOW-0168: follow waiting/reopen cost and `Error: ` prefix parity. diff --git a/.agents/sow/pending/SOW-0166-20261101-directory-realtime-seek-cost.md b/.agents/sow/pending/SOW-0166-20261101-directory-realtime-seek-cost.md new file mode 100644 index 00000000..958f5c80 --- /dev/null +++ b/.agents/sow/pending/SOW-0166-20261101-directory-realtime-seek-cost.md @@ -0,0 +1,83 @@ +# SOW-0166 - Directory Realtime Seek Cost + +## Status + +Status: open + +Sub-state: measured gap from SOW-0122; not activated. + +## Requirements + +### Purpose + +Directory realtime seeks (`--since`, `SeekRealtimeUsec` on directory readers) +should cost about what stock does, independent of how many entries precede the +target. + +### User Request + +Follow-up of SOW-0122 measurement (performance contract in `AGENTS.md`). + +### Assistant Understanding + +Facts (SOW-0122, `.local/sow0122/report.md`, 300-file 1.5M-entry directory): + +- `--since/--until` 60 s window mid-directory: Rust 3.54 s, Go 3.46 s, stock + v262 0.21 s. `--since` at the last minute: Rust 17.3 s, Go 5.4 s, stock + 0.16 s (one-off run). Single-file seeks are at parity. + +Inferences (unverified): + +- Rust `rust/src/journal/src/directory.rs` `fill_candidate` (around lines + 375-392) may step and discard older entries file by file instead of seeking + each file by realtime or skipping files whose tail is older than the target. + The Go directory path is not traced yet. + +### Acceptance Criteria + +- Trace both languages, fix the per-file positioning to use journal-native + realtime seeks (entry-array bisection) and header head/tail bounds. +- Directory realtime seek cost at or near stock on the SOW-0122 fixture, with + the query matrix and directory matrix still passing. + +## Analysis + +Pending activation. + +## Pre-Implementation Gate + +Not started. + +## Implications And Decisions + +None yet. + +## Plan + +Pending activation. + +## Execution Log + +### 2026-11-01 + +- Created from SOW-0122 measurements. + +## Validation + +Pending. + +## Outcome + +Pending. + +## Lessons Extracted + +None yet. + +## Followup + +None yet. + +## Regression Log + +None. diff --git a/.agents/sow/pending/SOW-0167-20261101-go-reader-eager-entry-array.md b/.agents/sow/pending/SOW-0167-20261101-go-reader-eager-entry-array.md new file mode 100644 index 00000000..bb480a3e --- /dev/null +++ b/.agents/sow/pending/SOW-0167-20261101-go-reader-eager-entry-array.md @@ -0,0 +1,84 @@ +# SOW-0167 - Go Reader Eager Entry Array Load + +## Status + +Status: open + +Sub-state: measured gap from SOW-0122; not activated. + +## Requirements + +### Purpose + +Opening a journal file with the Go reader should not load the complete global +entry array, so small queries cost what they cost in Rust and stock. + +### User Request + +Follow-up of SOW-0122 measurement (performance contract in `AGENTS.md`). + +### Assistant Understanding + +Facts: + +- `go/journal/reader.go` `openFileWithPolicy` calls `loadEntryArray` on open + (SOW-0122 trace: `reader.go:287` -> `:705`); strace shows the whole file + mapped. Even `--header` costs about 0.27 s and 160 MB per 931 MB file, about + 800 MB on a 300-file directory. +- Go `-n 100`, `_SYSTEMD_INVOCATION_ID=` and `-r -n 1000` are 26-54x slower + than stock on the file; Rust has no such floor. + +Unknowns: + +- Which Go reader paths depend on the materialized entry list (step, + seek, recovery, live growth, explorer), and whether a lazy chain-walk + accessor (as Rust uses) preserves their contracts. + +### Acceptance Criteria + +- Go reader opens without materializing the entry array; positioning uses + entry-array chains and bisection like Rust and libsystemd. +- Live-growth, recovery and position-contract tests pass; Go small-query cost + near Rust on the SOW-0122 fixture; no regression on full scans. + +## Analysis + +Pending activation. Likely a design-level change; options will be presented. + +## Pre-Implementation Gate + +Not started. + +## Implications And Decisions + +None yet. + +## Plan + +Pending activation. + +## Execution Log + +### 2026-11-01 + +- Created from SOW-0122 measurements. + +## Validation + +Pending. + +## Outcome + +Pending. + +## Lessons Extracted + +None yet. + +## Followup + +None yet. + +## Regression Log + +None. diff --git a/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md new file mode 100644 index 00000000..abbb236d --- /dev/null +++ b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md @@ -0,0 +1,89 @@ +# SOW-0168 - Journalctl Follow Wait And Error Parity + +## Status + +Status: open + +Sub-state: measured gap from SOW-0122; needs a user decision before design. + +## Requirements + +### Purpose + +Make `journalctl --follow` in both CLIs cheap while idle and fast to deliver +new entries, and align remaining stderr wording with stock. + +### User Request + +Follow-up of SOW-0122 measurement. + +### Assistant Understanding + +Facts (SOW-0122, `.local/sow0122/report.md`): + +- Idle follow CPU: Rust 2.1% (file) / 6.2% (directory), Go 21% / 29.7%, + stock 0%. New-entry latency: Rust 0.24-0.71 s, Go 0.31-0.81 s, stock + 2-4 ms. +- Both CLIs poll and reopen the whole input each poll (Rust `main.rs` + follow loop, Go `main.go` follow loop); the initial tail scans the input + forward to keep the last N entries. Stock waits on inotify. +- Both CLIs prefix errors with `Error: `; stock prints the bare message (for + example `Using --list-invocations requires a unit...`). + +### Acceptance Criteria + +- Follow reuses open readers and positions instead of reopening; the initial + tail uses tail seeks. +- Event-driven waiting where the platform allows it, per the user decision; + portable polling stays the fallback. +- Error output wording matches stock where the parser-parity manifest covers + it. + +## Analysis + +Decision needed (user): + +- A. Linux inotify (via the platform C library, which the CLI-only native + code exception already allows) with polling elsewhere. +- B. Portable polling only, made cheap: reuse readers, check header + `n_entries`/`tail_entry_seqnum` per poll, back off while idle. +- Recommendation: B first (removes most cost on every platform, no new + native code), then A if latency must reach stock. + +## Pre-Implementation Gate + +Not started: user decision first. + +## Implications And Decisions + +Pending user decision. + +## Plan + +Pending. + +## Execution Log + +### 2026-11-01 + +- Created from SOW-0122 measurements. + +## Validation + +Pending. + +## Outcome + +Pending. + +## Lessons Extracted + +None yet. + +## Followup + +None yet. + +## Regression Log + +None. diff --git a/SOW-status.md b/SOW-status.md index 71b93354..740f2e7a 100644 --- a/SOW-status.md +++ b/SOW-status.md @@ -10,7 +10,8 @@ is `.agents/sow/SOW-status.md`; if summaries differ, the canonical ledger wins. - SOW-0156 - Fixed-Header Consistency Audit: paused; audit done, Rust parity gaps fixed, user decisions D1-D6 pending. -SOW-0045, SOW-0071, SOW-0123, SOW-0137 and SOW-0161 completed on 2026-11-01. +SOW-0045, SOW-0071, SOW-0122, SOW-0123, SOW-0137 and SOW-0161 completed on +2026-11-01. ## Pending @@ -22,11 +23,10 @@ SOW-0045, SOW-0071, SOW-0123, SOW-0137 and SOW-0161 completed on 2026-11-01. - `SOW-0125-20260625-netdata-filter-operators-gap-analysis.md` - open, parked by user decision on 2026-06-25. Preserves string/numeric operator support matrix and semantic decisions; not executable until explicitly resumed. -- `SOW-0122-20260621-journalctl-performance-followups.md` - open. Tracks - non-blocking SOW-0121 broader performance follow-up for - `--list-invocations`, portable `--follow` scalability, and large-fixture - performance evidence. It does not replace the mandatory no-regression gates - in the requested SDK and journalctl workstreams. +- SOW-0166 - Directory Realtime Seek Cost: open; from SOW-0122 measurement. +- SOW-0167 - Go Reader Eager Entry Array Load: open; from SOW-0122. +- SOW-0168 - Journalctl Follow Wait And Error Parity: open; needs a user + decision. - `SOW-0066-20260530-v1-release-and-registry-publication.md` - open. Publish the final stable Rust/Go `v1.0.0` SDK with Rust registry publication where applicable, Go module tags, and clean consumer install validation after compatibility, portability, real-corpus validation, parity closure, and release checks pass. - `SOW-0097-20260607-go-codacy-metric-debt-refactor.md` - open. Follow-up from the Codacy Rust/Go metrics audit for Go production file-size/ownership and duplication reduction. - `SOW-0098-20260607-rust-legacy-core-duplication-debt.md` - open. Follow-up from the Codacy Rust/Go metrics audit for real Rust `jf`/`journal-core` duplication reduction. diff --git a/go/cmd/journalctl/colors.go b/go/cmd/journalctl/colors.go index 15bf296d..f40dda86 100644 --- a/go/cmd/journalctl/colors.go +++ b/go/cmd/journalctl/colors.go @@ -3,6 +3,7 @@ package main import ( + "fmt" "os" "strings" ) @@ -110,15 +111,17 @@ func (c outputColors) boot(id string) string { return ansiHighlight + strings.TrimSuffix(text, "\n") + ansiNormal + "\n" } -func (c outputColors) tableHeader(id string) string { +// indexWidth is the IDX column width; stock sizes it to the widest index. +func (c outputColors) tableHeader(id string, indexWidth int) string { + idx := fmt.Sprintf("%-*s", indexWidth, "IDX") if c.level == 0 || !c.underline { - return "IDX " + id + strings.Repeat(" ", 33-len(id)) + "FIRST ENTRY LAST ENTRY\n" + return idx + " " + id + strings.Repeat(" ", 33-len(id)) + "FIRST ENTRY LAST ENTRY\n" } on := "" if c.underline { on = "\x1b[4m" } - fields := []string{"IDX", id + strings.Repeat(" ", 32-len(id)), "FIRST ENTRY ", "LAST ENTRY "} + fields := []string{idx, id + strings.Repeat(" ", 32-len(id)), "FIRST ENTRY ", "LAST ENTRY "} return on + strings.Join(fields, ansiNormal+on+" "+ansiNormal+on) + ansiNormal + "\n" } diff --git a/go/cmd/journalctl/invocation.go b/go/cmd/journalctl/invocation.go index 9a272d6a..c7be7bdb 100644 --- a/go/cmd/journalctl/invocation.go +++ b/go/cmd/journalctl/invocation.go @@ -135,6 +135,12 @@ func newInvocationWalker(input cliInput, flags *cliFlags, option string, older, return nil, err } w.done = len(w.eligibility) == 0 + // The OR prefilter costs one term per step. With many IDs that is + // slower than visiting the unit's rows, which is what native discovery + // does; rows without a valid ID are skipped either way. + if len(w.eligibility) > invocationEligibilityMaxTerms { + w.eligibility = nil + } } if w.done { ok = true @@ -199,6 +205,10 @@ func parseInvocationIDValue(value []byte) string { return hex.EncodeToString(id[:]) } +// invocationEligibilityMaxTerms bounds the valid-ID prefilter; paired with +// Rust ELIGIBILITY_MAX_TERMS. +const invocationEligibilityMaxTerms = 32 + func invocationEligibility(j cliJournal) ([][]byte, error) { var matches [][]byte for _, field := range invocationIDFields { diff --git a/go/cmd/journalctl/main.go b/go/cmd/journalctl/main.go index fad58a73..ddb7c941 100644 --- a/go/cmd/journalctl/main.go +++ b/go/cmd/journalctl/main.go @@ -1982,18 +1982,13 @@ func runListBoots(input cliInput, flags *cliFlags, stdout io.Writer) error { } return writeTableJSON(stdout, *flags.output, rows) } - if !*flags.quietFlag { - fmt.Fprint(stdout, detectOutputColors().tableHeader("BOOT ID")) + indexes := make([]int, 0, len(display)) + for _, boot := range display { + indexes = append(indexes, boot.index) } - idxWidth := 1 - if *flags.quietFlag { - for _, boot := range display { - if width := len(fmt.Sprintf("%d", boot.index)); width > idxWidth { - idxWidth = width - } - } - } else { - idxWidth = 3 + idxWidth := tableIndexWidth(indexes, *flags.quietFlag) + if !*flags.quietFlag { + fmt.Fprint(stdout, detectOutputColors().tableHeader("BOOT ID", idxWidth)) } for _, boot := range display { fmt.Fprintf( @@ -3646,17 +3641,13 @@ func runListInvocations(input cliInput, flags *cliFlags, stdout io.Writer) error } return writeTableJSON(stdout, *flags.output, rows) } - if !*flags.quietFlag { - fmt.Fprint(stdout, detectOutputColors().tableHeader("INVOCATION ID")) + indexes := make([]int, 0, len(display)) + for _, row := range display { + indexes = append(indexes, row.index) } - idxWidth := 1 - if *flags.quietFlag { - for _, row := range display { - idxText := fmt.Sprintf("%d", row.index) - if len(idxText) > idxWidth { - idxWidth = len(idxText) - } - } + idxWidth := tableIndexWidth(indexes, *flags.quietFlag) + if !*flags.quietFlag { + fmt.Fprint(stdout, detectOutputColors().tableHeader("INVOCATION ID", idxWidth)) } for _, row := range display { info := row.info @@ -3673,7 +3664,8 @@ func runListInvocations(input cliInput, flags *cliFlags, stdout io.Writer) error } else { fmt.Fprintf( stdout, - "%3d %-32s %s %s\n", + "%*d %-32s %s %s\n", + idxWidth, row.index, info.id, formatHeaderTimestamp(info.firstUsec), @@ -4124,3 +4116,16 @@ func consumeHex(s string, start int) (int, bool) { func isHex(b byte) bool { return ('0' <= b && b <= '9') || ('a' <= b && b <= 'f') || ('A' <= b && b <= 'F') } + +// tableIndexWidth follows stock table formatting: IDX is as wide as the widest +// index, and at least as wide as its header when the header is shown. +func tableIndexWidth(indexes []int, quiet bool) int { + width := 1 + for _, index := range indexes { + width = max(width, len(strconv.Itoa(index))) + } + if !quiet { + width = max(width, 3) + } + return width +} diff --git a/go/cmd/journalctl/table_json_test.go b/go/cmd/journalctl/table_json_test.go index 1689ef35..d021987a 100644 --- a/go/cmd/journalctl/table_json_test.go +++ b/go/cmd/journalctl/table_json_test.go @@ -18,3 +18,25 @@ func TestTablePrettyJSONNativeSpacing(t *testing.T) { t.Fatalf("got %q, want %q", out.String(), want) } } + +// Paired with Rust table_index_width_follows_widest_index. +func TestTableIndexWidthFollowsWidestIndex(t *testing.T) { + for _, tc := range []struct { + indexes []int + quiet bool + want int + }{ + {[]int{-5, 0}, false, 3}, + {[]int{-5, 0}, true, 2}, + {[]int{-220, 0}, false, 4}, + {nil, true, 1}, + } { + if got := tableIndexWidth(tc.indexes, tc.quiet); got != tc.want { + t.Fatalf("tableIndexWidth(%v, %v) = %d, want %d", tc.indexes, tc.quiet, got, tc.want) + } + } + header := outputColors{}.tableHeader("INVOCATION ID", 4) + if want := "IDX INVOCATION ID "; header[:len(want)] != want { + t.Fatalf("header = %q", header) + } +} diff --git a/rust/src/cmd/journalctl/colors.rs b/rust/src/cmd/journalctl/colors.rs index 5000e672..76a6a8d1 100644 --- a/rust/src/cmd/journalctl/colors.rs +++ b/rust/src/cmd/journalctl/colors.rs @@ -108,13 +108,17 @@ impl Colors { } } - pub fn table_header(self, id: &str) -> String { + /// `index_width` is the IDX column width; stock sizes it to the widest index. + pub fn table_header(self, id: &str, index_width: usize) -> String { if !self.enabled() || !self.underline { - return format!("IDX {id:<32} FIRST ENTRY LAST ENTRY\n"); + return format!( + "{: Option<[u8; 16]> { (id != [0; 16]).then_some(id) } +/// Bounds the valid-ID prefilter; paired with Go invocationEligibilityMaxTerms. +const ELIGIBILITY_MAX_TERMS: usize = 32; + fn eligibility(journal: &mut SdJournal) -> Result>> { let mut matches = Vec::new(); for field in INVOCATION_ID_FIELDS { @@ -157,6 +160,14 @@ impl Walker { Vec::new() }; let done = option.is_some() && eligibility.is_empty(); + // The OR prefilter costs one term per step. With many IDs that is + // slower than visiting the unit's rows, which is what native discovery + // does; rows without a valid ID are skipped either way. + let eligibility = if eligibility.len() > ELIGIBILITY_MAX_TERMS { + Vec::new() + } else { + eligibility + }; let mut walker = Self { scan, bounds: if done { diff --git a/rust/src/cmd/journalctl/main.rs b/rust/src/cmd/journalctl/main.rs index fefed544..254821ff 100644 --- a/rust/src/cmd/journalctl/main.rs +++ b/rust/src/cmd/journalctl/main.rs @@ -1956,17 +1956,13 @@ fn run_list_boots(input: &CliInput, args: &Args) -> Result<()> { })).collect(); return print_json_table(args.output, &rows); } + let index_width = table_index_width(rows.iter().map(|boot| boot.index), args.quiet); if !args.quiet { - print!("{}", colors::Colors::detect().table_header("BOOT ID")); + print!( + "{}", + colors::Colors::detect().table_header("BOOT ID", index_width) + ); } - let index_width = if args.quiet { - rows.iter() - .map(|boot| boot.index.to_string().len()) - .max() - .unwrap_or(1) - } else { - 3 - }; for boot in &rows { println!( "{:>width$} {} {} {}", @@ -1996,17 +1992,13 @@ fn run_list_invocations(input: &CliInput, args: &Args) -> Result<()> { })).collect(); return print_json_table(args.output, &rows); } + let index_width = table_index_width(rows.iter().map(|(index, _)| *index), args.quiet); if !args.quiet { - print!("{}", colors::Colors::detect().table_header("INVOCATION ID")); + print!( + "{}", + colors::Colors::detect().table_header("INVOCATION ID", index_width) + ); } - let quiet_index_width = if args.quiet { - rows.iter() - .map(|(row_index, _)| row_index.to_string().len()) - .max() - .unwrap_or(1) - } else { - 1 - }; for (row_index, entry) in rows { if args.quiet { println!( @@ -2015,21 +2007,32 @@ fn run_list_invocations(input: &CliInput, args: &Args) -> Result<()> { entry.id, output::format_header_timestamp(entry.first_usec)?, output::format_header_timestamp(entry.last_usec)?, - width = quiet_index_width, + width = index_width, ); } else { println!( - "{:>3} {:<32} {} {}", + "{:>width$} {:<32} {} {}", row_index, entry.id, output::format_header_timestamp(entry.first_usec)?, output::format_header_timestamp(entry.last_usec)?, + width = index_width, ); } } Ok(()) } +/// Stock table formatting sizes IDX to the widest index, and to at least the +/// header when it is shown. +fn table_index_width(indexes: impl Iterator, quiet: bool) -> usize { + let widest = indexes + .map(|index| index.to_string().len()) + .max() + .unwrap_or(1); + if quiet { widest } else { widest.max(3) } +} + fn header_state_name(state: u8) -> &'static str { match state { 0 => "OFFLINE", @@ -4308,6 +4311,17 @@ fn run_follow( #[cfg(test)] mod tests { use super::*; + + // Paired with Go TestTableIndexWidthFollowsWidestIndex. + #[test] + fn table_index_width_follows_widest_index() { + assert_eq!(table_index_width([-5, 0].into_iter(), false), 3); + assert_eq!(table_index_width([-5, 0].into_iter(), true), 2); + assert_eq!(table_index_width([-220, 0].into_iter(), false), 4); + assert_eq!(table_index_width(std::iter::empty::(), true), 1); + let header = colors::Colors::default().table_header("INVOCATION ID", 4); + assert!(header.starts_with("IDX INVOCATION ID "), "{header:?}"); + } use journal_core::file::{JournalFile, JournalFileOptions, JournalWriter, MmapMut}; use journal_core::repository::File as RepoFile; use journal_core::seal::SealOptions; From 1c6105dd74c68d81c5209843097e472d47bec138 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sun, 1 Nov 2026 21:13:37 +0000 Subject: [PATCH 2/8] Drop the trailing space inside a code span in SOW-0168 --- ...SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md index abbb236d..b639d6ee 100644 --- a/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md +++ b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md @@ -27,7 +27,7 @@ Facts (SOW-0122, `.local/sow0122/report.md`): - Both CLIs poll and reopen the whole input each poll (Rust `main.rs` follow loop, Go `main.go` follow loop); the initial tail scans the input forward to keep the last N entries. Stock waits on inotify. -- Both CLIs prefix errors with `Error: `; stock prints the bare message (for +- Both CLIs prefix errors with an `Error:` prefix; stock prints the bare message (for example `Using --list-invocations requires a unit...`). ### Acceptance Criteria From b258c6017435fe2dd0d7febcf6191e8ffdb75b2d Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sun, 1 Nov 2026 21:13:54 +0000 Subject: [PATCH 3/8] Remove trailing spaces inside Error: code spans in SOW-0122 and SOW-0168 --- .../SOW-0122-20260621-journalctl-performance-followups.md | 4 ++-- ...W-0168-20261101-journalctl-follow-wait-and-error-parity.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md b/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md index f105a8a6..5ccf3936 100644 --- a/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md +++ b/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md @@ -194,7 +194,7 @@ of 5x or more are claimed). Full table and commands: - Output: default output byte-identical across all three except `--list-invocations`, where stock right-aligns a wider IDX column once indexes reach three digits plus sign. Fixed here for invocations and boots. - The `Error: ` stderr prefix difference is tracked by SOW-0168. + The `Error:` stderr prefix difference is tracked by SOW-0168. Decisions (agent-chosen, pending user review): @@ -319,4 +319,4 @@ both shapes before keeping it. - SOW-0166: directory realtime seek cost. - SOW-0167: Go reader eager entry-array load. -- SOW-0168: follow waiting/reopen cost and `Error: ` prefix parity. +- SOW-0168: follow waiting/reopen cost and `Error:` prefix parity. diff --git a/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md index b639d6ee..59e01ab8 100644 --- a/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md +++ b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md @@ -27,7 +27,7 @@ Facts (SOW-0122, `.local/sow0122/report.md`): - Both CLIs poll and reopen the whole input each poll (Rust `main.rs` follow loop, Go `main.go` follow loop); the initial tail scans the input forward to keep the last N entries. Stock waits on inotify. -- Both CLIs prefix errors with an `Error:` prefix; stock prints the bare message (for +- Both CLIs prefix errors with `Error:`; stock prints the bare message (for example `Using --list-invocations requires a unit...`). ### Acceptance Criteria From cad658a3fb2388cbf55dd0eebfd3a88761ad6572 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sun, 1 Nov 2026 21:16:36 +0000 Subject: [PATCH 4/8] Record SOW-0122 reproduction, aggregate results and full validation; tighten SOW-0166 gate and SOW-0168 policy note --- ...260621-journalctl-performance-followups.md | 129 ++++++++++++++++-- ...6-20261101-directory-realtime-seek-cost.md | 8 +- ...journalctl-follow-wait-and-error-parity.md | 5 +- 3 files changed, 125 insertions(+), 17 deletions(-) diff --git a/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md b/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md index 5ccf3936..6b37cc90 100644 --- a/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md +++ b/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md @@ -167,7 +167,9 @@ Open-source reference evidence: Open decisions: -- Pending activation priority and target performance envelope. +- Resolved 2026-11-01 (user backlog order): activated as item 8. Envelope + used: claim only gaps of 5x or more against stock, given host load. The fix + decisions below are agent-chosen and pending user review. ## Implications And Decisions @@ -175,8 +177,48 @@ Measurement (2026-11-01, release builds at commit 96eecf5, stock v262 built locally, synthetic fixtures from the Go writer: one 931 MB file and a 300-file 1.8 GB directory, 1.5M entries, 6 boots, 45 active units with 221 invocations each; 3 runs, median; host load 16-66 on 112 CPUs, so only gaps -of 5x or more are claimed). Full table and commands: -`.local/sow0122/report.md`, `.local/sow0122/run-all.sh` (local only). +of 5x or more are claimed). + +Reproduction (the scripts stay in ignored `.local/sow0122/`; the procedure is +recorded here so the numbers can be re-measured without them): + +1. Build release CLIs: `cargo build --release -p journalctl` and + `CGO_ENABLED=1 go build ./cmd/journalctl`, plus stock v262 journalctl. +2. Generate fixtures with a small Go program using the in-repo Go writer: + 1.5M entries over 6 boots; unit rows rotate over 50 synthetic services with + about 200 invocation IDs each, plus periodic kernel rows with no unit + (observed in output: 45 units with 221 invocations listed per unit); once + as one file and once as 300 files in one directory. +3. Run each command below with `--file`/`--directory` on each fixture, 3 runs, + median wall time, CPU time and max RSS (`/usr/bin/time -v`), and compare + output hashes against stock. +4. `--follow`: start each CLI on the fixture, append bursts with the writer + after a 15 s warm-up, record idle CPU over 30 s and append-to-print latency. + +Results before the fix (wall s / CPU s / max RSS MB): + +| command | src | rust wall/CPU/RSS | go wall/CPU/RSS | stock wall/CPU/RSS | rust/stock | go/stock | outputs | +|---|---|---|---|---|---|---|---| +| list-inv -u | file | 29.35 / 29.34 / 299 | 12.13 / 12.27 / 227 | 0.05 / 0.05 / 101 | 587.0x | 242.6x | DIFF (222 lines) | +| list-inv -u | dir | 11.88 / 11.87 / 3986 | 16.05 / 23.05 / 4226 | 0.96 / 0.96 / 1488 | 12.4x | 16.7x | DIFF (222 lines) | +| -r -n 1000 | file | 0.01 / 0.00 / 11 | 0.26 / 0.30 / 161 | 0.01 / 0.01 / 6 | 1.0x | 26.0x | match (1000 lines) | +| -r -n 1000 | dir | 0.02 / 0.02 / 12 | 0.40 / 0.45 / 806 | 0.07 / 0.07 / 142 | 0.3x | 5.7x | match (1000 lines) | +| -r (full) | file | 8.64 / 8.63 / 363 | 23.78 / 28.05 / 160 | 16.88 / 16.74 / 351 | 0.5x | 1.4x | match (1500005 lines) | +| -r (full) | dir | 10.48 / 10.47 / 766 | 23.90 / 32.91 / 821 | 45.17 / 45.15 / 780 | 0.2x | 0.5x | match (1500005 lines) | +| forward (full) | file | 8.18 / 8.17 / 220 | 26.80 / 33.02 / 161 | 15.90 / 15.89 / 251 | 0.5x | 1.7x | match (1500005 lines) | +| forward (full) | dir | 10.80 / 10.79 / 766 | 21.75 / 26.55 / 829 | 46.58 / 46.39 / 778 | 0.2x | 0.5x | match (1500005 lines) | +| --since/--until 60s | file | 0.03 / 0.03 / 11 | 0.34 / 0.38 / 161 | 0.06 / 0.06 / 9 | 0.5x | 5.7x | match (6001 lines) | +| --since/--until 60s | dir | 3.54 / 3.53 / 157 | 3.46 / 3.75 / 818 | 0.21 / 0.21 / 101 | 16.9x | 16.5x | match (6001 lines) | +| -u unit | file | 0.30 / 0.29 / 373 | 2.20 / 2.35 / 161 | 0.51 / 0.51 / 245 | 0.6x | 4.3x | match (30005 lines) | +| -u unit | dir | 0.34 / 0.33 / 821 | 0.91 / 1.04 / 942 | 1.11 / 1.10 / 788 | 0.3x | 0.8x | match (30005 lines) | +| -b -5 | file | 1.39 / 1.39 / 64 | 3.57 / 4.31 / 156 | 2.78 / 2.77 / 128 | 0.5x | 1.3x | match (250000 lines) | +| -b -5 | dir | 1.76 / 1.75 / 267 | 3.60 / 4.54 / 930 | 7.12 / 7.12 / 369 | 0.2x | 0.5x | match (250000 lines) | +| _SYSTEMD_INVOCATION_ID= | file | 0.00 / 0.00 / 7 | 0.27 / 0.30 / 161 | 0.00 / 0.00 / 8 | 0.0x | 54.0x | match (137 lines) | +| _SYSTEMD_INVOCATION_ID= | dir | 0.01 / 0.01 / 28 | 0.34 / 0.33 / 828 | 0.03 / 0.02 / 84 | 0.3x | 11.3x | match (137 lines) | +| -n 100 | file | 0.00 / 0.00 / 5 | 0.27 / 0.29 / 161 | 0.00 / 0.00 / 6 | 0.0x | 54.0x | match (100 lines) | +| -n 100 | dir | 0.01 / 0.01 / 6 | 0.33 / 0.30 / 808 | 0.04 / 0.04 / 142 | 0.2x | 8.2x | match (100 lines) | + +`DIFF` on `list-inv -u` is the IDX column width only (fixed here). - Rust is at or faster than stock on full forward/reverse output, `-u`, `-b -5`, field matches, `-n` and single-file `--since` (full output is 2x @@ -271,12 +313,20 @@ Failure handling: ## Validation +Acceptance criteria evidence: + +- Large-input envelope measured for both CLIs against stock v262 (table in + Implications And Decisions). - `--list-invocations -u` after the fix (same fixtures, single runs, load about 36): Rust 0.23 s file / 0.67 s directory (was 27.7 / 13.0); Go 0.66 s / 1.81 s (was 14.5 / 15.4); stock 0.06 s / 1.03 s. Output of the old and new binaries is identical before the IDX fix; after it, output is byte-identical to stock for `--list-invocations` with and without `-q`, and for `--list-boots`, on the file and the directory, in both languages. +- Each remaining gap of 5x or more has a pending SOW (Follow-up mapping). + +Tests or equivalent validation: + - New tests: Rust `table_index_width_follows_widest_index`, Go `TestTableIndexWidthFollowsWidestIndex`. - `go test ./cmd/journalctl` and `cargo test -p journalctl` pass. @@ -284,24 +334,77 @@ Failure handling: `run_parser_parity.py`: 134/134 in each language. - `tests/interoperability/run_journalctl_query_matrix.py --journalctl` (v262 oracle): 3587/3587 PASS. -- Same-failure search: the eligibility OR is the only value-list prefilter in - either CLI (`grep -n eligibility`); both tables share the width helper. -- Real-use path: the CLIs were run on the synthetic fixtures above. -- Runtime purity: CLI-only changes; no new native code. + +Real-use evidence: + +- Both release CLIs were run on the synthetic 931 MB file and the 300-file + directory with the commands in the table. + +Reviewer findings (self-review of the whole SOW diff; tests are the gate, no +external reviewers): + +1. The 32-ID bound is a heuristic, not a measured crossover. Journals with + 33 to a few hundred IDs and many ID-less rows were not measured. + Disposition: accepted and recorded as agent-chosen. Above the bound the + walker visits the unit's rows as stock does and skips ID-less rows, so + output is unchanged and cost is bounded by row count, not ID count. +2. IDX padding also changes `--list-boots` once indexes reach three digits + plus sign. Disposition: intended; verified byte-identical to stock. +3. Rust and Go bounds are separate constants. Disposition: each cites the + other in its comment; the parity tests cover the table width, not the + bound, which changes cost only. + +Same-failure scan: + +- The eligibility OR is the only value-list prefilter in either CLI + (`grep -n eligibility`); both tables share the width helper. Sensitive data gate: - Synthetic fixtures and identities only; fixtures and raw results stay in - ignored `.local/sow0122/`. + ignored `.local/sow0122/`. This SOW records aggregate numbers only. Artifact maintenance gate: -- AGENTS.md and project skills: no new rule. +- AGENTS.md: no new rule; the change follows the existing performance contract. +- Runtime project skills: `project-journal-compatibility` names the query + matrix and parser-parity gates already used here; no new rule. - Specs: no spec states the prefilter or IDX width; the parity matrix spec - is unaffected (output now closer to stock). -- End-user docs: `docs/Journalctl-CLI.md` makes no claim about these paths; - no change. -- Status indexes: updated. + is unaffected (output is now closer to stock). +- End-user/operator docs: `docs/Journalctl-CLI.md` makes no claim about + these paths; no change. +- End-user/operator skills: none exist. +- SOW lifecycle: `Status: completed` and the move to `done/` are in the same + commit as the code; three follow-up SOWs were created. +- SOW-status.md: both the canonical ledger and the root index are updated. + +Specs update: + +- None needed (see the artifact maintenance gate). + +Project skills update: + +- None needed. + +End-user/operator docs update: + +- None needed. + +End-user/operator skills update: + +- None exist. + +Lessons: + +- See Lessons Extracted. + +Follow-up mapping: + +- Directory `--since` seek gap: tracked by SOW-0166. +- Go eager entry-array load: tracked by SOW-0167. +- `--follow` polling/reopen cost and the `Error:` prefix: tracked by SOW-0168 + (needs a user decision). +- Runtime purity: CLI-only changes; no new native code. Nothing else remains. ## Outcome diff --git a/.agents/sow/pending/SOW-0166-20261101-directory-realtime-seek-cost.md b/.agents/sow/pending/SOW-0166-20261101-directory-realtime-seek-cost.md index 958f5c80..ef149f98 100644 --- a/.agents/sow/pending/SOW-0166-20261101-directory-realtime-seek-cost.md +++ b/.agents/sow/pending/SOW-0166-20261101-directory-realtime-seek-cost.md @@ -37,8 +37,12 @@ Inferences (unverified): - Trace both languages, fix the per-file positioning to use journal-native realtime seeks (entry-array bisection) and header head/tail bounds. -- Directory realtime seek cost at or near stock on the SOW-0122 fixture, with - the query matrix and directory matrix still passing. +- On the SOW-0122 300-file directory fixture and procedure (SOW-0122 + Implications And Decisions: 3 runs, median wall time), the 60 s + `--since/--until` window and the last-minute `--since` each run in at most + 2x stock's median, or under 0.5 s when stock is below 0.25 s, in both + languages (proposed gate, agent-chosen; confirm at activation). The query + matrix and directory matrix still pass. ## Analysis diff --git a/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md index 59e01ab8..5b00e4a0 100644 --- a/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md +++ b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md @@ -43,8 +43,9 @@ Facts (SOW-0122, `.local/sow0122/report.md`): Decision needed (user): -- A. Linux inotify (via the platform C library, which the CLI-only native - code exception already allows) with polling elsewhere. +- A. Linux inotify with polling elsewhere. This needs an explicit policy + decision: the CLI-only native code exception names PCRE2, Unix globbing, + thread-local locale and terminal queries, not file-change notification. - B. Portable polling only, made cheap: reuse readers, check header `n_entries`/`tail_entry_seqnum` per poll, back off while idle. - Recommendation: B first (removes most cost on every platform, no new From 2f11219ef62265917791d15ebbe6a77589873b80 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sun, 1 Nov 2026 21:20:00 +0000 Subject: [PATCH 5/8] List SOW-0166, SOW-0167 and SOW-0168 in the backlog order --- .agents/sow/SOW-status.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.agents/sow/SOW-status.md b/.agents/sow/SOW-status.md index 06f2f378..8c8e64cc 100644 --- a/.agents/sow/SOW-status.md +++ b/.agents/sow/SOW-status.md @@ -24,6 +24,9 @@ order; the user may reorder it. 13. SOW-0066 - v1.0.0 API stability decision. 14. SOW-0169 - Pin the stock journalctl oracle in interoperability matrices (needs a decision at activation). +15. SOW-0166 - Directory realtime seek cost. +16. SOW-0167 - Go reader eager entry-array load. +17. SOW-0168 - journalctl follow wait and error parity (needs a user decision). ## Current From e86de845d499e61151c43c4b10ec7bbd5a5fe8ed Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Tue, 3 Nov 2026 07:14:03 +0000 Subject: [PATCH 6/8] Record the user decisions for SOW-0122 and SOW-0168 --- ...0260621-journalctl-performance-followups.md | 8 ++++---- ...-journalctl-follow-wait-and-error-parity.md | 18 ++++++++++++------ 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md b/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md index 6b37cc90..9dd4ce43 100644 --- a/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md +++ b/.agents/sow/done/SOW-0122-20260621-journalctl-performance-followups.md @@ -7,7 +7,7 @@ Status: completed Sub-state: completed 2026-11-01 in the autonomous backlog run. Measured the envelope, fixed the `--list-invocations` cost and IDX alignment parity in both CLIs; the three other measured gaps are tracked by SOW-0166, SOW-0167 and -SOW-0168. Scope choices are agent-chosen, pending user review (PR body). +SOW-0168. Scope choices approved by the user 2026-11-03. ## Requirements @@ -169,7 +169,7 @@ Open decisions: - Resolved 2026-11-01 (user backlog order): activated as item 8. Envelope used: claim only gaps of 5x or more against stock, given host load. The fix - decisions below are agent-chosen and pending user review. + decisions below were approved by the user on 2026-11-03. ## Implications And Decisions @@ -238,7 +238,7 @@ Results before the fix (wall s / CPU s / max RSS MB): indexes reach three digits plus sign. Fixed here for invocations and boots. The `Error:` stderr prefix difference is tracked by SOW-0168. -Decisions (agent-chosen, pending user review): +Decisions (agent-chosen 2026-11-01, approved by the user 2026-11-03): 1. Root cause of gap 1: SOW-0121 added a valid-ID eligibility prefilter (an OR of every invocation ID value) to keep sparse-ID journals cheap. Each @@ -403,7 +403,7 @@ Follow-up mapping: - Directory `--since` seek gap: tracked by SOW-0166. - Go eager entry-array load: tracked by SOW-0167. - `--follow` polling/reopen cost and the `Error:` prefix: tracked by SOW-0168 - (needs a user decision). + (decided 2026-11-03: cheap portable polling first). - Runtime purity: CLI-only changes; no new native code. Nothing else remains. ## Outcome diff --git a/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md index 5b00e4a0..4bb17e99 100644 --- a/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md +++ b/.agents/sow/pending/SOW-0168-20261101-journalctl-follow-wait-and-error-parity.md @@ -4,7 +4,8 @@ Status: open -Sub-state: measured gap from SOW-0122; needs a user decision before design. +Sub-state: measured gap from SOW-0122; decided 2026-11-03 (B: cheap portable +polling first). Not activated. ## Requirements @@ -34,14 +35,14 @@ Facts (SOW-0122, `.local/sow0122/report.md`): - Follow reuses open readers and positions instead of reopening; the initial tail uses tail seeks. -- Event-driven waiting where the platform allows it, per the user decision; - portable polling stays the fallback. +- Idle polls check only header counters and back off, with no reopen + (decision 1); event-driven waiting is out of scope. - Error output wording matches stock where the parser-parity manifest covers it. ## Analysis -Decision needed (user): +Decision (user, 2026-11-03: B): - A. Linux inotify with polling elsewhere. This needs an explicit policy decision: the CLI-only native code exception names PCRE2, Unix globbing, @@ -53,11 +54,16 @@ Decision needed (user): ## Pre-Implementation Gate -Not started: user decision first. +Not started; the wait design is decided (B). Activation follows the backlog +order. ## Implications And Decisions -Pending user decision. +1. 2026-11-03 (user): B. Both CLIs make portable polling cheap: reuse open + readers and positions, check header `n_entries`/`tail_entry_seqnum` per + poll, and back off while idle. Linux inotify (A) is not approved; it would + need a separate policy decision on the CLI-only native code exception if + measured latency after B must still reach stock. ## Plan From 1dfff84889299d09af34940635549cab7a97c115 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Tue, 3 Nov 2026 10:21:29 +0000 Subject: [PATCH 7/8] Apply 2026-11-03 backlog decisions to the SOW ledger --- .agents/sow/SOW-status.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/sow/SOW-status.md b/.agents/sow/SOW-status.md index 745655d0..e30a5758 100644 --- a/.agents/sow/SOW-status.md +++ b/.agents/sow/SOW-status.md @@ -34,7 +34,7 @@ order; the user may reorder it. cheap polling first). 18. SOW-0170 - Rust rewrite of Netdata systemd-journal.plugin (tracked; not now, by user decision 2026-11-03). -16. SOW-0171 - Continue sealing when reopening a sealed file (tracked; SOW-0156 +19. SOW-0171 - Continue sealing when reopening a sealed file (tracked; SOW-0156 decision N1, 2026-11-03). ## Current From f9399a6b356a7ac8f26353475f44c16e3da3b156 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Wed, 4 Nov 2026 22:05:04 +0000 Subject: [PATCH 8/8] Restore SOW ledger entries lost in the stack merge --- .agents/sow/SOW-status.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.agents/sow/SOW-status.md b/.agents/sow/SOW-status.md index 70302f07..f80dbf1e 100644 --- a/.agents/sow/SOW-status.md +++ b/.agents/sow/SOW-status.md @@ -77,6 +77,9 @@ None. ## Recently Closed Or Completed +- SOW-0160 - Go XZ Reader Dictionary Allocation: completed 2026-11-04. The Go + writer declares a 256 KiB XZ dictionary like Rust; Go and Rust reads of new + Go-written XZ files cost 5-10x less CPU. Stock journalctl is unaffected. - SOW-0156 - Fixed-Header Consistency Audit: completed 2026-11-03. Both SDKs apply systemd v260.1's writable-open header rules and timestamp ranges, keep