Skip to content

Parse files in parallel on the read pool - #196

Merged
bentsherman merged 1 commit into
mainfrom
parallel-parsing
Oct 28, 2026
Merged

bentsherman merged 1 commit into
mainfrom
parallel-parsing

Conversation

@bentsherman

@bentsherman bentsherman commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

The engine rewrite (#194) moved parsing onto the engine thread, because nf-lang writes to Groovy's shared ClassHelper nodes when it parses a fully qualified type such as java.lang.String. That made initial indexing slower.

That write only happens for fully qualified names of a few core types, which are rare in practice and discouraged, and it is being fixed in nf-lang. This PR restores parallel parsing: each read-pool task now reads the file and then parses it into its own SourceUnit. Registering the unit with the compiler, include discovery, and all analysis stay on the engine thread.

  • Builder.parseAll parses in the read-pool task. A parse exception is reported as an internal error for that file, as before.
  • Docs: the thread table and invariants in docs/internals.md, and the "Work off the engine thread" section of the core ADR.

Benchmark

make bench-compare against main on nf-core/rnaseq (3 alternating rounds, 20 iterations per scenario):

scenario main this PR delta verdict
workspace indexed 4763 ms 3925 ms -17.6% faster (p < 0.001)
startup: initialize 426 ms 414 ms -2.7% noise
didOpen → diagnostics 503 ms 504 ms +0.2% noise
didChange → diagnostics 791 ms 787 ms -0.4% noise
peak RSS 708 MB 823 MB +16.2% noise (p = 0.08)

Requests (hover, completion, references, etc.) are unaffected. Peak memory is likely higher because several files are parsed at once during indexing, but it isn't significant over 3 rounds.

@bentsherman
bentsherman merged commit 8bd08d5 into main Oct 28, 2026
7 checks passed
@bentsherman
bentsherman deleted the parallel-parsing branch October 28, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant