Repository navigation
Parse files in parallel on the read pool - #196
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The engine rewrite (#194) moved parsing onto the engine thread, because nf-lang writes to Groovy's shared
ClassHelpernodes when it parses a fully qualified type such asjava.lang.String. That made initial indexing slower.That write only happens for fully qualified names of a few core types, which are rare in practice and discouraged, and it is being fixed in nf-lang. This PR restores parallel parsing: each read-pool task now reads the file and then parses it into its own
SourceUnit. Registering the unit with the compiler, include discovery, and all analysis stay on the engine thread.Builder.parseAllparses in the read-pool task. A parse exception is reported as an internal error for that file, as before.docs/internals.md, and the "Work off the engine thread" section of the core ADR.Benchmark
make bench-compareagainstmainon nf-core/rnaseq (3 alternating rounds, 20 iterations per scenario):Requests (hover, completion, references, etc.) are unaffected. Peak memory is likely higher because several files are parsed at once during indexing, but it isn't significant over 3 rounds.