Repository navigation
test(common): run 10 instead of 50 validations per thread in TestBootstrapIndex#testBootstrapIndexConcurrent - #20358
Closed
databird07 wants to merge 1 commit into
Closed
databird07 wants to merge 1 commit into
databird07 wants to merge 1 commit into
Conversation
…strapIndex#testBootstrapIndexConcurrent testBootstrapIndexConcurrent starts 20 threads that each open their own HFileBootstrapIndex reader over the same index files and validate the full bootstrap mapping 50 times (1,000 full validations). It took ~85s of the "UT - hudi-hadoop-common" GitHub step, almost all of it repeating the same reads after the readers were already open. Change numRequestsPerThread from 50 to 10. The test still runs 20 threads concurrently, each with an independent reader (200 full validations), so it still exercises concurrent reader creation and lookups against the same files. Sharing one reader across threads was considered and rejected: HFileBootstrapIndexReader only synchronizes lazy reader creation, while getAllKeys / getSourceFileMappingFor* move a stateful HFile cursor without locking, and production code opens one reader per use. Measured locally (JDK 17, 4 pinned CPUs, JaCoCo on), 20 runs each: - before: 20/20 passed (4 tests), median 50.2s, p90 52.6s - after: 20/20 passed (4 tests), median 14.5s, p90 15.3s - JaCoCo across all hudi modules: 3,128 lines / 489 branches covered before and after, no line lost or gained. Tradeoff: fewer iterations per run lowers the per-run chance of hitting a rare race in concurrent reader opening.
hudi-agent
reviewed
Oct 16, 2026
hudi-agent
left a comment
hudi-agent
left a comment
Contributor
There was a problem hiding this comment.
Thanks for trimming this! This test-only change drops the per-thread validation loop in testBootstrapIndexConcurrent from 50 to 10 iterations and keeps all 20 concurrent threads, each with its own reader. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.
cc @yihua
Contributor
Author
|
Combined into #20364 so the total impact can be reviewed in one place; this commit is included there unchanged. Closing. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #20358 +/- ##
============================================
- Coverage 80.73% 78.11% -2.63%
+ Complexity 35201 34015 -1186
============================================
Files 2553 2553
Lines 143629 143629
Branches 17504 17504
============================================
- Hits 115964 112192 -3772
- Misses 19703 23258 +3555
- Partials 7962 8179 +217
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
testBootstrapIndexConcurrenttook ~85s in theUT - hudi-hadoop-commonGitHub step. Almost all of that time went to repeating the same full-index validation 50 times per thread after the readers were already open.Summary and Changelog
numRequestsPerThread50 -> 10. Still 20 concurrent threads, each with its ownHFileBootstrapIndexreader (200 full validations instead of 1,000).Impact
Test-only. Local median 50.2s -> 14.5s over 20 runs each, 20/20 passing. JaCoCo coverage is identical (3,128 lines / 489 branches). Measured locally on JDK 17 with 4 pinned CPUs and JaCoCo on; full numbers are in the commit message.
Risk Level
low. Fewer iterations lowers the per-run chance of hitting a rare race in concurrent reader opening.
Documentation Update
none
Contributor's checklist