Repository navigation
[DNM] chore(ci): build Hudi once per profile set and reuse it in the GitHub test jobs - #20363
Closed
databird07 wants to merge 1 commit into
Closed
databird07 wants to merge 1 commit into
databird07 wants to merge 1 commit into
Conversation
… test jobs
Every test job of bot.yml ran its own `mvn clean install -DskipTests` before its
test steps (about 95 of ~620 job-minutes of a master push). Jobs that build the
same profile set now restore one shared build instead.
What:
- build-jdk11-spark35 (scala-2.12, spark3.5, flink2.2, -Phudi-platform-service,
-Pthrift-gen-source, in the hudi-ci docker image) serves
test-spark-client-and-hadoop-common, test-utilities,
test-common-and-other-modules and test-hudi-hadoop-mr-and-hudi-java-client.
- build-java17-spark42 (java17, scala-2.13, spark4.2) serves the five
test-spark-java17-* jobs.
- build-flink (flink2.2 with avro 1.11.4 / parquet 1.15.2) serves test-flink-1/2.
- validate-bundles, validate-bundle-spark4 and integration-tests keep their own
builds: they use other profiles (bundle deploy, -Pintegration-tests, other
Spark/Flink versions), so their output is not the same build.
How:
- .github/actions/save-hudi-build packs every module target/ directory and
~/.m2/repository/org/apache/hudi (pax tar + zstd, identical target/lib jars
hard-linked so they are stored once) and uploads them as a workflow artifact
named hudi-build-<sha>-<jdk>-<profiles...>.
- .github/actions/restore-hudi-build downloads that artifact, checks that its
manifest names this commit and this profile set, replaces the local
org/apache/hudi repository with it, unpacks the target/ directories and
re-stamps them (compress_mtimes.py) so Maven sees them as up to date: every
distinct original millisecond keeps its order and gets its own millisecond
just after the newest checked-out source, so neither javac, maven-jar-plugin
nor scala-maven-plugin (which recompiles when a source is not older than its
timestamp file) treats the restored outputs as stale.
- The test jobs drop their build step and restore instead; every test step
(module lists, profiles, -Dtest / wildcardSuites filters, excludes) is
unchanged. Docker based jobs mount the workspace at the same absolute path
in the build and test containers, since Maven metadata stores absolute paths.
Review follow-ups: the save action chowns the packed target/ dirs and
org/apache/hudi to the runner user before hard-linking and packing (builds
in the hudi-ci container leave root-owned files), artifact retention is 7
days so "re-run failed jobs" keeps working for a week, and the three build
jobs get a 120-minute timeout instead of 60.
Measured locally (JDK 17, Maven 3.10.0, taskset -c 0-3, 20 runs each of
the "Java UT 1 - Common & Spark" step of test-spark-java17-java-tests-part1
narrowed to -Dtest=TestHoodieAvroUtils, JaCoCo on):
- in-job build (today): 20/20 exit 0, 103 tests, 0/0/0 fail/err/skip,
median 126.3 s, p90 127.6 s
- fresh checkout + restored artifact: 20/20 exit 0, 103 tests, 0/0/0,
0 recompiled sources, median 126.9 s, p90 128.0 s
- JaCoCo (hudi-common + hudi-spark): 1021 covered lines in every run of
both series; intersection and union identical, no line lost or gained.
- Without the checkstyle cache refresh the restored runs took median
143.4 s (checkstyle re-checked every source of the fresh checkout).
Artifact for the Java 17 / Spark 4.2 group: 1.07 GB (1.04 GB targets after
hard-linking duplicate target/lib jars, was 3.76 GB; 35 MB of
org/apache/hudi). Upstream run 38362333401 spent 63.5 of 621.8 job-minutes
in the 11 builds removed here; with three shared builds and the restores
the projected saving is about 30 job-minutes per push, while the critical
path gets about 3 minutes longer because test jobs wait for a build job.
hudi-agent
reviewed
Oct 16, 2026
hudi-agent
left a comment
hudi-agent
left a comment
Contributor
There was a problem hiding this comment.
No reviewable code files in this PR.
cc @yihua
databird07
changed the title
Contributor
Author
|
Combined into #20364 so the total impact can be reviewed in one place; this commit is included there unchanged. Closing. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #20363 +/- ##
============================================
+ Coverage 80.70% 80.74% +0.04%
- Complexity 35124 35207 +83
============================================
Files 2553 2553
Lines 143524 143629 +105
Branches 17474 17623 +149
============================================
+ Hits 115830 115975 +145
+ Misses 19746 19694 -52
- Partials 7948 7960 +12
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Every test job in
bot.ymlruns its ownmvn clean install -DskipTestsbefore testing. In run 38362333401 the 11 builds that share a profile set took 63.5 of ~622 job-minutes.Summary and Changelog
target/directories and~/.m2/.../org/apache/hudias a workflow artifact (pax tar + zstd, duplicate jars hard-linked; ~1 GB).compress_mtimes.py) so Maven treats the outputs as up to date. Test steps themselves are unchanged.Impact
CI-only. Projected saving is ~30 job-minutes per push. The critical path gets ~3 minutes longer, because test jobs wait for a build job. Locally, 20 runs each of in-job build vs. restored artifact gave identical test counts and identical JaCoCo coverage, with 0 recompiled sources after restore. This hasn't run on GitHub Actions yet, so this PR's own CI run is the first real check.
Risk Level
medium. Workflow restructuring that depends on the restored outputs being treated as up to date. Mitigated by manifest checks (commit and profile set) on restore and by the local comparison above. Reverting is a single-commit revert.
Documentation Update
none
Contributor's checklist