Repository navigation
PIVOT-13635: Test bundling jdk.incubator.vector - #95
Closed
NathanEckert wants to merge 1 commit into
Closed
NathanEckert wants to merge 1 commit into
NathanEckert wants to merge 1 commit into
Conversation
roelbs
added a commit
to OvertureMaps/schema
that referenced
this pull request
Aug 11, 2026
The suite depended on machine state in two ways. Ambient Spark/JVM configuration (SPARK_HOME, PYSPARK_SUBMIT_ARGS, *_CONF_DIR, JVM option variables) leaked into the test JVM, where a system-wide Spark install fails session setup with opaque py4j "'JavaPackage' object is not callable" errors. And the pyspark wheel ships Spark's jars but no Java runtime, so tests ran on whatever Java the machine resolves. Scrub the leaking variables at conftest import and pin SPARK_HOME to the venv's bundled distribution and the worker/driver Python to sys.executable. SPARK_LOCAL_IP is deliberately left alone: it exists to make local sessions work (macOS/VPN hostname resolution). Only the test suite is affected; the CLI and library keep honoring ambient configuration. Add jdk4py to the dev group so the venv owns the JVM too: the <22 range makes lowest-direct resolve 17.0.9.2 (JDK 17, matching pyspark 3.4's ceiling) while the lock picks the maintained 21.x line for pyspark 4.x, letting CI drop its setup-java step entirely. jdk4py ships wheels only and its runtime is jlink-stripped without jdk.incubator.vector, which Spark 4.x demands from every JVM it launches via --add-modules (upstream declined to bundle the module: activeviam/jdk4py#95). The conftest therefore exposes the JDK through a mirrored JAVA_HOME whose bin/java drops that one flag -- the module only enables vectorized BLAS, which Spark treats as optional -- and falls back to the system Java on platforms without a wheel or a POSIX shell (BSD, Windows, musl), where the dependency marker skips the install. Refs #585 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test run to see the size