Skip to content

PIVOT-13635: Test bundling jdk.incubator.vector - #95

Closed
NathanEckert wants to merge 1 commit into
mainfrom
task/main/include_jdk_incubator_vector
Closed

NathanEckert wants to merge 1 commit into
mainfrom
task/main/include_jdk_incubator_vector

Conversation

@NathanEckert

Copy link
Copy Markdown
Contributor

Test run to see the size

roelbs added a commit to OvertureMaps/schema that referenced this pull request Aug 11, 2026
The suite depended on machine state in two ways. Ambient Spark/JVM
configuration (SPARK_HOME, PYSPARK_SUBMIT_ARGS, *_CONF_DIR, JVM option
variables) leaked into the test JVM, where a system-wide Spark install
fails session setup with opaque py4j "'JavaPackage' object is not
callable" errors. And the pyspark wheel ships Spark's jars but no Java
runtime, so tests ran on whatever Java the machine resolves.

Scrub the leaking variables at conftest import and pin SPARK_HOME to
the venv's bundled distribution and the worker/driver Python to
sys.executable. SPARK_LOCAL_IP is deliberately left alone: it exists
to make local sessions work (macOS/VPN hostname resolution). Only the
test suite is affected; the CLI and library keep honoring ambient
configuration.

Add jdk4py to the dev group so the venv owns the JVM too: the <22
range makes lowest-direct resolve 17.0.9.2 (JDK 17, matching pyspark
3.4's ceiling) while the lock picks the maintained 21.x line for
pyspark 4.x, letting CI drop its setup-java step entirely.

jdk4py ships wheels only and its runtime is jlink-stripped without
jdk.incubator.vector, which Spark 4.x demands from every JVM it
launches via --add-modules (upstream declined to bundle the module:
activeviam/jdk4py#95). The conftest therefore exposes the JDK through a
mirrored JAVA_HOME whose bin/java drops that one flag -- the module
only enables vectorized BLAS, which Spark treats as optional -- and
falls back to the system Java on platforms without a wheel or a POSIX
shell (BSD, Windows, musl), where the dependency marker skips the
install.

Refs #585

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant