Skip to content

Spark 4.0.1#549

Open
dsamaey wants to merge 69 commits intodevelopfrom
35-cross-compile-openeo-geotrellis-to-scala-213
Open

Spark 4.0.1#549
dsamaey wants to merge 69 commits intodevelopfrom
35-cross-compile-openeo-geotrellis-to-scala-213

Conversation

@dsamaey
Copy link
Contributor

@dsamaey dsamaey commented Oct 3, 2025

No description provided.

dsamaey added 30 commits September 3, 2025 19:15
@dsamaey
chore: upgrade to scala 2.13
3ba4ff1 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
e233f53 
…peneo-geotrellis-to-scala-213

# Conflicts:
#	openeo-geotrellis/src/main/scala/org/openeo/geotrellis/layers/FileLayerProvider.scala
@dsamaey
chore: upgrade to scala 2.13
bc9d161 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
chore: upgrade to scala 2.13
c5bd2a0 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
61d2e3b 
…peneo-geotrellis-to-scala-213

# Conflicts:
#	openeo-geotrellis/src/test/scala/org/openeo/geotrellis/OpenEOProcessesSpec.scala
@dsamaey
chore: upgrade to scala 2.13
0c59d9a 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
21ea8b6 
…peneo-geotrellis-to-scala-213

# Conflicts:
#	openeo-geotrellis/src/main/scala/org/openeo/geotrellis/layers/FileLayerProvider.scala
#	openeo-geotrellis/src/test/scala/org/openeo/geotrellis/OpenEOProcessesSpec.scala
#	pom.xml
@dsamaey
chore: upgrade to scala 2.13
f1524f1 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
chore: upgrade to scala 2.13
4a3dbd9 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
chore: upgrade to scala 2.13
a12945b 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
e205f8c 
…peneo-geotrellis-to-scala-213
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
61d3159 
…peneo-geotrellis-to-scala-213
@dsamaey
chore: upgrade to scala 2.13
e2eefe0 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
chore: upgrade to scala 2.13
603f5c1 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
chore: upgrade to scala 2.13
c2c97ab 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
chore: upgrade to scala 2.13
e45d540 
# Open-EO/openeo-geotrellis-kubernetes#35
@dsamaey
chore: upgrade to Spark 4.0.1
8efe65c
@dsamaey
chore: upgrade to Spark 4.0.1
bd48779
@dsamaey
chore: upgrade to Spark 4.0.1
525d60f
@dsamaey
chore: upgrade to Spark 4.0.1
5a21369
@dsamaey
chore: upgrade to Spark 4.0.1
8c9e815
@dsamaey
chore: upgrade to Spark 4.0.1
2c3e626
@dsamaey
chore: upgrade to Spark 4.0.1
2e0b128
@dsamaey
chore: upgrade to Spark 4.0.1
65b8ded
@dsamaey
chore: upgrade to Spark 4.0.1
67cff18
@dsamaey
chore: upgrade to Spark 4.0.1
721adcf
@dsamaey
chore: upgrade to Scala 2.13
d18b1ec
@dsamaey
chore: upgrade to Scala 2.13
89bb9ad
@dsamaey
chore: upgrade to Scala 2.13
5aded50
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
62ff46d 
…peneo-geotrellis-to-scala-213
dsamaey and others added 19 commits September 24, 2025 12:30
@dsamaey
chore: upgrade to Spark 4.0.1
7291d3d
@dsamaey
chore: upgrade to Spark 4.0.1
b54959f
@dsamaey
chore: upgrade to Spark 4.0.1
7a0faac
@JeroenVerstraelen
update netty to 4.1.118.Final for compatibility with kubernetes-clien…
488e0dd 
…t-7.1.0

ref eu-cdse/openeo-cdse-infra#690
@jdries
make sure scala is excluded
e796210
@dsamaey
chore: upgrade to Spark 4.0.1
6a0013e
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
de3aee4 
…peneo-geotrellis-to-scala-213

# Conflicts:
#	geotrellis-integrationtests/pom.xml
#	openeo-geotrellis/pom.xml
#	pom.xml
@dsamaey
chore: upgrade to Spark 3.5.5
8f65a81
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
a759e6f 
…peneo-geotrellis-to-scala-213

# Conflicts:
#	docker/tests_dockerfile
#	pom.xml
@dsamaey
chore: upgrade to Spark 3.5.5
3b58479
@dsamaey
chore: upgrade to Spark 3.5.5
eb786b9
@dsamaey
chore: upgrade to Spark 4.0.1
7ea7168 
# #528
@dsamaey
chore: upgrade to Spark 4.0.1
7fa05bb 
# #528
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
cc1dd2e 
…peneo-geotrellis-to-scala-213

# Conflicts:
#	pom.xml
@dsamaey
chore: update changelog
9ac1ea6
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
0381b2e 
…peneo-geotrellis-to-scala-213
@dsamaey
chore: fix opensearch ParVector bug
5e39ab6
@dsamaey
chore: clean up Jenkinsfile
e3739af
@dsamaey
Merge remote-tracking branch 'origin/develop' into 35-cross-compile-o…
fd43a86 
…peneo-geotrellis-to-scala-213

# Conflicts:
#	CHANGELOG.md
#	openeo-geotrellis/src/main/scala/org/openeo/geotrellis/aggregate_polygon/intern/PixelRateValidator.scala
soxofaan added a commit to Open-EO/openeo-geopyspark-driver that referenced this pull request Oct 23, 2025
@soxofaan
jars.json: update "spark4-scala213" variant to "2.5.1_2.13-SNAPSHOT"
fbd2ae5 
refs: eu-cdse/openeo-cdse-infra#764, Open-EO/openeo-geotrellis-extensions#549
jdries and others added 5 commits December 8, 2025 14:55
@jdries
Merge branch 'develop' into 35-cross-compile-openeo-geotrellis-to-sca…
85f14c1 
…la-213
@jdries
use next image with gdal 3.11
28e7dcc 
Open-EO/openeo-geopyspark-driver#1371
@jdries
bump jackson version
ded1ce1
@jdries
bump jackson version, align with spark4
88e7c6b
@bossie
add onnx dependencies to default Python 3.11 image for batch jobs on …
21e1151 
…Terrascope

#563
#577

java.lang.NoClassDefFoundError: ai/onnxruntime/OrtEnvironment
	at org.openeo.geotrellis.corsa.package$.Env$lzycompute(package.scala:36)
	at org.openeo.geotrellis.corsa.package$.Env(package.scala:36)
	at org.openeo.geotrellis.corsa.package$.$anonfun$processWindowOnnx$1(package.scala:144)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at org.openeo.geotrellis.corsa.package$.processWindowOnnx(package.scala:143)
	at org.openeo.geotrellis.corsa.package$.compress(package.scala:85)
	at org.openeo.geotrellis.OpenEOProcesses.$anonfun$corsaCompressGeneric$2(OpenEOProcesses.scala:1162)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$mapValues$3(PairRDDFunctions.scala:754)
	at scala.collection.parallel.IterableSplitter$Mapped.next(RemainsIterator.scala:463)
	at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:478)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:601)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:143)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:57)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:111)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:171)
	at org.apache.spark.scheduler.Task.run(Task.scala:147)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Executor.scala:647)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:80)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:77)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:650)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ClassNotFoundException: ai.onnxruntime.OrtEnvironment
	... 25 more
@bossie
Copy link
Collaborator

bossie commented Jan 5, 2026

@dsamaey AFAICT this branch produces the geotrellis-dependencies jar used in Terrascope's Python 3.11 images; ok to merge develop into this branch to keep it up-to-date?

@jdries
Copy link
Contributor

jdries commented Jan 5, 2026

@bossie normally this is not needed, there is a single dependencies jar that works for both spark versions.
(Specific Spark dependencies are installed directly in the image.)

@bossie
Copy link
Collaborator

bossie commented Jan 5, 2026

@jdries I'm referring to this line: https://github.com/Open-EO/openeo-geotrellis-kubernetes/blob/8ad31a688d91ee319ba60cadce50c3c942b0443c/docker/AlmaLinux/Jenkinsfile.almalinux9-spark4-py311-openeo#L12

@jdries
Copy link
Contributor

jdries commented Jan 5, 2026

Looks like a leftover, can you try replacing that with the default jar, like:
https://artifactory.vgt.vito.be/artifactory/libs-snapshot-public/org/openeo/geotrellis-dependencies/2.6.1_2.13-SNAPSHOT/geotrellis-dependencies-2.6.1_2.13-SNAPSHOT.jar
(geopyspark unit tests should already have tested this approach, but of course not impossible that it breaks something.)

bossie added a commit to Open-EO/openeo-geotrellis-kubernetes that referenced this pull request Jan 5, 2026
@bossie
point 3.11 image to Spark-agnostic dependency jar
863cc0c 
Open-EO/openeo-geotrellis-extensions#563
Open-EO/openeo-geotrellis-extensions#577

Open-EO/openeo-geotrellis-extensions#549 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dsamaey @bossie @jdries @JeroenVerstraelen