use a more recent commit of bss that should create valid zarr arrays

[d-v-b]

Uses the latest version of bigstitcher. This should fix the issue with bigstitcher creating broken zarr arrays. But I haven't tested it!

[winnubstj]

Can you tell me what part in the zarr metadata should be updated with this latest change?

[d-v-b]

Prior to the proposed commit, Bigstitcher-spark depended on n5-zarr version 1.2.1, which had a bug resulting in Zarr arrays being saved with an invalid fill_value attribute. Instead of a JSON number, the fill_value would end up as a string, and neuroglancer in particular would refuse to load these Zarr arrays.

The bug in n5-zarr was fixed back in may, but bigstitcher-spark had not updated that particular dependency until recently.

[winnubstj]

Updated relevant references. Running test
nf-test test tests/workflows/test_render.nf.test
fails with error

Starting the spark driver
Checking for /shared/scratch/nf-test/spark/.sessionId
Found /shared/scratch/nf-test/spark/.sessionId in 0 s
Use Spark IP: 10.0.10.58
[-x, /home/johan/repos/voxelflow/tests/test-data/2tile_dataset/base-dataset.xml, -o, /shared/scratch/nf-test/tests/3902ce8de2b89bb77831efa63014a5a2/work/render.zarr, -d, ch0/s0, -ds, 1,1,1; 2,2,1; 4,4,2, --UINT16, --minIntensity, 0.0, --maxIntensity, 65535.0, --channelId, 0, --storage, ZARR, --preserveAnisotropy, --blockSize, 128,128,64]
xml: /home/johan/repos/voxelflow/tests/test-data/2tile_dataset/base-dataset.xml
Parsing selected angle ids ... all
Parsing selected channel ids ... [0]
Parsing selected illumination ids ... all
Parsing selected tile ids ... all
Parsing selected timepoint ids ... all
The following ViewIds will be processed: 
[0,0] [0,2] 
Fusing: All Views: [6923, -6638, 7] -> [7396, -6382, 73], dimensions (474, 257, 67) with blocksize (128, 128, 64) and (4, 4, 1) blocks per job
Fusing to UINT16, min intensity = 0.0, max intensity = 65535.0
Preserving anisotropy.
Anisotropy factor [computed from data]: 1.6683250414593698
Adjusted bounding box (anisotropy preserved: [6923, -6638, 4] -> [7396, -6382, 44], dimensions (474, 257, 41)
Exception in thread "main" java.lang.NoClassDefFoundError: org/janelia/saalfeldlab/n5/N5Exception$N5NoSuchKeyException
	at net.preibisch.bigstitcher.spark.util.N5Util.createWriter(N5Util.java:45)
	at net.preibisch.bigstitcher.spark.SparkAffineFusion.call(SparkAffineFusion.java:275)
	at net.preibisch.bigstitcher.spark.SparkAffineFusion.call(SparkAffineFusion.java:63)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
	at picocli.CommandLine.execute(CommandLine.java:2170)
	at net.preibisch.bigstitcher.spark.SparkAffineFusion.main(SparkAffineFusion.java:420)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.janelia.saalfeldlab.n5.N5Exception$N5NoSuchKeyException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 24 more
Warning: Voxelsize in x and y is different.
Warning: Voxelsize in x and y is different.

[d-v-b]

very cool 🙃 I wonder if n5-zarr now depends on a newer version of n5, but n5-zarr does not declare that dependency

[d-v-b]

looks like that N5NoSuchKeyException was added last month. Not clear to me how bigstitcher-spark could even compile with this issue, but maybe preibisch merged the PR without running any tests 🤔

I think it will be much faster to make changes to a fork of bigstitcher-spark rather than waiting for preibisch to implement stuff. Would that work for you?

[winnubstj]

Unless this is a pressing breaking problem for Kathleen I think we should just hold off till they fix it on their end.

[kathleenleeper]

Just flagging @winnubstj that this is a problem for us now :) we are producing many more single round datasets and so visualizing them has in fact become gating -- we can edit manually but it's kind of silly