Introduce DaskReader with ts.virtual_chunked; unify get_tensorstore() API#2
Open
mkitti wants to merge 1 commit intoJaneliaSciComp:unifiedfrom
Open
Introduce DaskReader with ts.virtual_chunked; unify get_tensorstore() API#2mkitti wants to merge 1 commit intoJaneliaSciComp:unifiedfrom
mkitti wants to merge 1 commit intoJaneliaSciComp:unifiedfrom
Conversation
… API
All readers now expose get_tensorstore() -> ts.TensorStore instead of
get_tensorstore_spec() -> dict.
- Add DaskReader(BaseReader) intermediate class in readers/base.py:
wraps dask arrays via ts.virtual_chunked with a synchronous _read_fn
callback dispatched on TensorStore's thread pool. Chunk layout is
aligned to the dask array's native chunk shape.
- Tier 1 readers (N5, Zarr2, Zarr3, Precomputed): rename method and
call ts.open() internally, returning an opened ts.TensorStore.
- Tier 2 readers (TIFF, ND2, IMS, HDF5, CZI) and Tier 3 (BIOIOReader):
extend DaskReader; remove sentinel {'driver': 'array', 'array': ...}
dicts that were never opened with ts.open().
- Converter: remove _is_tier2 branching and _domain_to_slices(); call
reader.get_tensorstore() uniformly for all formats.
- api/dataset.py, __main__.py: update all call sites accordingly.
- Tests: replace get_tensorstore_spec() assertions with
store.spec().to_json()['driver'] checks for Tier 1, and
store[...].read().result() data access for all tiers.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
All readers now expose get_tensorstore() -> ts.TensorStore instead of
get_tensorstore_spec() -> dict.
wraps dask arrays via ts.virtual_chunked with a synchronous _read_fn
callback dispatched on TensorStore's thread pool. Chunk layout is
aligned to the dask array's native chunk shape.
call ts.open() internally, returning an opened ts.TensorStore.
extend DaskReader; remove sentinel {'driver': 'array', 'array': ...}
dicts that were never opened with ts.open().
reader.get_tensorstore() uniformly for all formats.
store.spec().to_json()['driver'] checks for Tier 1, and
store[...].read().result() data access for all tiers.
Co-Authored-By: Claude Sonnet 4.6 [email protected]