Skip to content

Commit 57eeddb

Browse files
Add dataset optimization: fast access patterns (#754)
* suppress Zarr V3 numcodecs warnings * add statistical properties (mean, variance, std) to stats schema * monkey-patch ZFP codec to fix bug temporarily * add ZFP-based access pattern optimization utilities with Dask support * refactor ZFP monkey patch to handle optional Dask dependency * refactor Dask integration to simplify ZFP plugin setup and import handling * refactor Dask integration to simplify ZFP plugin setup and import handling * ZFP plugin import by switching to local scoped import * refactor access pattern logic to rename `access_patterns` to `optimize_dimensions` and update related usage * fix syntax error on warning ignore * update rechunking tutorial to v1 * lint * remove TYPE_CHECKING import and adjust ZfpQuality import * add unit tests for optimize_access_pattern module * set pytest fixtures to class scope and mock environment variables in tests for speed * add unit test for user-provided Dask client in optimize_access_patterns * update test cases to have a slightly different chunksize * add 'distributed' package to test dependencies in noxfile * add 'zfpy' package to test dependencies in noxfile * refactor: adjust zfpy codec handling and refactor related tests * Update src/mdio/optimize/common.py Co-authored-by: Brian Michell <[email protected]> * update test environment variable values in optimize_access_pattern unit test * nit * refactor: replace ZfpQuality with compressor in optimize_access_patterns and update tests * patch: configure zarr concurrency and clarify docstring on Blosc threading * fix: ensure stats check is conditional on compressor in access patterns * Revert "patch: configure zarr concurrency and clarify docstring on Blosc threading" This reverts commit 64cfa7d. --------- Co-authored-by: Altay Sansal <[email protected]> Co-authored-by: Brian Michell <[email protected]>
1 parent 73ab646 commit 57eeddb

File tree

14 files changed

+1889
-406
lines changed

14 files changed

+1889
-406
lines changed

docs/api_reference.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,3 +40,10 @@ and
4040
.. automodule:: mdio.core.dimension
4141
:members:
4242
```
43+
44+
## Optimization
45+
46+
```{eval-rst}
47+
.. automodule:: mdio.optimize.access_pattern
48+
:members:
49+
```

0 commit comments

Comments
 (0)