Commit 57eeddb
Add dataset optimization: fast access patterns (#754)
* suppress Zarr V3 numcodecs warnings
* add statistical properties (mean, variance, std) to stats schema
* monkey-patch ZFP codec to fix bug temporarily
* add ZFP-based access pattern optimization utilities with Dask support
* refactor ZFP monkey patch to handle optional Dask dependency
* refactor Dask integration to simplify ZFP plugin setup and import handling
* refactor Dask integration to simplify ZFP plugin setup and import handling
* ZFP plugin import by switching to local scoped import
* refactor access pattern logic to rename `access_patterns` to `optimize_dimensions` and update related usage
* fix syntax error on warning ignore
* update rechunking tutorial to v1
* lint
* remove TYPE_CHECKING import and adjust ZfpQuality import
* add unit tests for optimize_access_pattern module
* set pytest fixtures to class scope and mock environment variables in tests for speed
* add unit test for user-provided Dask client in optimize_access_patterns
* update test cases to have a slightly different chunksize
* add 'distributed' package to test dependencies in noxfile
* add 'zfpy' package to test dependencies in noxfile
* refactor: adjust zfpy codec handling and refactor related tests
* Update src/mdio/optimize/common.py
Co-authored-by: Brian Michell <[email protected]>
* update test environment variable values in optimize_access_pattern unit test
* nit
* refactor: replace ZfpQuality with compressor in optimize_access_patterns and update tests
* patch: configure zarr concurrency and clarify docstring on Blosc threading
* fix: ensure stats check is conditional on compressor in access patterns
* Revert "patch: configure zarr concurrency and clarify docstring on Blosc threading"
This reverts commit 64cfa7d.
---------
Co-authored-by: Altay Sansal <[email protected]>
Co-authored-by: Brian Michell <[email protected]>1 parent 73ab646 commit 57eeddb
File tree
14 files changed
+1889
-406
lines changed- docs
- tutorials
- src/mdio
- api
- builder
- schemas/v1
- core
- optimize
- tests/unit
- v1
14 files changed
+1889
-406
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
0 commit comments