GEOPY-2602: Validate grid size for clipping 2D grids for a given amount of RAM#879
GEOPY-2602: Validate grid size for clipping 2D grids for a given amount of RAM#879domfournier merged 4 commits intorelease/GA_4.8from
Conversation
Add a safe loader to anticipate the memory Tests everything
for more information, see https://pre-commit.ci
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release/GA_4.8 #879 +/- ##
==================================================
- Coverage 91.11% 91.09% -0.02%
==================================================
Files 113 113
Lines 10460 10459 -1
Branches 1931 1933 +2
==================================================
- Hits 9531 9528 -3
- Misses 492 493 +1
- Partials 437 438 +1
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR primarily improves robustness around large HDF5 dataset reads by pre-checking available system memory (and surfacing a clearer custom exception), while also refactoring UI JSON handling (dependency/group enablement, promotion/validation) and updating/expanding the test suite accordingly.
Changes:
- Add
safe_load_datasetwith memory pre-check +MemoryValidationError, and addpsutildependency to support this. - Refactor
BaseUIJson/ UI JSON validation + form behaviors (dependencies, enabled-state propagation, flattening/promotion, updated error messages). - Expand and adjust tests across UI JSON, input-file utilities, referenced data maps, and dataset loading.
Reviewed changes
Copilot reviewed 22 out of 25 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/utils_test.py | Adds coverage for enum_name_to_str utility behavior. |
| tests/ui_json/uijson_test.py | Refactors UI JSON tests to new enablement/dependency logic and adds new scenarios (group + dependency enablement, copy_relatives). |
| tests/ui_json/forms_test.py | Updates form tests for changed value/serialization/flatten semantics (Data*, MultiSelect, DataRange). |
| tests/safe_load_dataset_test.py | New unit tests validating safe_load_dataset success/failure paths and error messaging. |
| tests/reference_data_test.py | Adjusts referenced data map copy/rename expectations. |
| tests/input_file_utils_test.py | New tests for UI JSON utility functions (flatten/collect/enabled/group behavior). |
| tests/input_file_test.py | New comprehensive InputFile tests covering promotions, validations, serialization, and copy behaviors. |
| recipe.yaml | Sets placeholder package version for build replacement. |
| pyproject.toml | Adds psutil runtime dependency. |
| poetry.lock | Updates lockfile (Poetry generator version + psutil resolved). |
| geoh5py/ui_json/validation.py | Refactors UIJson cross-validations to operate on BaseUIJson/forms rather than raw dicts; adds promotion helper. |
| geoh5py/ui_json/ui_json.py | Major refactor of UIJson read/from-dict/infer, enabled/dependency mechanics, cross-validation and copy_relatives support. |
| geoh5py/ui_json/input_file.py | Minor typing fix when passing stringify() output into InputFile. |
| geoh5py/ui_json/forms.py | Updates form optional/enabled semantics, DataForm value typing, and adds flatten/set_value support for grouped/range forms. |
| geoh5py/ui_json/annotations.py | Adjusts serializers/validators (enum serialization, workspace/path handling, UUID list typing). |
| geoh5py/shared/validators.py | Removes unused types_to_string helper (serializer path changed). |
| geoh5py/shared/utils.py | Expands stringify mappings, updates str2none, extends enum_name_to_str, adds type2uuid. |
| geoh5py/shared/exceptions.py | Unifies validation errors under BaseValidationError and introduces MemoryValidationError. |
| geoh5py/objects/surveys/magnetics.py | Adds GroundMagnetics survey object type. |
| geoh5py/objects/surveys/gravity.py | Replaces prior accidental content with survey object definitions for gravity. |
| geoh5py/objects/init.py | Exports new gravity/magnetics survey classes. |
| geoh5py/io/h5_reader.py | Adds safe_load_dataset and uses it when loading array-like attributes to avoid MemoryError crashes. |
| geoh5py/data/referenced_data.py | Makes remove_data_map operate on a copied mapping to avoid in-place mutation issues. |
| geoh5py/data/geometric_data.py | Adds name setter that updates on-file attributes and keeps referenced data-map keys in sync. |
| geoh5py/data/data_type.py | Adds parent-reference renaming support and updates value-map lookup to use data_map ordering. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
GEOPY-2602 - Validate grid size for clipping 2D grids for a given amount of RAM
catch memory error.
I changed a bit the error messages too....