Skip to content

GEOPY-2602: Validate grid size for clipping 2D grids for a given amount of RAM#879

Merged
domfournier merged 4 commits intorelease/GA_4.8from
GEOPY-2602
May 5, 2026
Merged

GEOPY-2602: Validate grid size for clipping 2D grids for a given amount of RAM#879
domfournier merged 4 commits intorelease/GA_4.8from
GEOPY-2602

Conversation

@MatthieuCMira
Copy link
Copy Markdown
Contributor

@MatthieuCMira MatthieuCMira commented May 1, 2026

GEOPY-2602 - Validate grid size for clipping 2D grids for a given amount of RAM
catch memory error.

I changed a bit the error messages too....

Add a safe loader to anticipate the memory

Tests everything
Copilot AI review requested due to automatic review settings May 1, 2026 20:06
@github-actions github-actions Bot changed the title Geopy 2602 GEOPY-2602: Validate grid size for clipping 2D grids for a given amount of RAM May 1, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented May 1, 2026

Codecov Report

❌ Patch coverage is 91.30435% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.09%. Comparing base (c0aebed) to head (8a82284).

Files with missing lines Patch % Lines
geoh5py/io/h5_reader.py 87.50% 1 Missing and 1 partial ⚠️
geoh5py/shared/exceptions.py 93.33% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@                Coverage Diff                 @@
##           release/GA_4.8     #879      +/-   ##
==================================================
- Coverage           91.11%   91.09%   -0.02%     
==================================================
  Files                 113      113              
  Lines               10460    10459       -1     
  Branches             1931     1933       +2     
==================================================
- Hits                 9531     9528       -3     
- Misses                492      493       +1     
- Partials              437      438       +1     
Files with missing lines Coverage Δ
geoh5py/shared/utils.py 91.14% <ø> (+0.36%) ⬆️
geoh5py/ui_json/ui_json.py 88.18% <ø> (ø)
geoh5py/io/h5_reader.py 96.22% <87.50%> (-0.75%) ⬇️
geoh5py/shared/exceptions.py 97.14% <93.33%> (-2.86%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR primarily improves robustness around large HDF5 dataset reads by pre-checking available system memory (and surfacing a clearer custom exception), while also refactoring UI JSON handling (dependency/group enablement, promotion/validation) and updating/expanding the test suite accordingly.

Changes:

  • Add safe_load_dataset with memory pre-check + MemoryValidationError, and add psutil dependency to support this.
  • Refactor BaseUIJson / UI JSON validation + form behaviors (dependencies, enabled-state propagation, flattening/promotion, updated error messages).
  • Expand and adjust tests across UI JSON, input-file utilities, referenced data maps, and dataset loading.

Reviewed changes

Copilot reviewed 22 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tests/utils_test.py Adds coverage for enum_name_to_str utility behavior.
tests/ui_json/uijson_test.py Refactors UI JSON tests to new enablement/dependency logic and adds new scenarios (group + dependency enablement, copy_relatives).
tests/ui_json/forms_test.py Updates form tests for changed value/serialization/flatten semantics (Data*, MultiSelect, DataRange).
tests/safe_load_dataset_test.py New unit tests validating safe_load_dataset success/failure paths and error messaging.
tests/reference_data_test.py Adjusts referenced data map copy/rename expectations.
tests/input_file_utils_test.py New tests for UI JSON utility functions (flatten/collect/enabled/group behavior).
tests/input_file_test.py New comprehensive InputFile tests covering promotions, validations, serialization, and copy behaviors.
recipe.yaml Sets placeholder package version for build replacement.
pyproject.toml Adds psutil runtime dependency.
poetry.lock Updates lockfile (Poetry generator version + psutil resolved).
geoh5py/ui_json/validation.py Refactors UIJson cross-validations to operate on BaseUIJson/forms rather than raw dicts; adds promotion helper.
geoh5py/ui_json/ui_json.py Major refactor of UIJson read/from-dict/infer, enabled/dependency mechanics, cross-validation and copy_relatives support.
geoh5py/ui_json/input_file.py Minor typing fix when passing stringify() output into InputFile.
geoh5py/ui_json/forms.py Updates form optional/enabled semantics, DataForm value typing, and adds flatten/set_value support for grouped/range forms.
geoh5py/ui_json/annotations.py Adjusts serializers/validators (enum serialization, workspace/path handling, UUID list typing).
geoh5py/shared/validators.py Removes unused types_to_string helper (serializer path changed).
geoh5py/shared/utils.py Expands stringify mappings, updates str2none, extends enum_name_to_str, adds type2uuid.
geoh5py/shared/exceptions.py Unifies validation errors under BaseValidationError and introduces MemoryValidationError.
geoh5py/objects/surveys/magnetics.py Adds GroundMagnetics survey object type.
geoh5py/objects/surveys/gravity.py Replaces prior accidental content with survey object definitions for gravity.
geoh5py/objects/init.py Exports new gravity/magnetics survey classes.
geoh5py/io/h5_reader.py Adds safe_load_dataset and uses it when loading array-like attributes to avoid MemoryError crashes.
geoh5py/data/referenced_data.py Makes remove_data_map operate on a copied mapping to avoid in-place mutation issues.
geoh5py/data/geometric_data.py Adds name setter that updates on-file attributes and keeps referenced data-map keys in sync.
geoh5py/data/data_type.py Adds parent-reference renaming support and updates value-map lookup to use data_map ordering.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread geoh5py/shared/utils.py
Comment thread geoh5py/ui_json/ui_json.py Outdated
Comment thread geoh5py/ui_json/ui_json.py Outdated
Comment thread geoh5py/io/h5_reader.py Outdated
@MatthieuCMira MatthieuCMira changed the base branch from develop to release/GA_4.8 May 4, 2026 13:13
@MatthieuCMira MatthieuCMira requested a review from Copilot May 4, 2026 13:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread geoh5py/shared/exceptions.py
Comment thread geoh5py/io/h5_reader.py Outdated
Comment thread geoh5py/io/h5_reader.py
Comment thread tests/safe_load_dataset_test.py Outdated
domfournier
domfournier previously approved these changes May 4, 2026
@domfournier domfournier merged commit 5a3b845 into release/GA_4.8 May 5, 2026
18 of 19 checks passed
@domfournier domfournier deleted the GEOPY-2602 branch May 5, 2026 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants