Merge branch 'main' of github.com:ArcticSnow/TopoPyScale into main

Author: joelfiddes

.github/copilot-instructions.md

@@ -0,0 +1,52 @@
+# Copilot Instructions for TopoPyScale
+
+## Project Overview
+TopoPyScale is a Python package for topography-based downscaling of climate data to hillslope scale. It processes climate reanalysis data (ERA5, etc.) and digital elevation models (DEM) to produce outputs for models like Cryogrid and FSM. The codebase is organized as a PyPI library, with main logic in the `TopoPyScale/` directory and documentation in `doc/`.
+
+## Key Components
+- **Core Library:** All main modules are in `TopoPyScale/`. Each file typically handles a specific aspect (e.g., `fetch_dem.py` for DEM fetching, `topo_scale.py` for downscaling logic).
+- **Documentation:** Markdown docs are in `doc/docs/`. API docs are auto-generated using `lazydocs`.
+- **Configuration:** Project uses `pyproject.toml` and `MANIFEST.in` for packaging. Documentation build is managed by `mkdocs.yml`.
+
+## Developer Workflows
+- **Testing:** GitHub Actions run tests via `test_topopyscale.yml`. Local test commands are not specified; add/maintain tests in the main package directory.
+- **Documentation:**
+  - Edit docs in `doc/docs/`.
+  - Build locally: `mkdocs serve` (see `doc/README.md`).
+  - API docs: Run `lazydocs` as described in `doc/README.md`.
+- **Releases:** Create a new branch for features/bugfixes, merge to `main`, then follow release video guide (see `README.md`).
+
+## Project-Specific Patterns
+- **Data Flow:**
+  - Inputs: Climate data (ERA5, CORDEX), DEM (local or fetched).
+  - Processing: DEM-derived values, clustering (k-means), interpolation (bilinear, inverse square distance).
+  - Outputs: Cryogrid/FSM formats.
+- **Contribution:**
+  - Issues/discussions on GitHub.
+  - Branching for features/bugfixes is encouraged.
+- **Docs Build:** Changes to docs auto-build on ReadTheDocs when committed.
+
+## Integration Points
+- **External Data:** Fetches DEM from public repositories (SRTM, ArcticDEM, ASTER).
+- **Dependencies:** See `pyproject.toml` and `doc/requirements.txt` for required packages (e.g., `mkdocs`, `lazydocs`).
+
+## Examples
+- To fetch DEM: see `fetch_dem.py`.
+- To run downscaling: see `topo_scale.py` and related modules.
+- To build docs locally:
+  ```bash
+  pip install mkdocs sphinx_rtd_theme mkdocs-material pygments
+  mkdocs serve
+  # Open http://127.0.0.1:8000/
+  ```
+- To update API docs:
+  ```bash
+  lazydocs --output-path="doc/docs" --overview-file="README.md" --src-base-url="https://github.com/ArcticSnow/TopoPyScale" TopoPyScale
+  ```
+
+## References
+- Main documentation: https://topopyscale.readthedocs.io
+- Release workflow: https://www.youtube.com/watch?v=Ob9llA_QhQY
+
+---
+**Feedback:** Please review and suggest updates for any unclear or missing sections.

.vscode/settings.json

@@ -0,0 +1,5 @@
+{
+    "python-envs.defaultEnvManager": "ms-python.python:conda",
+    "python-envs.defaultPackageManager": "ms-python.python:conda",
+    "python-envs.pythonProjects": []
+}

TopoPyScale/topo_param.py

@@ -190,7 +190,24 @@ def compute_dem_param(dem_file, fname='ds_param.nc', project_directory=Path('./'
     file_ds = pdir / output_folder / fname
     if file_ds.is_file():
         print(f'\n---> Dataset {fname} found.')
-        ds = xr.open_dataset(file_ds)
+        try:
+            ds = xr.open_dataset(file_ds)
+            # Test if we can actually read the elevation data
+            _ = ds.elevation.values
+        except (RuntimeError, OSError) as e:
+            if "NetCDF: HDF error" in str(e) or "filter returned failure" in str(e):
+                print(f'\n---> Dataset {fname} corrupted (HDF/compression error). Trying h5netcdf backend...')
+                try:
+                    ds = xr.open_dataset(file_ds, engine='h5netcdf')
+                    _ = ds.elevation.values
+                except Exception:
+                    print(f'\n---> h5netcdf backend failed. Regenerating dataset from DEM...')
+                    if Path(dem_file).is_file():
+                        ds = open_dem(dem_file)
+                    else:
+                        raise ValueError(f'ERROR: Dataset corrupted and no DEM available to regenerate')
+            else:
+                raise e
 
     else:
         if Path(dem_file).is_file():
@@ -204,7 +221,29 @@ def compute_dem_param(dem_file, fname='ds_param.nc', project_directory=Path('./'
     print('\n---> Extracting DEM parameters (slope, aspect, svf)')
     dx = ds.x.diff('x').median().values
     dy = ds.y.diff('y').median().values
-    dem_arr = ds.elevation.values
+    
+    # Safely access elevation data with error handling
+    try:
+        dem_arr = ds.elevation.values
+    except (RuntimeError, OSError) as e:
+        if "NetCDF: HDF error" in str(e) or "filter returned failure" in str(e):
+            print(f'---> Error reading elevation data: {str(e)}')
+            print('---> Attempting to load elevation data with different method...')
+            try:
+                # Try loading the data chunk by chunk or using compute()
+                dem_arr = ds.elevation.load().values
+            except Exception:
+                # If all else fails, regenerate from DEM file
+                print('---> All methods failed. Regenerating from original DEM...')
+                if Path(dem_file).is_file():
+                    ds_new = open_dem(dem_file)
+                    dem_arr = ds_new.elevation.values
+                    ds = ds_new
+                    var_in = list(ds.variables.keys())
+                else:
+                    raise ValueError(f'ERROR: Cannot read elevation data and no DEM file available')
+        else:
+            raise e
 
     if ('slope' not in var_in) or ('aspect' not in var_in):
         print('Computing slope and aspect ...')