Merge branch 'pik-piam:main' into main

Author: irr-github

README.md

@@ -1,7 +1,11 @@
 # PyPSA-China：An Open-Source Optimisation model of the Chinese Energy System
 
+PyPSA-China (PIK) is a open-source model of the Chinese energy system covering electricity and heat. It co-optimizes dispatch and investments under user-set constraints, such as limits to environmental impacts, to minimize costs. The model works at provincial resolution and can simulate a full year at hourly resolution.
+
+## PIK version
 This is the PIK implementation of the PyPSA-China power model, first published by Hailiang Liu et al for their study of [hydro-power in china](https://doi.org/10.1016/j.apenergy.2019.02.009) and extended by Xiaowei Zhou et al for their  ["Multi-energy system horizon planning: Early decarbonisation in China avoids stranded assets"](doi.org/10.1049/ein2.12011) paper. It is adapted from the Zhou version by the [PIK RD3-ETL team](https://www.pik-potsdam.de/en/institute/labs/energy-transition/energy-transition-lab), with the aim of coupling it to the [REMIND](https://www.pik-potsdam.de/en/institute/departments/transformation-pathways/models/remind) integrated assessment model. A reference guide is available as part of the [documentation](https://pik-piam.github.io/PyPSA-China-PIK/).
 
+## Overview
 PyPSA-China should be understood as a modelling worklow, using snakemake as workflow manager, around the [PyPSA python power system analysis](https://pypsa.org/) package. The workflow collects data, builds the power system network and plots the results. It is akin to its more mature sister project, [PyPSA-EUR](https://github.com/PyPSA/pypsa-eur), from which it is derived.
 
 Unlike PyPSA-EUR, which simplifies high resolution data into a user-defined network size, the PyPSA-China network is currently fixed to one node per province. This is in large part due to data availability issues.
@@ -10,57 +14,61 @@ The PyPSA can perform a number of different study types (investment decision, op
 
 The PyPSA-CHINA-PIK is currently under development. Please contact us if you intend to use it for publications.
 
+# Documentation
+The documentation can be found at https://pik-piam.github.io/PyPSA-China-PIK/
 
-# Installation
+# Getting started
 
-## Set-up on the PIK cluster
-Gurobi license activation from the compute nodes requires internet access. The workaround is an ssh tunnel to the login nodes, which can be set-up on the compute nodes with
-```
-# interactive session on the compute nodes
-srun --qos=priority --pty bash
-# key pair gen (here ed25518 but can be rsa)
-ssh-keygen -t ed25519 -f ~/.ssh/id_rsa.cluster_internal_exchange -C "$USER@cluster_internal_exchange"
-# leave the compute nodes
-exit
+## Installation
+
+An installation guide is provided at https://pik-piam.github.io/PyPSA-China-PIK/ 
+
+> [!NOTE] Set-up on the PIK cluster
+>    Gurobi license activation from the compute nodes requires internet access. The workaround is an ssh tunnel to the login nodes, which can be set-up on the compute nodes with
+```bash
+    # interactive session on the compute nodes
+    srun --qos=priority --pty bash
+    # key pair gen (here ed25518 but can be rsa)
+    ssh-keygen -t ed25519 -f ~/.ssh/id_rsa.cluster_internal_exchange -C "$USER@cluster_internal_exchange"
+    # leave the compute nodes
+    exit
 ```
-You will then need to add the contents of the public key `~/.ssh/id_rsa.cluster_internal_exchange.pub` to your authorised `~/.ssh/authorized_keys`, eg. with `cat <key_name> >> authorized_keys`
+
+> You will then need to add the contents of the public key `~/.ssh/id_rsa.cluster_internal_exchange.pub` to your authorised `~/.ssh/authorized_keys`, eg. with `cat <key_name> >> authorized_keys`
 
 > TROUBLE SHOOTING
 > you may have some issues with the solver tunnel failing (permission denied). One of these two steps should solve it
 > option 1: name the exchange key `id_rsa`.
 > option 2: copy the contents to authorized_keys from the compute nodes (from the ssh_dir `srun --qos=priority --pty bash; cat <key_name> >> authorized_keys;exit`)
 
-In addition you should have your .profile & .bashrc setup as per https://gitlab.pik-potsdam.de/rse/rsewiki/-/wikis/Cluster-Access
+> In addition you should have your .profile & .bashrc setup as per https://gitlab.pik-potsdam.de/rse/rsewiki/-/wikis/Cluster-Access
 and add `module load anaconda/2024.10` (or latest) to it
 
-## General installation
-- Create the conda environment in workflow/envs/ (maybe snakemake does it automatically for you provided the profile has use-conda) `conda env create --file path_to_env` (name is opt.). You can use either the pinned (exact) or the loose env (will install later package versions too).
-- If you experience issues switch to the pinned environment #TODO: generate
-- NB! you may need to modify atlite for things to work. Instructions to follow.
+## Getting the data
+You will need to enable data retrieval in the config
+```yaml
+enable:
+  build_cutout: false # if you want to build your own (requires ERA5 api access)
+  retrieve_cutout: true # if you want to download the pre-computed one from zenodo
+  retrieve_raster: true # get raster data
+```
+Some of the files are very large - expect a slow process!
 
+- You can also download the data manually and  copy it over to the correct folder. The source and target destinations are the input/output of the `fetch_` rules in `workflow/rules/fetch_data.smk`
+- **PIK HPC users only** you can also copy the data from other users
 
-## Getting the data
-- some of the data is downloaded by the snakemake workflow (e.g. cutouts). Just make sure te relevant config options are set to true if it is your first run
-- the shapely files can be generated with the build_province_shapes script
-- the [zeonodo bundle](https://zenodo.org/records/13987282) from the pypsa-China v3 comes with the data but in the old format, you will have to manually restructure it (or we can write a script)
-- [PIK HPC users only] you can also copy the data from the tmp folder
+## Usage
 
-# Usage
-- If you are not running on the PIK hpc, you will need make a new profile for your machine under `config/<myprofile>/config.yaml`
+Detailed instructions in the documentation. 
+### local execution
+- local execution can be started (once the environment is activated) with `snakemake`
+- to customize the options, create `my_config.yaml` and launch `snakemake --configfile `my_config.yaml`. Configuration options are summarised in the documentation.
+### Remote execution
+This is relevant for slurm HPCs and other remotes with a submit job command
 - The workflow can be launched with `snakemake --profile config/compute_profile`
 - [PIK HPC users only] use `snakemake --profile config/pik_hpc_profile`
+- If you are not running on the PIK hpc, you will need make a new profile for your machine under `config/<compute_profile>/config.yaml`
+
+
+
 
-# Changelog
-- add path managers and merge duplicate functionalities
-- moved hardcoded data paths to config
-- restructure project to match snakemake8 guidelines & update to snakemake8
-- move hardcoded to centralised store constants.py (file paths still partially hardcoded)
-- start adding typing
-- add scripts to pull data
-- add derived_data/ folder and change target of data cleaning/prep steps for clarity
-
-# TODOs:
-- see issues
-- add pinned env
-- make a PR on atlite to add the squeeze array, which may be needed
-- integrate various into settings

config/default_config.yaml

@@ -4,7 +4,7 @@
 # DO NOT FORGET TO LOOK AT THE TECHNOLOGY CONFIGURATION FILES
 
 run:
-  name: 
+  name: "unamed_default_run"
 foresight: "overnight"
 
 paths:
@@ -178,7 +178,7 @@ Techs:
   coal_ccs_retrofit: true # currently myopic pathway only.  CC = co2 cap 
 
 ## add components (overwrites vre tech choice)
-heat_coupling: false
+heat_coupling: true
 add_biomass: True
 add_hydro: True
 add_H2: True
@@ -286,14 +286,6 @@ solving:
 
   mem: 80000 #memory in MB; 20 GB enough for 50+B+I+H2; 100 GB for 181+B+I+H2
 
-# brownfield options
-existing_capacities:
-  add: True # whether to add brownfield capacities
-  grouping_years: [1980, 1985, 1990, 1995, 2000, 2005, 2010, 2015, 2019, 2020, 2025, 2030, 2035, 2040, 2045, 2050, 2055, 2060]
-  collapse_years: False # Treat as a single unit when preparing & solving network 
-  threshold_capacity: 1 # TODO UNIT
-  techs: ['coal','CHP coal', 'CHP gas', 'OCGT', 'CCGT', 'solar', 'onwind', 'offwind', 'nuclear']
-
 # transmission options
 lines:
   line_length_factor: 1.25
@@ -303,8 +295,25 @@ lines:
 
 security:
   line_margin: 70 # max percent of line capacity
+  
+# brownfield options
+existing_capacities:
+  add: True # whether to add brownfield capacities
+  grouping_years: [1985, 1990, 1995, 2000, 2005, 2010, 2015, 2020, 2025, 2030, 2035, 2040, 2045, 2050, 2055, 2060]
+  collapse_years: False # Treat as a single unit when preparing & solving network 
+  threshold_capacity: 1 # TODO UNIT
+  techs: ['coal','CHP coal', 'CHP gas', 'OCGT', 'CCGT', 'solar', 'onwind', 'offwind', 'nuclear', "PHS"]
+  node_assignment: simple # simple | gps
+
+# brownfield options
+existing_capacities:
+  add: True # whether to add brownfield capacities
+  grouping_years: [1985, 1990, 1995, 2000, 2005, 2010, 2015, 2020, 2025, 2030, 2035, 2040, 2045, 2050, 2055, 2060]
+  collapse_years: False # Treat as a single unit when preparing & solving network 
+  threshold_capacity: 1 # TODO UNIT
+  techs: ['coal','CHP coal', 'CHP gas', 'OCGT', 'CCGT', 'solar', 'onwind', 'offwind', 'nuclear', "PHS"]
+  node_assignment: simple # simple | gps
 
-# contingencies (slower!) -> epsilon is fraction of vre gen or load for which reserves should exist
 operational_reserve:
   activate: false
   epsilon_load: 0.02

config/global_energy_monitor.yaml

@@ -0,0 +1,52 @@
+global_energy_monitor_plants:
+
+  # split GEM CHP plants into separate CHP technology
+  CHP:
+    split: true
+    aliases: ["CHP", "cogeneration", "cogen", "heat", "heating"]
+
+  tech_map: 
+  # map GEM Type based on technology
+  #       Type: technology: rename_to
+  # Set rename_to: null to drop
+  # default is a * wildcard (except pre-defined). 
+      gas: 
+          combined cycle: CCGT
+          default: OCGT
+      wind: 
+          Onshore: onwind
+          Offshore hard mount: offwind
+          Offshore floating: offwind
+      solar: 
+          Solar Thermal: solar thermal
+          PV: solar
+          default: null
+      # tricky because some conventional is also PHS
+      hydropower:
+          pumped storage: PHS
+          # run-of-river: ror
+          default: hydro
+          
+  # level of brownfield
+  base_year: 2020  # units retired before this date will be dropped
+  status:
+      - operating
+      - retired
+      # - construction
+  # for more brownfield add `construction`, `pre-construction` and/or `announced`
+
+  relevant_columns:
+    - "Type"
+    - "Plant name"
+    - "Technology"
+    - "Fuel"
+    - "CHP"
+    - "Capacity (MW)"
+    - "Start year"
+    - "Retired year"
+    - "Subregion"
+    - "Region"
+    - "Local area (taluk, county)"
+    - "Major area (prefecture, district)"
+    - "Subnational unit (state, province)"
+    - "Status"

config/pik_hpc_profile/config.yaml

@@ -91,7 +91,7 @@ set-resources:
     threads: 1
     partition: io # login
     qos: io
-  retrieve_build_up_raster:
+  retrieve_Build_up_raster:
     time: 300
     threads: 1
     partition: io # login
@@ -106,11 +106,27 @@ set-resources:
     threads: 1
     partition: io
     qos: io
+  retrieve_powerplants:
+    time: 300
+    threads: 2
+    partition: io
+    qos: io
+  retrieve_bathymetry_raster:
+    time: 300
+    threads: 1
+    partition: io
+    qos: io
   retrieve_Shrubland_raster:
     time: 300
     threads: 1
     partition: io
     qos: io
+  retrieve_cutout:
+    time: 300
+    threads: 1
+    partition: io
+    qos: io
+
 
 # GROUPS
 
@@ -123,10 +139,17 @@ groups:
   prepare_networks: prep_network
   add_existing_baseyear: prep_network2
   make_summary: summary
+  retrieve_cutout: retrieve
+  retrieve_region_shapes: retrieve
+  retrieve_rasters: retrieve 
+  retrieve_Build_up_raster: retrieve
+  retrieve_Grass_raster: retrieve
+  retrieve_Bare_raster: retrieve
 group-components:
   plot: 3
   build_load: 3
   prep_network: 3
   prep_network2: 3
+  retrieve: 1 # io queue
   summary: 3
-  prep_capacities: 3
+  prep_capacities: 3

config/plot_config.yaml

@@ -6,6 +6,11 @@ plotting:
   transparent: false
   statistics:
     clip_numerical_zeroes: true
+    add_labels:
+      - "capacity_factor"
+      - "installed_capacity"
+      - "lcoe"
+      - "mv_minus_lcoe"
   cost_panel:
     y_axis: "annualized system cost bEUR/a" # only true if investment period is one year
   cost_map: