Merge pull request pypsa-meets-earth#1086 from merge-pyspa-earth-sec

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Comment add_brownfield to bypass linter

* Add myopic test

* Revise irena and minor fixes

* Revise test name

* Add existing_heating

* implement review suggesstions

* bug fix in solve_network with reference case

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update submodule

* Add zeros for missing entries - aluminium

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adding missing templates

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix MissingOutputException

* fix TypeError (pypsa-meets-earth#321)

* fix TypeError

* Update scripts/prepare_gas_network.py

Co-authored-by: Davide Fioriti <[email protected]>

---------

Co-authored-by: Davide Fioriti <[email protected]>

* solve pandas deprecations

* Add Params for Rule add_export.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to Rule build_base_energy_totals

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to Rule build_base_industry_totals

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to Rule build_cop_profiles

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to Rule build_heat_demand

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to build_industrial_distribution_key

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to build_industry_demand

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params and urban_percentage effect all Rules

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to build_ship_profile

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params build_solar_thermal_profiles

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to build_temperature_profiles

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to copy_config

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params make_summary

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add to override_respot + panning_horizons wildcard

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Snakefile

* Update Snakefile

* Add Params to prepare_airports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to prepare_gas_network

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Params to prepare_db and energy_totals

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Snakefile

* Update add_export.py

* Add Params to build_population_layout

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* enhance the industry scripts and adapt the fuel aggregation

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove legacy EUR code

* remove legacy params form config

* remove legacy params form config

* Update build_base_industry_totals.py

* Update build_base_energy_totals.py

* omit double transpose

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove plotting from workflow

* remove plotting from prepare_gas_network

* Test routine: add Makefile
CI: consolidate ci.yaml files

* CI: bump cache number, remove os prefixes and labels
env: remove ipopt restriction

* Makefile: add tutorial yaml as additional config

* snakefile: use config.default as basis
ci: use tutorial config as secondary config

* snakefile: fix databundle config path

* debug: print out downloaded files

* snakefile: use renewables as bases for used cutouts

* Include scenario management

* Add scenario in tutorial

* Update submodule

* Add shared_cutouts

* Fix missing RDIR_PE

* fix IndexError (pypsa-meets-earth#326)

* fix IndexError

* '.' gets only added when len(id) > 3

* fix with layer_id

* Update scripts/prepare_gas_network.py

Co-authored-by: Davide Fioriti <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix index error when layer_id == 0

---------

Co-authored-by: Davide Fioriti <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* solve_network: modularize solver_options to allow for incremental config changes
config: harmonize and apply yaml linting

* refactor configs: boil down to effective diff in tutorial yaml and scenarios yaml

* test: remove config no_progress

* ci: disable for windows

* helpers: use copy_default_files

* config: make config.default + config.tutorial match old config.tutorial

* solve_network: fix options assignment in prepare

* add build_heating_distribution

* Update licensing

* Set location to Earth

* Update cluster pop

* Fix typos

* Bugfix to skip h2 pipelines with missing buses

* Bugfix h2_network loc

* reintroduce config.tutorial.yaml as basis (possibly revert this commit later to reenable config.default)

* doc: update testing documentation [skip ci]

* add comments to makefile

* Update myopic test

* Bugfix none location in build_industrial_database

* Revise run_test myopic name

* revert transpose

* delete plot_network_eur.py

* minor bug in build_base_energy_totals

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update build_base_industry_totals.py

* adapt build_base_energy and build_base_industry to params

* add missing param in snakefile

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revise myopic test file

* fix param in snakefile

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reset config.pypsa-earth.yaml

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Snakefile

* handle exception to avoid reference before assignment error

* Add different scenario name for subworkflow

* Fix myopic test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update config.test_myopic.yaml to include new parameter

* replace snakemake subworkflow by submodule; fix path relations

* fill NaN without international bunkers

* remove factor for gas emissions from residential and services sector

* account for multi-country-cases

* snakefile: replace rdir by resdir for compatibility with pypsa-earth

* Update build_base_industry_totals.py

* Update Snakefile

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update build_base_industry_totals.py

* modified config to fix nan objective value

* print objective value to terminal

* fix for urls not working

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added license for datasets

* Update README.md

* Update README.md

* Final README adaptations before v0.2 release (pypsa-meets-earth#364)

* Adding the updated network representation figure

* Delete docs/SCPE_v0.2.pdf

* Include new network representaion figure as png

* Update README.md

* Embed the new network figure in the readme

* Delete outdated incomplete network configurations

* updated pypsa-earth submodule to v0.4.0

* adjust modular solver options

* resolve global paths

* workflow: use global root directory to avoid recursive upwards chdir

* update pypsa-earth commit

* revert changes to config.default and config.tutorial priority

* udpate pypsa-earth

* consolidate CI yaml

* yamllint: align formatting and config

* Snakefile: fix SDIR and RESDIR path ending

* ci.yaml include git submodule

* fix duplicated "/"

* remove config.pypsa-earth.yaml in favour of actual pypsa-earth default config

* remove pypsa-earth.config from copy_config

* ci: use one core to better track log

* test: use more core

* make yamllint compatible

* build_renewable_profiles: use local client in order to suppress verbose dask output

* update submodule

* config: consolidate clustering key
snakefile: fix cost retrieval

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* solve_networks: fix network reference for sector coupled case

* address review comments

* config.myopic.yaml: revert country order again

* Update README.md

Co-authored-by: Davide Fioriti <[email protected]>

* address davides comments (first round)

* consolidate current working directory; considate helpers scripts

* fix reference to helpers script; fix fiona version

* env: update ppm version

* build_industrial_database: make retrieval robust against restrictive permissions

* follow up: consolidate restrictive url retrievals

* helpers: reinsert gadm functions from pypsa-earth-sec due to discrepancies (solve those later)

* env: temporarily install new earth osm version from pypi

* ci: restrict ipopt for windows

* env: follow up

* ci: roll back and disable windows

* _helpers: try make content_retrieval more robust

* Update README.md

Co-authored-by: Ekaterina <[email protected]>

* config: bump version; add `allow_scenario_failure` flag; remove `base` cutout comment

* Snakefile: make cutout path consistent for sector-coupled version
config.default: make hydro extedable again

* helpers: fix numpy random usage

* properly remove submodule

* add version tag to all configs

* env: update powerplantmatching

* add missing config version tag

* update powerplantmatching

* follow up: move ppm installation to pip while conda is not out

* readme: add description on running previous models

* helpers: add HTTPError as allowed exception

* config: remove lifetime from config.default

* harmonize cost calculation in sector coupled model

* bump version tag in readme [skip ci]

* update README to account for Hazem's comments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: davide-f <[email protected]>
Co-authored-by: Hazem-IEG <[email protected]>
Co-authored-by: Davide Fioriti <[email protected]>
Co-authored-by: Hazem <[email protected]>
Co-authored-by: Eddy Jalbout <[email protected]>
Co-authored-by: finozzifa <[email protected]>
Co-authored-by: Anton Achhammer <[email protected]>
Co-authored-by: Anton Achhammer <[email protected]>
Co-authored-by: Fabrizio Finozzi <[email protected]>
Co-authored-by: Eddy-JV <[email protected]>
Co-authored-by: energyLS <[email protected]>
Co-authored-by: cpschau <[email protected]>
Co-authored-by: Emmanuel Bolarinwa <[email protected]>
Co-authored-by: Ekaterina <[email protected]>

.github/workflows/ci.yml

@@ -0,0 +1,72 @@
+name: CI
+
+on:
+  push:
+    branches:
+    - main
+  pull_request:
+    branches:
+    - main
+  schedule:
+  - cron: "0 5 * * TUE"
+
+env:
+  CACHE_NUMBER: 2  # Change this value to manually reset the environment cache
+
+jobs:
+  build:
+    strategy:
+      fail-fast: false
+      max-parallel: 3
+      matrix:
+        os:
+        - ubuntu-latest
+        - macos-latest
+        # - windows-latest
+
+    runs-on: ${{ matrix.os }}
+
+    defaults:
+      run:
+        shell: bash -l {0}
+
+    steps:
+    - uses: actions/checkout@v2
+
+
+    - name: Setup micromamba
+      uses: mamba-org/setup-micromamba@v1
+      with:
+        micromamba-version: latest
+        environment-file: envs/environment.yaml
+        log-level: debug
+        init-shell: bash
+        cache-environment: true
+        cache-downloads: true
+
+
+    - name: Set cache dates
+      run: |
+        echo "WEEK=$(date +'%Y%U')" >> $GITHUB_ENV
+
+    - name: Cache data and cutouts folders
+      uses: actions/cache@v3
+      with:
+        path: |
+          data
+          cutouts
+        key: data-cutouts-${{ env.WEEK }}-${{ env.CACHE_NUMBER }}
+
+
+    - name: Conda list
+      run: conda list
+
+    - name: Run Test
+      run: make test
+
+      # - name: Test plotting and summaries
+      #   run: |
+      #     snakemake --cores all plot_all_p_nom
+      #     snakemake --cores all plot_all_summaries
+      #     snakemake --cores all make_all_summaries
+      #     rm -rf resources/*.nc resources/*.geojson resources/*.h5 networks results