Verify the effect of using/disregarding clustering by subnetworks in cluster_network #531

davide-f · 2022-12-02T23:24:32Z

Describe the Bug

In cluster_network L243 and L403, the clustering may be by country AND sub_network, as done in PyPSA-Eur.
In PyPSA-Earth we disregard this for simplicity, yet it now needs some study.
Moreover, the clustering shall address the improper matching of buses of different carrier types.

Consequences shall be investigated at least on:

a sample test case such as the tutorial NG+BJ
another country example with DC lines (es DRC or Brazil)

The tests that shall be done are:

verifying the total capacities of generation and networks after add_electricity (hence elec.nc), after the simplify (elec_s{}.nc} and after the clustering
verifying what happens when DC lines belong to the clustered nodes. Most likely, there will be the need for some fix here.

This issue is a remark that leverages on #506 and #530

ekatef · 2022-12-07T23:13:31Z

There is an issue for a run considering China under a conservative assumption "all AC -> DC" due to mismatch of the sub-networks with AssertionError: In Bus cluster sub_network the values of attribute sub_network do not agree. The issue may be resolved by an increase of the clusters number from 10 to 20.

The network looks like that:

The full error listing:

INFO:snakemake.logging:
This is the repository path:  ~pypsa-earth
Had to go 0 folder(s) up.
INFO:pypsa.io:Imported network elec_s.nc has buses, carriers, generators, lines, loads, storage_units
Traceback (most recent call last):
  File "~pypsa-earth/.snakemake/scripts/tmp85yv8yyb.cluster_network.py", line 581, in <module>
    clustering = clustering_for_n_clusters(
  File "~pypsa-earth/.snakemake/scripts/tmp85yv8yyb.cluster_network.py", line 448, in clustering_for_n_clusters
    clustering = get_clustering_from_busmap(
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 379, in get_clustering_from_busmap
    buses, linemap, linemap_p, linemap_n, lines, lines_t = get_buses_linemap_and_lines(
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 344, in get_buses_linemap_and_lines
    lines, linemap_p, linemap_n, linemap, lines_t = aggregatelines(
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 308, in aggregatelines
    lines = interlines_c.groupby(["bus0_s", "bus1_s"]).apply(aggregatelinegroup)
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1558, in apply
    result = self._python_apply_general(f, self._selected_obj)
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1610, in _python_apply_general
    values, mutated = self.grouper.apply(f, data, self.axis)
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 839, in apply
    res = f(group)
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 301, in aggregatelinegroup
    sub_network=consense["sub_network"](l["sub_network"]),
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 51, in consense
    assert (
AssertionError: In Bus cluster sub_network the values of attribute sub_network do not agree:
Line
1020046762-1     0
201000349-1      0
313954981-1      0
313954976-1      0
670958477-1      0
317593825-1      0
1058226063-1     0
671196699-1      0
1026941772-1    33
670958474-1      0
670958479-1      0
863798817-1      0
614954284-1     33
Name: sub_network, dtype: object
[Wed Dec  7 14:53:36 2022]
INFO:snakemake.logging:[Wed Dec  7 14:53:36 2022]
Error in rule cluster_network:
    jobid: 4
    input: networks/elec_s.nc, resources/shapes/country_shapes.geojson, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/shapes/gadm_shapes.geojson, data/costs.csv
    output: networks/elec_s_10.nc, resources/bus_regions/regions_onshore_elec_s_10.geojson, resources/bus_regions/regions_offshore_elec_s_10.geojson, resources/bus_regions/busmap_elec_s_10.csv, resources/bus_regions/linemap_elec_s_10.csv
    log: logs/cluster_network/elec_s_10.log (check log file(s) for error message)

ERROR:snakemake.logging:Error in rule cluster_network:
    jobid: 4
    input: networks/elec_s.nc, resources/shapes/country_shapes.geojson, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/shapes/gadm_shapes.geojson, data/costs.csv
    output: networks/elec_s_10.nc, resources/bus_regions/regions_onshore_elec_s_10.geojson, resources/bus_regions/regions_offshore_elec_s_10.geojson, resources/bus_regions/busmap_elec_s_10.csv, resources/bus_regions/linemap_elec_s_10.csv
    log: logs/cluster_network/elec_s_10.log (check log file(s) for error message)

RuleException:
CalledProcessError in line 526 of ~pypsa-earth/Snakefile:
Command 'set -euo pipefail;  /Users/ekatef/miniconda3/envs/pypsa-earth2/bin/python3.10 ~pypsa-earth/.snakemake/scripts/tmp85yv8yyb.cluster_network.py' returned non-zero exit status 1.
  File "~pypsa-earth/Snakefile", line 526, in __rule_cluster_network
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in line 526 of ~pypsa-earth/Snakefile:
Command 'set -euo pipefail;  /Users/ekatef/miniconda3/envs/pypsa-earth2/bin/python3.10 ~pypsa-earth/.snakemake/scripts/tmp85yv8yyb.cluster_network.py' returned non-zero exit status 1.
  File "~pypsa-earth/Snakefile", line 526, in __rule_cluster_network
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-12-07T101257.360751.snakemake.log

davide-f · 2023-01-30T09:37:21Z

Same problem arises for Azerbaijan; 5 clusters:

INFO:pypsa.io:Imported network elec_s.nc has buses, carriers, generators, lines, loads, storage_units
Traceback (most recent call last):
  File "/data/davidef/git_world/pypsa-earth/.snakemake/scripts/tmp_d6dfkjw.cluster_network.py", line 601, in <module>
    clustering = clustering_for_n_clusters(
  File "/data/davidef/git_world/pypsa-earth/.snakemake/scripts/tmp_d6dfkjw.cluster_network.py", line 466, in clustering_for_n_clusters
    clustering = get_clustering_from_busmap(
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 378, in get_clustering_from_busmap
    buses, linemap, linemap_p, linemap_n, lines, lines_t = get_buses_linemap_and_lines(
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 343, in get_buses_linemap_and_lines
    lines, linemap_p, linemap_n, linemap, lines_t = aggregatelines(
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 307, in aggregatelines
    lines = interlines_c.groupby(["bus0_s", "bus1_s"]).apply(aggregatelinegroup)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pandas/core/groupby/groupby.py", line 1567, in apply
    result = self._python_apply_general(f, self._selected_obj)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pandas/core/groupby/groupby.py", line 1629, in _python_apply_general
    values, mutated = self.grouper.apply(f, data, self.axis)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pandas/core/groupby/ops.py", line 839, in apply
    res = f(group)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 300, in aggregatelinegroup
    sub_network=consense["sub_network"](l["sub_network"]),
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 50, in consense
    assert (
AssertionError: In Bus cluster sub_network the values of attribute sub_network do not agree:
Line
317780756_0    0
533034445_0    0
677666942_0    9
317109714_0    0
908699840_0    0
908980191_1    0
909703866_0    9
Name: sub_network, dtype: object

ekatef · 2023-02-01T16:38:48Z

It looks like the the isolated buses can remain in the network model after simplification. Our current approach (when we apply grouping by country only) captures such isolates buses when doing aggregation while PyPSA-Europe approach (when grouping by country and subnetwork is applied) just keep such buses as they are

The graphic example for Azerbaijan (bus colors by sub_network) look as follows.
Buses colouring:

green is prescribed to the sub_network with the maximal buses number,
violet -- to the subnetwork with three buses,
red -- to all the other buses.

Caution: the pictures in this comment were obtained with PyPSA.Network.plot() setting some jitter value. However, the jitter values are being changed randomly as seed() is not defined in the plot() source. That means that each of the plots bellow contain some random noise in terms of geographic location of the buses

network after simplification

network after clustering with country+sub_network grouping
(that is the approach being currently used in PyPSA-Eur)

network after clustering with country-only grouping
(that is our current approach introduced in Fix not-connected nodes #530)

cpschau · 2023-02-03T16:37:31Z

For the solved Brazilian network with 10 clusters a comparison has been made between three different grouping approaches:

a) Grouping by country after modifying the clean osm data and forcing all lines and substations to be AC
b) Grouping by country and substation
c) Grouping by country and carrier

a) Applying the quick fix which is described above the resulting network comprises of 30 connected buses (with 10 different locations), which correponds to the number of clusters chosen times the amount of different carriers (AC, battery, H2).

b) For the second approach a similiar behavior to the one described in the comments before was observed. Grouping by country and sub_network still allows for isolated buses. Furthermore the network's amount of buses exceeds the one which is yielded in the first approach. It consists of 219 buses, of which 25 are connected to each other via line components.

c) Grouping by country and carrier yields a network which comprises twice as many nodes as the one generated using the first approach. The AC nodes are connected by line components, the DC nodes by link components. Moreover there are link components, which connect AC and DC nodes.

ekatef · 2023-02-09T15:40:12Z

According to the discussion of this issue with @davide-f (Davide thanks a lot for insightful comments!), an additional analysis has been done to estimate generation and demand attached to the isolated buses. The network for Azerbaijan after simplification has been taken for this assessment considering two different tolerance values (group_tolerance_buses).

Quantification of the generation and demand attached to the isolated buses

group_tolerance_buses: 500m

generation

5100.6 MW for the connected buses
714.2 MW for the isolated ones

demand

1676.5 for the connected buses
1156.2 for the isolated ones

group_tolerance_buses: 10km

generation

5787.45 for the connected buses
27.45 for the isolated ones

demand

2315.4 for the connected buses
517.3 for the isolated ones

Network outline

group_tolerance_buses: 500m

group_tolerance_buses: 10km

ekatef · 2023-02-09T15:53:08Z

It looks like this issue is connected both with the clustering and simplification procedures and eventually is influenced with the osm cleaning approach. As discussed with @davide-f the plan to tackle it is

implement updates on simplification and clustering aligning with recent PyPSA-Eur developments (Update simplify network along PyPSA-Eur dev #592 and Add hierachical clustering and config adjustment #404);
add a fix for isolated buses treatment into simplify_network considering two options:
- the option to disregard buses below a given threshold;
- the option to merge buses below another threshold;
eventually further improve clean_osm_data to avoid data loss (the cleaning approach has been fixed by 0 column and frequency is not properly cleaned in 'clean_osm_data.py' #469, some more work may be needed in split_cells_multiple).

ekatef · 2023-02-09T22:00:46Z

The comment corrected. @davide-f thanks a lot for cross-checking!

Actually, I remember that we have discussed improvements of split_cells_multiple but it looks like that has not been written down yet. Does it probably make sense to add an issue on that?

ekatef · 2023-02-10T21:55:43Z

Just to keep a track of this issue: introducing the recent updates from PyPSA-Eur (#404 and #592) leads to resolving the issue observed for Azerbaijan model with 5 clusters but isolated buses still remain:

Generally, the updates give a bit more realistic spatial distribution of the nodes:

original version

updated version

That is hardly very meaningful from the modeling perspective but nice for the eye.

davide-f added the bug Something isn't working label Dec 2, 2022

ekatef mentioned this issue Dec 17, 2022

Fix geometry issues #532

Closed

12 tasks

ekatef mentioned this issue Dec 30, 2022

Add run scenarios #543

Merged

7 tasks

davide-f added this to the Country-wise model milestone Feb 1, 2023

ekatef mentioned this issue Feb 10, 2023

Update simplify network along PyPSA-Eur dev #597

Merged

7 tasks

ekatef mentioned this issue Feb 26, 2023

Fix islanded buses and add clustering by networks #614

Closed

7 tasks

davide-f assigned ekatef Mar 9, 2023

davide-f closed this as completed Mar 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Verify the effect of using/disregarding clustering by subnetworks in cluster_network #531

Verify the effect of using/disregarding clustering by subnetworks in cluster_network #531

davide-f commented Dec 2, 2022 •

edited

Loading

ekatef commented Dec 7, 2022 •

edited

Loading

davide-f commented Jan 30, 2023

ekatef commented Feb 1, 2023 •

edited

Loading

cpschau commented Feb 3, 2023

ekatef commented Feb 9, 2023 •

edited

Loading

ekatef commented Feb 9, 2023 •

edited

Loading

ekatef commented Feb 9, 2023

ekatef commented Feb 10, 2023

Verify the effect of using/disregarding clustering by subnetworks in cluster_network #531

Verify the effect of using/disregarding clustering by subnetworks in cluster_network #531

Comments

davide-f commented Dec 2, 2022 • edited Loading

Describe the Bug

ekatef commented Dec 7, 2022 • edited Loading

davide-f commented Jan 30, 2023

ekatef commented Feb 1, 2023 • edited Loading

cpschau commented Feb 3, 2023

ekatef commented Feb 9, 2023 • edited Loading

Quantification of the generation and demand attached to the isolated buses

group_tolerance_buses: 500m

generation

demand

group_tolerance_buses: 10km

generation

demand

Network outline

group_tolerance_buses: 500m

group_tolerance_buses: 10km

ekatef commented Feb 9, 2023 • edited Loading

ekatef commented Feb 9, 2023

ekatef commented Feb 10, 2023

davide-f commented Dec 2, 2022 •

edited

Loading

ekatef commented Dec 7, 2022 •

edited

Loading

ekatef commented Feb 1, 2023 •

edited

Loading

ekatef commented Feb 9, 2023 •

edited

Loading

ekatef commented Feb 9, 2023 •

edited

Loading