Fix broken equity constraints logic #679

ricnogfer · 2023-06-12T18:43:25Z

…A-Eur-Sec

Closes # (if applicable).

Changes proposed in this Pull Request

Checklist

I tested my contribution locally and it seems to work fine.
Code and workflow changes are sufficiently documented.
Changed dependencies are added to envs/environment.yaml.
Changes in configuration options are added in all of config.default.yaml.
Changes in configuration options are also documented in doc/configtables/*.csv.
A release note doc/release_notes.rst is added.

…A-Eur-Sec

ricnogfer · 2023-06-12T18:56:27Z

@pz-max: yes, to_xarray() is needed but only for variables ggrouper and sgrouper (when these are consumed by groupby()). The other variable, lgrouper, does not need to_xarray().

When running the original solve_network.py, this is the error that is thrown:

INFO:pypsa.io:Imported network elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050.nc has buses, carriers, generators, global_constraints, lines, links, loads, storage_units, stores
INFO:__main__:No expandable lines found. Skipping iterative solving.
Traceback (most recent call last):
  File "/home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmphcji4yu1/.snakemake/scripts/tmpv3xlw8ms.solve_network.py", line 684, in <module>
    n = solve_network(
  File "/home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmphcji4yu1/.snakemake/scripts/tmpv3xlw8ms.solve_network.py", line 619, in solve_network
    status, condition = n.optimize(
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pypsa/optimization/optimize.py", line 479, in __call__
    return optimize(self._parent, *args, **kwargs)
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pypsa/optimization/optimize.py", line 457, in optimize
    extra_functionality(n, sns)
  File "/home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmphcji4yu1/.snakemake/scripts/tmpv3xlw8ms.solve_network.py", line 592, in extra_functionality
    add_EQ_constraints(n, o)
  File "/home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmphcji4yu1/.snakemake/scripts/tmpv3xlw8ms.solve_network.py", line 290, in add_EQ_constraints
    @ n.loads_t.p_set.groupby(lgrouper, axis=1).sum()
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/frame.py", line 8402, in groupby
    return DataFrameGroupBy(
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 965, in __init__
    grouper, exclusions, obj = get_grouper(
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/groupby/grouper.py", line 899, in get_grouper
    Grouping(
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/groupby/grouper.py", line 544, in __init__
    self.grouping_vector = index.map(self.grouping_vector)
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6361, in map
    new_values = self._map_values(mapper, na_action=na_action)
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/base.py", line 890, in _map_values
    new_values = map_f(values, mapper)
  File "pandas/_libs/lib.pyx", line 2924, in pandas._libs.lib.map_infer
TypeError: 'DataArray' object is not callable
Not cleaning up /home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmphcji4yu1/.snakemake/scripts/tmpv3xlw8ms.solve_network.py
[Mon Jun 12 20:17:41 2023]
Error in rule solve_sector_network:
    jobid: 0
    input: data/override_component_attrs, results/my_run/prenetworks/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050.nc, data/costs_2030.csv, results/my_run/configs/config.yaml
    output: results/my_run/postnetworks/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050.nc
    log: logs/my_run/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050_solver.log, logs/my_run/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050_python.log, logs/my_run/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050_memory.log (check log file(s) for error details)
    conda-env: /home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/conda/06dc3f75cc469070d08036b51833b529_

Removing completely to_xarray() will throw a different type of error and in another location:

INFO:pypsa.io:Imported network elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050.nc has buses, carriers, generators, global_constraints, lines, links, loads, storage_units, stores
INFO:__main__:No expandable lines found. Skipping iterative solving.
Traceback (most recent call last):
  File "/home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmp6h2yatv0/.snakemake/scripts/tmpkh11e0_u.solve_network.py", line 684, in <module>
    n = solve_network(
  File "/home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmp6h2yatv0/.snakemake/scripts/tmpkh11e0_u.solve_network.py", line 619, in solve_network
    status, condition = n.optimize(
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pypsa/optimization/optimize.py", line 479, in __call__
    return optimize(self._parent, *args, **kwargs)
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pypsa/optimization/optimize.py", line 457, in optimize
    extra_functionality(n, sns)
  File "/home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmp6h2yatv0/.snakemake/scripts/tmpkh11e0_u.solve_network.py", line 592, in extra_functionality
    add_EQ_constraints(n, o)
  File "/home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmp6h2yatv0/.snakemake/scripts/tmpkh11e0_u.solve_network.py", line 301, in add_EQ_constraints
    .groupby(ggrouper)
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/linopy/expressions.py", line 674, in groupby
    groups = ds.groupby(
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/xarray/core/dataset.py", line 8950, in groupby
    return DatasetGroupBy(
  File "/home/ricnogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/xarray/core/groupby.py", line 368, in __init__
    raise TypeError(
TypeError: `group` must be an xarray.DataArray or the name of an xarray variable or dimension. Received Generator
AL1 0 offwind-ac                   AL1 0
AL1 0 onwind                       AL1 0
AL1 0 ror                          AL1 0
AL1 0 solar                        AL1 0
AT1 0 onwind                       AT1 0
                             ...        
RO1 0 solar rooftop    RO1 0 low voltage
RS1 0 solar rooftop    RS1 0 low voltage
SE2 0 solar rooftop    SE2 0 low voltage
SI1 0 solar rooftop    SI1 0 low voltage
SK1 0 solar rooftop    SK1 0 low voltage
Name: bus, Length: 378, dtype: object instead.
Not cleaning up /home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmp6h2yatv0/.snakemake/scripts/tmpkh11e0_u.solve_network.py
[Mon Jun 12 20:24:09 2023]
Error in rule solve_sector_network:
    jobid: 0
    input: data/override_component_attrs, results/my_run/prenetworks/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050.nc, data/costs_2030.csv, results/my_run/configs/config.yaml
    output: results/my_run/postnetworks/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050.nc
    log: logs/my_run/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050_solver.log, logs/my_run/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050_python.log, logs/my_run/elec_s_37_lv1.0__Co2L0-EQ0.7-168H-T-H-B-I-A-solar+p3-dist1_2050_memory.log (check log file(s) for error details)
    conda-env: /home/ricnogfer/workspace/pypsa-eur_merge/.snakemake/conda/06dc3f75cc469070d08036b51833b529_

pz-max · 2023-06-12T19:20:03Z

@ricnogfer , that for the excellent report. Are your PR changes fixing everything locally now?

ricnogfer · 2023-06-12T19:23:44Z

@pz-max, you are welcome.

Yes, the present PR fixes everything locally. Hope it helps!

pz-max · 2023-06-12T19:35:22Z

@ricnogfer checking the other constraints, e.g.

pypsa-eur/scripts/solve_network.py

Line 242 in b7aa122

maximum = xr.DataArray(agg_p_nom_minmax["max"].dropna()).rename(dim_0="group")

As you mentioned, our current error without using any to_xarray() is:

TypeError: `group` must be an xarray.DataArray or the name of an xarray variable or dimension. Received Generator

Something like: xr.DataArray(sgrouper).rename(dim_0="group") could help.

Actually, could you report the n.objective for both solutions?
One with your approach, one with the xr.DataArray() approach.
Sometimes things work with the constraint but lead to different results.
The xr.DataArray() might also be good for consistency, but your solution seems more compact and might be potentially better if both solutions lead to the same objective.

ricnogfer · 2023-06-13T06:25:21Z

@pz-max, tried to use the alternative that you pointed out (xr.DataArray(sgrouper).rename(dim_0="group")) but, unfortunately, without success. It throws the following error:

INFO:__main__:No expandable lines found. Skipping iterative solving.
Traceback (most recent call last):
  File "/home/rignogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmpzgv7r8n4/.snakemake/scripts/tmp66fhp6xt.solve_network.py", line 674, in <module>
    n = solve_network(
  File "/home/rignogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmpzgv7r8n4/.snakemake/scripts/tmp66fhp6xt.solve_network.py", line 607, in solve_network
    status, condition = n.optimize(
  File "/home/rignogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pypsa/optimization/optimize.py", line 479, in __call__
    return optimize(self._parent, *args, **kwargs)
  File "/home/rignogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/pypsa/optimization/optimize.py", line 457, in optimize
    extra_functionality(n, sns)
  File "/home/rignogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmpzgv7r8n4/.snakemake/scripts/tmp66fhp6xt.solve_network.py", line 581, in extra_functionality
    add_EQ_constraints(n, o)
  File "/home/rignogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmpzgv7r8n4/.snakemake/scripts/tmp66fhp6xt.solve_network.py", line 301, in add_EQ_constraints
    .groupby(xr.DataArray(ggrouper).rename(dim_0="group"))
  File "/home/rignogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/xarray/core/dataarray.py", line 2359, in rename
    dataset = self._to_temp_dataset()._rename(name_dict)
  File "/home/rignogfer/workspace/anaconda3/envs/pypsa-eur/lib/python3.10/site-packages/xarray/core/dataset.py", line 3609, in _rename
    raise ValueError(
ValueError: cannot rename 'dim_0' because it is not a variable or dimension in this dataset
Not cleaning up /home/rignogfer/workspace/pypsa-eur_merge/.snakemake/shadow/tmpzgv7r8n4/.snakemake/scripts/tmp66fhp6xt.solve_network.py

pz-max · 2023-06-13T08:49:40Z

@ricnogfer, sure. The code snippet I showed was just an example on how other parts are doing that.
You need to debug and analyse:

where you use the DataArray
whats the dimension to use etc.

koen-vg · 2023-06-16T08:01:10Z

If there's any findings on grouping and xarrays that I should incorporate into #659, let me know. That version of the equity constraint seems to work fine but it's possible I missed something.

Fixed issue (most likely) introduced when merging PyPSA-Eur with PyPS…

b7aa122

…A-Eur-Sec

ricnogfer mentioned this pull request Jun 12, 2023

Fix broken equity constraints logic #677

Closed

6 tasks

fneum added 2 commits July 18, 2023 14:07

Merge branch 'master' into master

268391e

Merge branch 'master' into master

5cab4f4

fneum merged commit 68dc44d into PyPSA:master Jul 23, 2023
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix broken equity constraints logic #679

Fix broken equity constraints logic #679

ricnogfer commented Jun 12, 2023

ricnogfer commented Jun 12, 2023

pz-max commented Jun 12, 2023

ricnogfer commented Jun 12, 2023 •

edited

Loading

pz-max commented Jun 12, 2023

ricnogfer commented Jun 13, 2023

pz-max commented Jun 13, 2023

koen-vg commented Jun 16, 2023

Fix broken equity constraints logic #679

Fix broken equity constraints logic #679

Conversation

ricnogfer commented Jun 12, 2023

Changes proposed in this Pull Request

Checklist

ricnogfer commented Jun 12, 2023

pz-max commented Jun 12, 2023

ricnogfer commented Jun 12, 2023 • edited Loading

pz-max commented Jun 12, 2023

ricnogfer commented Jun 13, 2023

pz-max commented Jun 13, 2023

koen-vg commented Jun 16, 2023

ricnogfer commented Jun 12, 2023 •

edited

Loading