Revise network simplification to account for DC lines #743

ekatef · 2023-05-23T22:17:42Z

Closes #621

Changes proposed in this Pull Request

DC lines are added to a simplification procedure to be treated in the same way as links.

Checklist

NB Testing configuration assumed dc_as_lines for all the countries above and gadm clustesing for CD, PH and NG (as gadm clustering seems currently to be more prone to troubles) while usual hierarchical clustering was used for JP, IN and CN to use pre-calculated shapes keeping computational costs moderate.

ekatef · 2023-05-23T23:28:25Z

Current implementation seems to reproduce desirable topology transformations, but should be adjusted in part of related to equivalent parameters calculation of the resulted links

for more information, see https://pre-commit.ci

ekatef · 2023-06-17T21:23:48Z

Detailed checking of the network structure for CD has revealed some issues related to simplification of "meshed" DC grid. HVDC part consists of two parallel lines. Geometrically each of the parallel lines connects the same buses but has slightly different coordinates:

These two lines get merged in course of simplification, except a single segment where lines duplication is kept:

(different colours are used just to highlight different lines of the same DC part)

ekatef · 2023-06-17T21:35:13Z

The suggested modification of simplify_network works with hvdc_as_lines: true in the same way as hvdc_as_lines: false. To check it an "analytical" approach should be used as simple visual check is a bit tricky due to overlapped geometries. A good indicator is n.lines[n.lines.dc][["bus0", "bus1"]] vs n.links][["bus0", "bus1"]] which is for a simplified network as follows:

before modification

Line,bus0,bus1
49423895_0,30,36
224147292_0,36,51
261165614_0,37,53
383097966_0,37,52
460419487_0,30,36
461638193_0,36,51
626214770_0,44,37
778351007_0,51,8
778351009_0,51,8
778351020_0,44,37
979570585_0,53,52
979570586_0,52,30
979570588_0,52,30

Link,bus0,bus1
convert_3_8,8,7
convert_29_44,44,43

after modification

Line,bus0,bus1
383097966_0,37,52

Link,bus0,bus1
261165614_0+1,52,37
460419487_0+4,52,7
778351020_0+1,43,37

The PR ensures that a result of the network simplification is the same regardsless of hvdc_as_lines value with the only difference that for the buses 37,52 a duplicating connection is a (second) link if hvdc_as_lines: false and a DC line otherwise

So, the next step is to resolve an issue with the duplication remained in the simplified network

ekatef · 2023-06-18T15:15:30Z

It looks like split_links collects chains of links starting from each supernode going along a graph to be further used for merging into a single link. All parallel links (like those between 51 and 8 buses, for example) are being captured by adj() function

So, merging 37-53 + 53-52 is in scope of the algorithm, while adding 37-52 seems to need to introduce some modifications into it

davide-f · 2023-06-18T15:20:05Z

Hello @ekatef and very interesting contributions! Much needed.
I understand this is a work in progress, but I plan to give it a look for a preliminary review later.
As a quick comment, what do you mean with 'merging 37-53 + 53-52 is in scope of the algorithm, while adding 37-52 seems to need to introduce some modifications into it'?
Especially the second half of the sentence.

Many thanks! :)

ekatef · 2023-06-18T15:28:21Z

Hello @ekatef and very interesting contributions! Much needed. I understand this is a work in progress, but I plan to give it a look for a preliminary review later. As a quick comment, what do you mean with 'merging 37-53 + 53-52 is in scope of the algorithm, while adding 37-52 seems to need to introduce some modifications into it'? Especially the second half of the sentence.

Many thanks! :)

Hello @davide-f and many thanks for support, as usually! :)

The numbers refer to the second picture in the comment above. I have been thinking that lines duplication left after simplification could be a bug which can be fixed by some simple trick. But it looks some deeper changes are needed if we are not happy with the resulted configuration of a simplified network. The question is if it's really needed :D

ekatef · 2023-06-18T17:00:43Z

As discussed with @davide-f, the best approach to tackle this issue with the parallel links remained after simplification might be just to apply simplify_links twice

…s implementation

ekatef · 2023-06-25T19:36:15Z

It looks like we have previously missed some modifications in calculations on transmission costs needed to make representation of HVDC as lines work properly. In particular:

underwater_fraction should be calculated for n.lines with "DC" carries in base_network;
when updating transmission costs in add_electricity, the parameters should be taken from n.lines instead of n.links

Have drafted possible implementation of these changes in base_network and add_electricity. There is some code repetition in both cases as I haven't been able to find a flexible enough way to distinguish between lines and links.

@davide-f would be very grateful if you could find time to have a look :)

davide-f · 2023-06-30T23:21:25Z

Hello katia :)
I've scanned it and there are quite interesting changes.
It is a big PR, so I may need some time to review entirely.
I'll go thought it in the weekend, I give some brief comments now as a preliminary element, but I need to go better in the code later on.

davide-f

Amazing job! :D
I've added some comments, on the chaining functionality in simplify_network I recall we talked about that and was ok.
I'll need another round maybe guided to address the functionality, but it was working well , right?
Fantastic Katia! :D

scripts/add_electricity.py

scripts/base_network.py

scripts/add_electricity.py

ekatef · 2023-07-01T07:49:39Z

Thank you so much, Davide! There are still some troubles with functionality which I'm going to work on during the weekend. I'll ping you when ready :)

ekatef · 2023-07-06T16:47:45Z

Testing status: seems to work except a case when the augmentation option on:

in case of "CD" the troubles are linked with mixed-carrier issue (most likely due to lines/links overlap as explained in the first comments) [seems to be fixed, although there is still a mixed-carriers warning in case of expansion with [HVAC]];
in case of "PH" there are some NaNs introduced into the network [fixed]

So, augmentation needs better investigation while otherwise PR is ready for testing.

@Tomkourou if you wish to help with testing, you are absolutely welcome :)

davide-f

Very nice! I think this PR will be merged by next meeting , we are very close and great contribution! :)

So, is the algorithm finalized in your opinion? @ekatef

Is the only problem related to CD with augmentation by hvac?

scripts/add_electricity.py

scripts/base_network.py

scripts/cluster_network.py

scripts/simplify_network.py

davide-f · 2023-07-08T18:13:00Z

scripts/simplify_network.py

+    dc_as_links = not n.links.loc[n.links.carrier == "DC"].empty
+    dc_as_lines = not n.lines.loc[n.lines.carrier == "DC"].empty


were you looking for the flag in config maybe? In case, you could pass it as a parameter here, what do you think?

Yeah, I have been thinking about taking a flag directly from the config, but decided to try reading it from the network itself. The idea was to make it safer, although agree that it might be not clear.

Happy to revise if you think it'd be better to used a flag from the config

Mah... agree that we could avoid overcomplicating the code.
We could do the check internally, however, it may be safer to check for dc lines:
dc_as_links = not (n.lines.carrier == "DC").any()
and implicitly assume that when it is false than we used lines

As even when lines are included, converters represented as links will be necessary as well, therefore, with the current formulation dc_as_links will always be true.

the if below where you check: if dc_as_links: .... elif dc_as_lines, may be revised with: if dc_as_links: ... else: ... so replacing the elif with else.

I'm unsure if some issues may be fixed with this but i'm unsure

That is definitely cleaner :)
Thanks for explanations! Done

scripts/simplify_network.py

davide-f · 2023-07-08T18:23:33Z

scripts/simplify_network.py

+                p_nom = (
+                    n.lines.loc[all_dc_lines]
+                    .apply(
+                        lambda x: x["v_nom"] * n.line_types.i_nom[x["type"]],
+                        axis=1,
+                        result_type="reduce",
+                    )
+                    .min()
+                )


Theoretically, there should be the column s_nom that contains the nominal capacity of the line: is it filled properly?
If so, you may use that column here.

However, you raised an interesting point: in simplify_network_to_380 we still assume that all lines are AC, it may be interesting to debug what happens there with DC lines.
In particular, the num_parallel is most likely calculated in a wrong way

Mmm, you are absolutely right: there is s_nom in n.lines, and it's value is exactly 3 times lower as compared with U*I used above. It feels like there is in fact some discrepancy with num_parallel

Interesting check, so, the comment here may be that instead of calculating the power by using V*I , it may be better to directly use s_nom.
So regarding the code, I recommend to use s_nom here directly, ideally fixing the calculation of V*I for dc lines.
What do you think?

scripts/simplify_network.py

ekatef · 2023-07-08T22:33:26Z

Very nice! I think this PR will be merged by next meeting , we are very close and great contribution! :)

So, is the algorithm finalized in your opinion? @ekatef

Is the only problem related to CD with augmentation by hvac?

@davide-f thank you for the amazing review! Very heplful :D

My feeling is that we are in fact quite close to finalise. From my perspective, augmentation is a major problem left. Apart of that, I'm also concerned with the recent testing results.

All the tested countries work with dc_as_lines: true and the results up to simplify_network looks quite nice. But clustering outputs are sometimes a bit puzzling. In particular, the usual hierarchical algorithm sometimes complains that n_cluster is too big and agrees to work only if n_cluster is decreased from 10 to only 2. Obviously, that leads to some strange outputs, for example:

base network

simplified network

clustered network

ekatef · 2023-07-08T22:45:46Z

Clustering results might be generally not very stable in respect to parameters selection like to remove stubs or not to remove :) But it would be a good idea to double-check that the clustering issues above are not being introduced with this PR. On the other hand, clustering problems were found for CN and IN only where old pre-calculated shapes were used. Probably, that has also played a role...

@davide-f I do remember that you have suggested to account for generators/loads attached to the nodes when making simplification. At the moment the algorithm doesn't do for that, while my feeling is that it could be worth to improve this

ekatef · 2023-07-09T21:45:20Z

I confirm that clustering troubles are not connected with this PR. A test for IN with the same configuration parameters has resulted in the same limitation on n_cluster. Not sure it's good news from the general perspective, but for this PR that means that basically the only point left is fixing of the equivalence calculations

davide-f · 2023-07-09T17:12:33Z

scripts/simplify_network.py

+                p_nom = (
+                    n.lines.loc[all_dc_lines]
+                    .apply(
+                        lambda x: x["v_nom"] * n.line_types.i_nom[x["type"]],
+                        axis=1,
+                        result_type="reduce",
+                    )
+                    .min()
+                )


Interesting check, so, the comment here may be that instead of calculating the power by using V*I , it may be better to directly use s_nom.
So regarding the code, I recommend to use s_nom here directly, ideally fixing the calculation of V*I for dc lines.
What do you think?

scripts/simplify_network.py

davide-f · 2023-07-12T13:57:54Z

Great @ekatef :D merged :)

ekatef · 2023-07-12T14:19:40Z

Great @ekatef :D merged :)

Awesome! Thank you so much @davide-f :D

ekatef added 2 commits May 24, 2023 01:00

Add filtering by frequency

44ee30d

Simplify topology of DC lines

64cf5d0

ekatef mentioned this pull request May 23, 2023

Fix a KeyError in simplify_links caused by misinterpretation of AC lines #740

Merged

14 tasks

ekatef and others added 4 commits June 2, 2023 00:18

Merge branch 'main' into simplify_dc_lines

874e777

[pre-commit.ci] auto fixes from pre-commit.com hooks

cb20186

for more information, see https://pre-commit.ci

Fix typos

e3a1ca5

Merge branch 'main' into simplify_dc_lines

a525321

ekatef added 2 commits June 18, 2023 00:53

Fix merge

97fa953

Filter supernodes by polarity

cd8b430

ekatef added 3 commits June 25, 2023 22:15

Add an underwater_fraction for DC lines

9847ca1

Account for DC lines implementation when attaching DC transmission costs

b7c2534

Revise costs calculations for underwater lines to account for DC line…

5082e04

…s implementation

ekatef added 3 commits June 25, 2023 23:41

Fix links check

baa817b

Add minor fixes

4a21588

Revise equivalence calculations

18e63ed

ekatef mentioned this pull request Jun 30, 2023

Fix augmentation of links #768

Closed

8 tasks

davide-f reviewed Jun 30, 2023

View reviewed changes

ekatef added 3 commits July 1, 2023 19:00

Fix type

9abb586

Generalize costs attachment for links and dc-lines

906588f

Generalize adding underwater parts for links and dc-lines

bd2bb13

Fix call of _set_links_underwater_fraction

6864268

ekatef added 2 commits July 7, 2023 23:53

Add a fix for augmentation

4b7ada4

Merge branch 'main' into simplify_dc_lines

5c377b1

ekatef marked this pull request as ready for review July 8, 2023 09:45

Add release note

0f727bf

davide-f reviewed Jul 8, 2023

View reviewed changes

ekatef added 6 commits July 8, 2023 23:46

Improve naming

de4a2bb

Simplify assignment

4d017cc

Fix import

505213b

Improve naming

50b558f

Fix duplication

08ac4e3

Simplify calculations

6af3be2

ekatef added 2 commits July 9, 2023 20:18

Remove an outdated comment

e44cdb1

Simplify definition of dc_as_links flag

29dc7d6

ekatef force-pushed the simplify_dc_lines branch from 782653b to 29dc7d6 Compare July 9, 2023 19:17

Fix typo

81678ef

ekatef added 3 commits July 10, 2023 23:11

Use s_nom parameter directly

8bed8c4

Account for HVDC part in equivalence calculations

b1a11bd

Revise calculations implementation

e3c87f7

davide-f reviewed Jul 11, 2023

View reviewed changes

ekatef added 2 commits July 12, 2023 01:10

Implement Davide's suggestion

121bf2d

Revise naming

0b9f7a8

davide-f merged commit 186d2d2 into pypsa-meets-earth:main Jul 12, 2023
3 checks passed

ekatef deleted the simplify_dc_lines branch November 14, 2023 22:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revise network simplification to account for DC lines #743

Revise network simplification to account for DC lines #743

ekatef commented May 23, 2023 •

edited

Loading

ekatef commented May 23, 2023

ekatef commented Jun 17, 2023

ekatef commented Jun 17, 2023 •

edited

Loading

ekatef commented Jun 18, 2023 •

edited

Loading

davide-f commented Jun 18, 2023

ekatef commented Jun 18, 2023 •

edited

Loading

ekatef commented Jun 18, 2023

ekatef commented Jun 25, 2023

davide-f commented Jun 30, 2023

davide-f left a comment

ekatef commented Jul 1, 2023

ekatef commented Jul 6, 2023 •

edited

Loading

davide-f left a comment •

edited

Loading

davide-f Jul 8, 2023

ekatef Jul 8, 2023

davide-f Jul 9, 2023

ekatef Jul 9, 2023

davide-f Jul 8, 2023

ekatef Jul 8, 2023 •

edited

Loading

davide-f Jul 9, 2023

ekatef Jul 10, 2023

ekatef commented Jul 8, 2023

ekatef commented Jul 8, 2023

ekatef commented Jul 9, 2023

davide-f Jul 9, 2023

davide-f commented Jul 12, 2023

ekatef commented Jul 12, 2023

		dc_as_links = not n.links.loc[n.links.carrier == "DC"].empty
		dc_as_lines = not n.lines.loc[n.lines.carrier == "DC"].empty

Revise network simplification to account for DC lines #743

Revise network simplification to account for DC lines #743

Conversation

ekatef commented May 23, 2023 • edited Loading

Closes #621

Changes proposed in this Pull Request

Checklist

ekatef commented May 23, 2023

ekatef commented Jun 17, 2023

ekatef commented Jun 17, 2023 • edited Loading

before modification

after modification

ekatef commented Jun 18, 2023 • edited Loading

davide-f commented Jun 18, 2023

ekatef commented Jun 18, 2023 • edited Loading

ekatef commented Jun 18, 2023

ekatef commented Jun 25, 2023

davide-f commented Jun 30, 2023

davide-f left a comment

Choose a reason for hiding this comment

ekatef commented Jul 1, 2023

ekatef commented Jul 6, 2023 • edited Loading

davide-f left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekatef Jul 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekatef commented Jul 8, 2023

base network

simplified network

clustered network

ekatef commented Jul 8, 2023

ekatef commented Jul 9, 2023

Choose a reason for hiding this comment

davide-f commented Jul 12, 2023

ekatef commented Jul 12, 2023

ekatef commented May 23, 2023 •

edited

Loading

ekatef commented Jun 17, 2023 •

edited

Loading

ekatef commented Jun 18, 2023 •

edited

Loading

ekatef commented Jun 18, 2023 •

edited

Loading

ekatef commented Jul 6, 2023 •

edited

Loading

davide-f left a comment •

edited

Loading

ekatef Jul 8, 2023 •

edited

Loading