Bump pyam-iamc to >= 1.9.0

[phackstock]

closes #27.

• Tests added
• Documentation added
• Example added (in the documentation, to an existing notebook, or in a new notebook)
• Description in CHANGELOG.rst added (single line such as: (`#XX <https://github.com/iiasa/climate-assessment/pull/XX>`_) Added feature which does something)

[phackstock]

Looks like there's four tests that are breaking. If any of you @znicholls, @lewisjared or @jkikstra could have a quick look that would be great.

[znicholls]

I think I worked it out. Put the code below after line 360 of checks.py and you'll see. I think the original regression is in pandas, but you can work around it by tweaking pyam

        if model == "model7":
            import pdb
            pdb.set_trace()
            df_scen.timeseries()  # No CO2|Energy and Industrial as expected
            df_scen._data.index.get_level_values("variable").unique()  # No CO2|Energy and Industrial as expected

            # Using pyam accessors gives wrong result so `has_co2_energy`
            # evaluates incorrectly above
            df_scen.variable  # Contains CO2|Energy and Industrial ?!

            # What pyam uses internally. I think pyam is correct, but pandas
            # has a crazy regression where this returns incorrect information?!
            list(df_scen._data.index.levels[df_scen._data.index._get_level_number("variable")])  # Contains CO2|Energy and Industrial ?!
            # I assume that accessing levels directly on a MultiIndex is somehow not the intended pattern

            # I'd suggest updating pyam's internal's to use this, which seems to work
            list(df_scen._data.index.get_level_values("variable").unique())  # No CO2|Energy and Industrial as expected

[phackstock]

Thanks @znicholls, that's super weird. I've implemented your suggestion.
Is this a pyam bug? Could it be related to a variable being dropped and df.variable not being updated accordingly?

[znicholls]

I think once the change is applied globally this should pass through. Option b is you do a bugfix release in pyam (fixing the pandas bug), do a new pyam release then change this MR and you don't have to do any fix at all here.

[znicholls]

I think you'll need to make this replacement everywhere that df_scen.variable appears to get things to work properly.

[phackstock]

Ah right, of course

[phackstock]

Option b is you do a bugfix release in pyam (fixing the pandas bug), do a new pyam release then change this MR and you don't have to do any fix at all here.

So you think it's a pandas bug?
Not sure what's the best option here, if you can reproduce it, maybe open a pyam issue and we discuss over there? It's not super critical to update pyam for climate-assessment.
We could do #34 now then and just increase the pyam version later.

[danielhuppmann]

So I"m trying to get to the bottom of this, but I can't replicate the observation by @znicholls about df.variable returning something else than the correct values (with pyam 1.9 and pandas 1.5.3)... It seems all correct to me.

[danielhuppmann]

So I ran the tests along the entire history of pyam and it seems that IAMconsortium/pyam#731 breaks the test.

[znicholls]

pandas 1.5.3

I think I had pandas 2 maybe (memory fading now)

I ran the tests along the entire history of pyam

Nice detective work, did you use git bisect?

[znicholls]

(I've made a bit of a mess of this, sorry)

• this CI is passing https://github.com/iiasa/climate-assessment/actions/runs/5752989542/job/15595215041
• it's based on the branch in Try global replace #41
• that branch is the same as the branch in Try global replace #42
• so (I think), if you merge Try global replace #42, this will pass (once black is applied). The fix is super yuck to look at, but that might be the 10 minute solution before working out the pandas headache if you need something quick

[danielhuppmann]

Implement a fix based on @znicholls' observation in IAMconsortium/pyam#762

[phackstock]

After the pyam update, I wanted to update this PR to verify that that fixes the issue. However, as I just saw the tests were already passing yesterday? I'm a bit confused now but I guess it's all good?

[danielhuppmann]

See ccf0e7b - the problem was caused by the very inefficient groupby, so I refactored that piece of the code.

[phackstock]

Since the tests are passing and I'd love to get this out before the weekend, I'll move on with the merge. Hope that's ok @znicholls.

[znicholls]

Hope that's ok @znicholls.

If all passing, all good (particularly given how much we've talked about this already)