ValueError in .add_year.interpolate_2d() with pandas 1.4.0

[LauWien]

The tests test_add_year.py::test_add_year_cli and test_add_year.py::test_add_year start to fail seemingly randomly in various PRs on different OS e.g. here. Looking deeper, failing test are using pandas==1.4.0, passing tests are using e.g. pandas==1.1.5. With upgrading pandas to 1.4.0 locally these errors can be replicated, while with version 1.1.5 the test succeed.

[khaeru]

The fundamental cause here is code that's not compatible with current pandas; the failing CI is only a symptom. The offending lines are:

message_ix/message_ix/tools/add_year/__init__.py

Lines 849 to 851 in 1df3619

	df2_t[yr].loc[cond1 & cond2] = intpol(
	df2_t[year_pre], df2_t[year_pp], year_pre, year_pp, yr
	)

ValueError: cannot set using a list-like indexer with a different length than the value

NB #494 was started to simplify these very long methods that mostly duplicate upstream features and are also quite slow, but I lacked clear tests for the expected behaviour for some combinations of arguments.

[LauWien]

The fundamental cause here is code that's not compatible with current pandas;

Yes, was already checking where and how to adapt the code. Thank you for linking the PR!

[khaeru]

#557 (comment)

It looks like there were several regressions related to pd.DataFrame.loc fixed in pandas 1.4.1: https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.4.1.html#fixed-regressions.

i.e. perhaps this is only with pandas 1.4.0 precisely.