type_year="cumulative" is only populated at .solve()

[SongminYu]

In the emission bound (cumulative) tutorial, it seems one step is missing: add category for year, so "type_year" has "cumulative".

Besides, the following line is not clear enough:

scen.add_par(
    "bound_emission", [country, "GHG", "all", "cumulative"], value=500.0, unit="MtCO2"
)

The users do not know the keys in the key_or_data. I found this error by comparing with the emission tax tutorial... good training :)

[khaeru]

@SongminYu, thanks for these two suggestions.

First, when opening an issue on this repository you should have seen an a template (this one) that asks you to give certain information—including "the output of message-ix show-versions". We need this information—including which version(s) of the tutorials you are using—in order to understand the nature of the issue.

[In] The emission bound (cumulative) tutorial, it seems one step is missing: add category for year, so "type_year" has "cumulative".

The latest versions of the tutorials are automatically tested daily; see here. These tests all pass, which means that the code in each notebook cell runs without any error. When you say "one step is missing", what specifically happens without that step? Do you see an error message? If so, please paste the full text.

the following line is not clear enough … the users do not know the keys in the key_or_data.

This is a good suggestion. Although (a) this is a documented and supported usage of .add_par() and (b) the dimensions of bound_emission are documented, here —in general we try to use make_df() in our tutorials. The reason is to be clear and explicit about the data that is being entered, as you suggest, without the reader needing to consult the documentation.

@behnam-zakeri @glatterf42 perhaps we can include a small change to this line into #815.

[SongminYu]

Hi @khaeru, thanks for your fast reply. I will remember to use the template next time.

Regarding the tutorial error I saw, first I need to say that I am not running the Jupyter file but using PyCharm that I am more used to. I just pushed my code to GitHub and you can try it by running the main.py in the westeros_tutorial folder.

If you comment out line 91 in the main.py file and run it, you can see the error message:

ValueError: The index set 'type_year' does not have an element 'cumulative'!

The error was fixed after I added this line 91, setting the category (or type_year) "cumulative" to "700, 710, 720".

# Line 91: 

scenario.add_category(set_name="year", file_name="Category_Year")

The solved OBJ is 206280.146577.

The excel file Category_Year and function add_category are as shown below:

    def add_category(self, set_name: str, file_name: str):
        df = self.load_xlsx(file_name)
        for _, row in df.iterrows():
            self.scenario.add_cat(set_name, row["category"], row["name"])

Output of message-ix show-versions

<!--
ixmp:        3.8.0
message_ix:  0.0.0
message_ix_models: None
message_data: None

click:       8.1.7
dask:        2023.11.0
genno:       installed
graphviz:    None
jpype:       1.4.1
… JVM path:  /Users/songminyu/miniconda3/envs/MESSAGE/lib/jli/libjli.dylib
openpyxl:    3.1.2
pandas:      2.1.3
pint:        0.22
xarray:      2023.10.1
yaml:        6.0.1

iam_units:   installed
jupyter:     installed
matplotlib:  3.8.1
plotnine:    0.12.4
pyam:        0.7.0

GAMS:        40.4.0

python:      3.10.13 (main, Sep 11 2023, 08:39:02) [Clang 14.0.6 ]
python-bits: 64
OS:          Darwin
OS-release:  22.1.0
machine:     x86_64
processor:   i386
byteorder:   little
LC_ALL:      None
LANG:        None
LOCALE:      (None, 'UTF-8')
-->

[SongminYu]

After replying above, I continued implementing the other two emission bound examples in the tutorial:

• emission_bound_year
• emission_bound_cumulative_tax

Now they are also in the repo. I noticed that,

First, in the emission_bound_year example, I met another similar error:

ValueError: The index set 'type_year' does not have an element '700'!

This error can be solved by using either of the following two lines:

scenario.add_category(set_name="year", file_name="Category_Year_EmissionBoundYear")   # Line 120 in main.py
scenario.add_set(set_name="type_year", file_name="Set_TypeYear")   # Line 121 in main.py

The two excel files are as below:

Actually, the second line is used in the emission_bound_cumulative_tax example in the tutorial. Adding type_year info seems necessary in this example, but not mentioned in the other two emission bound examples.

Second, I tried running the baseline example again, and surprisingly found:

In the "cat_year" sheet generated (and exported) by the reporter, the type_year info exists... But I didn't add such info by using any code. So, my guess is, that such information was somehow added to the scenario when solving it. In the tutorial of emission_bound and emission_bound_year, the scenario was cloned from baseline, so the info was there. However, in my PyCharm workflow, the scenario was newly created so I need to add it by myself, or I will see the error message. Is this right?

[glatterf42]

Hi @SongminYu and thanks for your extensive report! There are a few things that jump out to me here:

1. Your output of message-ix show-versions shows that you're using v3.8.0 of ixmp, but v0.0.0 of message_ix. This is not how it should be; with current versions, both packages should have an identical version. So in your case, we would expect message_ix to also have v3.8.0.
   It looks like you installed via miniconda. Do you still have the output of the conda install command or can you reproduce it? Maybe from that we'll get a hint which version of the tutorial files you tried to use.
2. The output also shows you're using v0.7.0 of pyam, which was released in 2020, so is severely outdated by now (they just released v2.2.2). This could be related to the reason why you're message_ix version is not 3.8.0.
3. In the code you shared with us, you create a Scenario with Scenario(config=config) and then add a category for the set named year. However, the current message_ix.Scenario doesn't take a config keyword, but it would be passed on to ixmp.Scenario. This doesn't use a config keyword either, but passes it on to the GAMSModel class (most likely), which doesn't use it, either. So I'm not sure what's going on here. Based on the above, it seems like the values you set in config would not be used at all, but Scenarios typically need at least a model name, which would be given with the model key, though, not model_name. Also, Scenarios are stored in so-called Platforms, which provide access to underlying databases via ixmp. For example in the westeros_baseline tutorial, you would call scenario = message_ix.Scenario(mp, model="Westeros Electrified", scenario="baseline", version="new") to create a Scenario (where mp is the name of a Platform object). Your main.py is missing Platforms completely, so that seems severely outdated, too.
   Finally, also from the westeros_baseline tutorial, here's how we currently add years to a Scenario:

history = [690]
model_horizon = [700, 710, 720]
scenario.add_horizon(year=history + model_horizon, firstmodelyear=model_horizon[0])

The add_horizon() function is a convenience function that automatically adds required categorys, too.

So I think the first thing we need to help you is to figure out which version of MESSAGE you're using. Some of the aspects that are not in your main.py (like ixmp.Platform) seem so fundamental to MESSAGEix that I'm beginning to wonder: are you perhaps still using MESSAGE-V?

[khaeru]

I'm beginning to wonder: are you perhaps still using MESSAGE-V?

Looking at the repo shared in the earlier comment, we see e.g. https://github.com/SongminYu/message_westeros/blob/main/message/scenario.py —so it seems the user has written their own code including classes with similar names to Scenario that somehow wrap the actual message_ix.Scenario class.

@SongminYu, to be clear, we can provide support for the features of message_ix per se, but your own code is your own code, and we obviously can't be responsible for errors you've made in writing it.

[glatterf42]

the user has written their own code including classes with similar names to Scenario that somehow wrap the actual message_ix.Scenario class.

I see, that makes more sense :)

[khaeru]

Referring to our own westeros_baseline.ipynb, I modify the following cell:

scenario.solve()

To the following

from icecream import ic

# Prior to solve
ic(scenario.set("type_year"))
ic(scenario.set("cat_year"))

scenario.solve()

# After solve
ic(scenario.set("type_year"))
ic(scenario.set("cat_year"))

And see this output:

ic| scenario.set("type_year"): 0          firstmodelyear
                               1           lastmodelyear
                               2    initializeyear_macro
                               dtype: object
ic| scenario.set("cat_year"):         type_year  year
                              0  firstmodelyear   700

…

ic| scenario.set("type_year"): 0          firstmodelyear
                               1           lastmodelyear
                               2    initializeyear_macro
                               3              cumulative
                               4                     700
                               5                     710
                               6                     720
                               dtype: object
ic| scenario.set("cat_year"):         type_year  year
                              0  firstmodelyear   700
                              1      cumulative   700
                              2             700   700
                              3      cumulative   710
                              4             710   710
                              5      cumulative   720
                              6             720   720

This tells me a few things:

• The elements in the type_year set named "cumulative" and "700" etc. (the latter corresponding to elements "700" etc. in the set year) seem to be magically created when solve() is called.
• Looking more closely, this happens in the ixmp_source/JDBCBackend code. This violates stack separation: this behaviour (the existence of a type_year member named "cumulative") should be implemented within message_ix itself, and not ixmp or other backends (for instance, ixmp4). So this is indeed a bug that needs fixing—not in the tutorial, but within message_ix itself.
• In @SongminYu's custom code, I suspect (haven't looked) that Scenario.solve() is not called before they try to .add_par() with a reference to "cumulative". This is different from the message_ix tutorials and test suite, and that's why they encounter an error that we do not.

I will retitle this issue to cover the point in message_ix to be resolved.

@SongminYu, for your own code, I think the work-arounds already identified are fine:

• Call .add_set() manually.
• .solve() the scenario before using type_year="cumulative".

[khaeru]

The fix should be to extend message_ix.models.MESSAGE.initialize():

message_ix/message_ix/models.py

Lines 304 to 327 in aabb789

	@classmethod
	def initialize(cls, scenario):
	"""Set up `scenario` with required sets and parameters for MESSAGE.
	See Also
	--------
	:attr:`items`
	"""
	# Check for storage items that may contain incompatible data or need to be
	# re-initialized
	state = None
	for name, ix_type, N, message in _check_structure(scenario):
	if len(message):
	warn(message) # Existing, incompatible data → conspicuous warning
	elif N == 0:
	# Existing, empty item → remove, even if it has the correct dimensions.
	state = maybe_check_out(scenario, state)
	getattr(scenario, f"remove_{ix_type}")(name)
	# Initialize the ixmp items for MESSAGE
	cls.initialize_items(scenario, {k: v.to_dict() for k, v in cls.items.items()})
	# Commit if anything was removed
	maybe_commit(scenario, state, f"{cls.__name__}.initialize")

[SongminYu]

Hi @khaeru, hi @glatterf42, thanks for your reply! Yes, that's exactly how I suspect: the "cumulative" category info was added in the .solve() step. So, when cloning baseline from the database, this info was carried back into the emission_bound model. That's why it works in the tutorial but not in my code. Thanks for the clarification!

[SongminYu]

Hi @glatterf42, yes, I installed the packages via miniconda. It was a long time ago when I first want to play with MESSAGE but was interrupted by other work. I just checked the version problem and managed to update the pyam-iamc package to V2.0.0.

Output of message-ix show-versions

<!--
ixmp:        3.8.0
message_ix:  0.0.0
message_ix_models: None
message_data: None

click:       8.1.7
dask:        2024.4.2
genno:       installed
graphviz:    None
jpype:       1.5.0
… JVM path:  /Users/songminyu/miniconda3/envs/MESSAGE/lib/jvm/lib/libjli.dylib
openpyxl:    3.1.2
pandas:      2.2.2
pint:        0.23
xarray:      2024.3.0
yaml:        6.0.1

iam_units:   installed
jupyter:     installed
matplotlib:  3.6.2
plotnine:    0.13.4
pyam:        2.0.0

GAMS:        40.4.0

python:      3.10.13 (main, Sep 11 2023, 08:39:02) [Clang 14.0.6 ]
python-bits: 64
OS:          Darwin
OS-release:  22.1.0
machine:     x86_64
processor:   i386
byteorder:   little
LC_ALL:      None
LANG:        None
LOCALE:      (None, 'UTF-8')
-->

But, the message_ix version is still V0.0.0 printed here. However, it should be V3.8.0 as shown in the screenshot below. I am not sure why it is printed differently in the terminal. Anyway, it seems working fine... Thanks!