Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disentangle message_ix and ixmp_source #254

Open
khaeru opened this issue Nov 2, 2019 · 5 comments
Open

Disentangle message_ix and ixmp_source #254

khaeru opened this issue Nov 2, 2019 · 5 comments
Labels
discuss Discussion, design, and planning

Comments

@khaeru
Copy link
Member

khaeru commented Nov 2, 2019

One goal of iiasa/ixmp#182 / #249 was to clarify what core functionality of the message_ix modeling framework currently resides in the Java code of ixmp_source (i.e. behind ixmp's JDBCBackend).

As the name suggests, it is not the task of a storage backend to provide core model/framework logic; this logic should instead live in this repo. (In the current situation, new, non-Java backends would each be forced to re-implement that logic, which would be duplicative and increase points of failure.)

This issue is for:

  • @danielhuppmann @gidden @zikolach myself and possibly others to identify these pieces, and
  • a broader set of people to divide and coordinate (in further issues/PRs) the work of porting them from ixmp_source to message_ix.

Pieces

(@khaeru to flesh these out/keep them updated based on discussion below; others feel free to edit)

  1. Scenario initialization. message_ix.Scenario(..., version='new') returns a Scenario pre-populated with certain sets and parameters.
  2. Mapping sets. message_ix.Scenario.cat_list(...), .cat(...) and .add_cat(...) operate by creating and manipulating related sets named type_* and map_*.
  3. GDX output. The ixmp_source method Scenario.toGDX() writes Scenario data to a GDX file.
    • Depends on item 2.
  4. GDX input. The ixmp_source method Scenario.readSolutionFromGDX() reads the solution (levels of specific variables and marginals of specific equations) from a GDX file. It does not read any structure (i.e., sets) or input data (i.e., parameters).

Notes

  • @zikolach pointed out that some of the Java code is used by other builds from ixmp_source, e.g. behind the Scenario Explorer. So, re-implementing the functionality in Python does not always mean it can be removed from ixmp_source. On a case-by-case basis, we will need to decide this; and if the code is in both places, then there must be clear warnings to future developers that the behaviour must be kept in sync, perhaps by marking one or the other as authoritative.
  • Items 3 and 4 are needed to make ixmp's GAMSModel non-backend-specific. Currently it only works with JDBCBackend.
  • See also Update MESSAGEix scheme defaults in the ixmp core ixmp#70.
@danielhuppmann
Copy link
Member

I'm not sure why you think that item 3 depends on item 2?

@iiasa iiasa deleted a comment from danielhuppmann Nov 4, 2019
@iiasa iiasa deleted a comment from gidden Nov 4, 2019
@khaeru
Copy link
Member Author

khaeru commented Nov 4, 2019

I see the relationships as follows:

  • In order to write to GDX (item 3), message_ix (as opposed to ixmp_source) needs a full list of which sets and parameters to write (item 1), plus or including any ‘hidden’, ‘implicit’, or ‘auto-generated’ sets and parameters (item 2).
  • In order to read from GDX (item 4), message_ix would either:
    • use the same full list (item 1 + item 2) plus another list of solution-related items (equ, var, and possibly set/par), or
    • just the list of solution-related items, or
    • implicitly determine that any items found in the GDX file which are not in the (item 1 + item 2) list are solution-related.

As the latter point shows, this is somewhat implementation-dependent.

@danielhuppmann
Copy link
Member

I would not describe type_* and map_* as ‘hidden’, ‘implicit’, or ‘auto-generated’. These are two perfectly normal sets to define the mathematical structure of the equations. The cat_* feature described in term 2 just makes it slightly more user-friendly to work with them.

There are a number of additional mapping sets used as flags for GAMS, generally named is_*. These are only required in the workflow "write to gdx" and are completely unrelated to item 2.

@khaeru
Copy link
Member Author

khaeru commented Nov 7, 2019

The cat_* feature described in term 2 just makes it slightly more user-friendly to work with them.

This is good news, then—if the logic is simple and there's no special treatment of any of the categorizations, then it's easy to port.

There are a number of additional mapping sets used as flags for GAMS, generally named is_*.

Okay, thanks for pointing these out. Searching led me to MsgScenario.writeParametersToGDX() where these are defined; this should be the reference for whoever tackles these tasks.

@khaeru
Copy link
Member Author

khaeru commented Dec 13, 2019

@francescolovat's question on #275 revealed some additional issues to address/code to be consulted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discuss Discussion, design, and planning
Projects
None yet
Development

No branches or pull requests

2 participants