Develop better input data system for MESSAGEix-Materials

[macflo8]

At the moment, most of the input data for MESSAGEix-Materials is stored in xlsx files. This has the following advantages and disadvantages as a consequence.:

Pros

• xlsx files can hold various data types including data visualizations and images, which can be used to collect related literature all in one file
• Since many people are familiar with spreadsheet software, the xlsx files can increase the accessibility for new users

Cons

• Since xlsx files are binary, changes in these files picked up by version control are not human readable
• Merging these files with git is not possible

To resolve the version control issues, it would be desirable to move to csv and yaml files to store input data. There is however a need to find a solution for the remaining data documentation currently stored in the xlsx files.

[khaeru]

Another con is that reading data from .xlsx files is slower than reading the same number of data points from .csv and some other formats. This impact is more noticeable if the files are large or numerous.

We have some approaches in MESSAGEix-Transport for handling collections of exogenous data files, while keeping track of their provenance through inline comments and documentation. Some of that could be emulated, and some utilities could be migrated to message_ix_models.util (or similar) for use by other modules.