Setup osemosys_global as an installable Python package and deploy to PyPI

[willu47]

We'd like to use elements of osemosys_global in the Climate Compatible Growth project to extract data from the Plexos World dataset.

To do this we'd like to install osemosys_global as a Python package so that we don't have to copy and paste pieces of code from this repository, and also so we'd be able to take advantage of bug fixes as you implement them.

[willu47]

@abhishek0208 @tniet - what do you think about this idea? It's perhaps a bit different to thinking of osemosys-global as a data pipeline, but maybe it's a good way to get everything into order and think through the "user interface" - perhaps a simple command line interface to start with, and later a web-interface?

[tniet]

Hey @willu47 - that's an interesting idea. I recently went through the packaging challenges with clewsy and it was both worse, and not as bad, as I expected. The main question I would have is the value of this outside of the CCG project. If you think there would be value to a broad swath of users to package things it might be worth the effort.

[willu47]

I think there would be value, also in preparation for a snakemake version, which anyway will need us to tidy up each of the scripts a little bit (they work fine, just to make the inputs and outputs better defined).

[tniet]

Yes, much tidying up is needed, and if that moves us towards a package that can be imported/used by others that's great. Will discuss approaches, etc. at our meeting tomorrow with @abhishek0208 and Maarten.

[maartenbrinkerink]

@willu47 @tniet @trevorb1 Trying to do a little housekeeping and just wanted to know if this issue should remain open. As also commented in #35, we are slowly moving away from using the original PLEXOS-World datasets as it is quite outdated. Feel free to close this issue or if it still has future relevance some suggested next steps would be useful.

[trevorb1]

@maartenbrinkerink I think OG is actually setup as an installable package! (ie. you can pip install . and do from osemsoys_global import ... within scripts). It is not deployed to pypi though. (ie. you can not do pip install osemosys_global).

I dont necessarily know the best way to have the package interface with data that needs to be locally downloaded. So if we were to deploy to pypi right now, no functions would work because the retrieval of data is built into the scripts that are not well contained to do single functions. If I were completing this ticket, I would need to consult with someone, or just do some research, on the best way to handle the separation of data and processing logic before implementation (but others in this thread may already have an idea on how to handle this!)

I think having this as a long term goal would be nice. Also, it may force us to be more diligent about doing specified version releases.

[willu47]

@trevorb1 - see #35 and #39, which touch on the "where to put the data issues".