Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove 'xls' as by-default-supported file format #708

Merged
merged 16 commits into from
Oct 4, 2022

Conversation

danielhuppmann
Copy link
Member

@danielhuppmann danielhuppmann commented Sep 30, 2022

Please confirm that this PR has done the following:

  • Tests Added
  • Documentation Added
  • Name of contributors Added to AUTHORS.rst
  • Description in RELEASE_NOTES.md Added

Description of PR

This PR

  • removes xlrd as a standard requirement
  • removes hard-coded engines for reading/writing to Excel, letting pandas handle the file-type
  • adds a test that "meta"-indicators are also parsed imported correctly from an xls file

When a user tries to import an xls-file, pandas will show this error message

ImportError: Missing optional dependency 'xlrd'. Install xlrd >= 1.0.0 for Excel support Use pip or conda to install xlrd.

Install xlrd manually and it will work fine - and there is a test to prove it.

Writing to xls is not support any more by pandas.

closes #706

@danielhuppmann danielhuppmann self-assigned this Sep 30, 2022
@codecov
Copy link

codecov bot commented Sep 30, 2022

Codecov Report

Merging #708 (f2068e9) into main (947d8c4) will decrease coverage by 0.0%.
The diff coverage is 87.5%.

@@           Coverage Diff           @@
##            main    #708     +/-   ##
=======================================
- Coverage   94.8%   94.8%   -0.1%     
=======================================
  Files         58      58             
  Lines       5899    5907      +8     
=======================================
+ Hits        5598    5604      +6     
- Misses       301     303      +2     
Impacted Files Coverage Δ
pyam/iiasa.py 86.7% <ø> (ø)
tests/test_iiasa.py 97.8% <ø> (-0.1%) ⬇️
tests/test_io.py 98.4% <85.7%> (-1.6%) ⬇️
pyam/core.py 94.5% <87.5%> (ø)
pyam/utils.py 91.6% <100.0%> (-0.1%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@danielhuppmann danielhuppmann marked this pull request as ready for review September 30, 2022 18:49
@danielhuppmann
Copy link
Member Author

@Rlamboll, in line with our earlier discussions, I tried to have the cake and eat it, too - reading xls is now properly supported (again), but only with a dedicated step for a user of installing the xlrd package - and with a test to prove that it works...

Plus we let pandas handle file-types and engines natively, which is less overhead in our code and probably more forward-compatible.

Copy link
Collaborator

@Rlamboll Rlamboll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, just have a few concerns that the documentation (in some cases historic) may be confusing.

pyam/core.py Outdated
iamc_index : bool, default False
if True, use `['model', 'scenario', 'region', 'variable', 'unit']`;
else, use all 'data' columns
path : path-like or file-like
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this description is unclear - it's presumably still accepting strings, as well as file objects or paths?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for noticing... I looked at the pandas to_csv in more detail to be sure, and noticed that you can even use path=None- now supported by pyam, and with a test.. (and I also improved the docs).

pyam/core.py Outdated
File path or :class:`pathlib.Path`.
iamc_index : bool, optional
If True, use `['model', 'scenario', 'region', 'variable', 'unit']`;
else, use all :meth:`dimensions`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to describe how you're using these variables

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a reference to the timeseries() method which uses that argument. The documentation for the extra-columns feature needs an overhaul, but not as part of this PR.

pyam/core.py Outdated
any valid string path, :class:`pathlib.Path`
or :class:`pandas.ExcelWriter`
excel_writer : path-like, file-like, or ExcelWriter object
File path, :class:`pathlib.Path`, or existing :class:`pandas.ExcelWriter`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar to above

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised.

if this is a string, use it as sheet name
Name of sheet which will contain :meth:`timeseries` data.
iamc_index : bool, optional
If True, use `['model', 'scenario', 'region', 'variable', 'unit']`;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as above

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See answer above.

Copy link
Collaborator

@Rlamboll Rlamboll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Copy link
Contributor

@phackstock phackstock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one small comment from my side.
Also without that would be good to merge from my side.
Very much appreciate the usage of the pandas defaults 👍.

tests/test_io.py Show resolved Hide resolved
@danielhuppmann danielhuppmann merged commit 60d84d8 into IAMconsortium:main Oct 4, 2022
@danielhuppmann danielhuppmann deleted the remove-xls branch October 4, 2022 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove support of xls files
3 participants