Pass through Engine kwargs in ExcelWriter

[feefladder]

• closes BUG: ExcelWriter engine_kwargs not passed through to other engines than xlsxwriter #42292 #42214 #43440 #43442
• closes DOC: no docs for ExcelWriter engine_kwargs #42292 #42214 #43440
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

This is the revised version of #42214. In fact, in xlsxwriter, all kwargs are passed like so:

ExcelWriter("path_to_file.xlsx",engine="xlsxwriter",engine_kwargs={"options":{"nan_inf_to_errors":True}})

which then becomes internally:

from xlsxwriter import Workbook
Workbook("path_to_file.xlsx",options={"nan_inf_to_errors":True})

so my initial fix was wrong in that sense, because I made engine_kwargs able to be passed in like this:

ExcelWriter("path_to_file.xlsx",engine="xlsxwriter",engine_kwargs={"nan_inf_to_errors":True})

and the resulted internals:

Workbook("pat_to_file.xlsx",{"nan_inf_to_errors":True})

so #42214 was actually wrong (we want users to actually pass in kwargs.

However, there was still the issue of #43442, where the following engines do have some options that were silently ignored:

• openpyxl can set iso_dates and write_only
• xlwt can set style_compression (not sure why anybody would want this, but people may)
• xlsxwriter has many options

so I added tests for that in engine_kwargs.

[rhshadrach]

Minor request below. Also some CI failures related to the docstrings.

[feefladder]

I had 6 times in python scripts/validate_docstrings.py pandas.ExcelWriter that ExcelWriter is not defined So I changed those to pd.ExcelWriter.
Then that

>>> with pd.ExcelWriter(
    ...     "path_to_file.xlsx",
    ...     engine="openpyxl",
    ...     mode="a",
    ...     engine_kwargs={"keep_vba": True}
    ... ) as writer:
    ...     df.to_excel(writer)

had that Sheet1 already exists, and if_sheet_exists is set to "error". I added sheet_name="Sheet2
so that is changed, but I think this is a bit weird, and does not add to simplicity of the example.
and twice that I forgot a space after colon
Now only 'error' is that there is no "See Also" section, but I think that should pass?

[rhshadrach]

Thanks @joeperdefloep and apologies for the delay. Docs look good (addition of pd. is on master now as well). Looks like there is one failure that is related:

FAILED pandas/tests/io/excel/test_openpyxl.py::test_engine_kwargs_append[.xlsx]

[jreback]

@joeperdefloep can you merge master and fixup according to @rhshadrach comments

[erfannariman]

@joeperdefloep @alimcmaster1 I am willing to finish this one up, if you guys don't have the time, since we need this in some of our projects.

[feefladder]

Thanks @joeperdefloep and apologies for the delay. Docs look good (addition of pd. is on master now as well). Looks like there is one failure that is related:

FAILED pandas/tests/io/excel/test_openpyxl.py::test_engine_kwargs_append[.xlsx]

Hello everyone. Apologies for the delays

That's funny... It passed on my machine.... This is the error:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\wrE4sPxHRxGbeUSKoQgfnJd1mC0iTd.xlsx'

I just changed the tests. Should be good now. ARM seems to be failing, but that is quite the black box to me...

[feefladder]

I now get a lot of the following errors:

AttributeError: module 'aiobotocore' has no attribute 'AioSession'

seems to be related to this issue:
fsspec/s3fs#514

[rhshadrach]

Looking good, some minor requests on the tests.

[rhshadrach]

I think read_only would be sufficient. Need to verify the kwarg gets through.

[feefladder]

I thought testing for all available parameters would be nice, since then we will also be warned when API changes on openpyxl's side...

[feefladder]

Actually, read-only will not work here, and give different errors on windows than linux, this is why the previous test failed

[rhshadrach]

If I'm understanding the test correctly, the passing of kwargs through to openpyxl could be entirely removed from pandas and this test would pass. Let me know if you're not sure of a way to improve that (or if my assessment is wrong).

[feefladder]

hmm, that's true. That's why I have the other test: test_engine_kwargs_append_invalid that tests for an invalid keyword argument in load_workbook. Also the other test test_engine_kwargs_append_data_only tests for correct use of openpyxl's data_only being passed through. That would make this test redundant indeed? It just felt logical to first test if passing an engine kwarg does not raise an error normally, then test if it does raise with an invalid argument and then test if the passing of engine kwargs works well together with openpyxl. Then in that order, people will know more precisely where the error lies.

[feefladder]

Tell me if you want it removed!

[rhshadrach]

Thanks for the summary, very helpful. Yes, I think this test can be removed as it's redundant with test_engine_kwargs_append_data_only. In general, we'd rather not have to change tests if the API of a third party engine changes and it doesn't negatively impact pandas. It is okay to do so when necessary, as is in the data_only test.

[feefladder]

@jreback friendly ping :)

[rhshadrach]

lgtm, can you resolve conflict.

[jreback]

thanks @joeperdefloep very nice!