PyInstaller, Nuitka compatibility?

[bede]

I've noticed that PyInstaller and Nuitka seem to dislike defopt for some reason but am not sure why. I have a package with a bunch of dependencies that works fine without defopt, but breaks when using it, using both of these popular distribution mechanisms. I have tried forcing import of sphinxcontrib.napoleon and pockets using PyInstaller's built-in mechanism for doing so, although naively I can't see anything odd about the import that would prevent pockets from being picked up automatically, as is supposed to happen. What might be the issue here?

Of course this does not indicate a defopt issue, but might be something many run into, and I wondered if you might have any thoughts @anntzer ?

PyInstaller

% dist/gpas-cli/gpas-cli status                                                        
Traceback (most recent call last):
  File "gpas/cli.py", line 6, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "defopt.py", line 48, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "sphinxcontrib/napoleon/__init__.py", line 13, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "sphinxcontrib/napoleon/docstring.py", line 22, in <module>
ImportError: cannot import name 'modify_iter' from 'pockets' (/Users/bede/Research/Git/gpas-cli/dist/gpas-cli/pockets/__init__.pyc)
[76308] Failed to execute script 'cli' due to unhandled exception!

Nuitka

% ./cli.bin 
Traceback (most recent call last):
  File "/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_81481_1655893898_954284/cli.py", line 6, in <module>
  File "/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_81481_1655893898_954284/defopt.py", line 48, in <module defopt>
  File "/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_81481_1655893898_954284/sphinxcontrib/napoleon/__init__.py", line 13, in <module sphinxcontrib.napoleon>
  File "/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_81481_1655893898_954284/sphinxcontrib/napoleon/docstring.py", line 22, in <module sphinxcontrib.napoleon.docstring>
ImportError: cannot import name 'modify_iter' from 'pockets' (/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_81481_1655893898_954284/pockets/__init__.py)

[anntzer]

1. Does PyInstaller/nuitka work for a simple application that simply declares a dependency on sphinxcontrib-napoleon or even just pockets and tries to import it? If it doesn't, then the bug isn't really here, but rather on one of PyInstaller, nuitka, sphinxcontrib-napoleon or pockets, and should be reported there.
2. I have, in fact, been trying to get rid of the pockets (and sphinxcontrib-napoleon) dependency, but this is currently blocked by Replace napoleon.iterators by simpler stack implementation sphinx-doc/sphinx#9856 (as I certainly don't want to maintain a full independent fork of s-n myself).

[bede]

Thanks @anntzer : )
I just made a MWE package called defoptmwe and yes, it does break Nuitka! Nuitka is amazingly clever and so I'm surprised that napolean / pockets breaks it (and Pyinstaller) so effectively.

I love defopt so much more than the alternatives. Would you suggest vendoring in my case?

(defopt) bede@yoshina defoptmwe % python -m nuitka --onefile --include-package=defoptmwe src/defoptmwe/cli.py
Nuitka-Options:INFO: Used command line options: --onefile --include-package=defoptmwe src/defoptmwe/cli.py
Nuitka:INFO: Starting Python compilation with Nuitka '0.8.4' on Python '3.10' commercial None.
Nuitka-Plugins:INFO: anti-bloat: Handling module 'pydoc' with 1 change(s) for: remove module ability to display GUI with tkinter and topics data.                                                              
Nuitka-Plugins:INFO: implicit-imports: Implicit dependencies of module 'zipfile' added 'py_compile'.                                                                                                           
Nuitka-Plugins:INFO: implicit-imports: Implicit dependencies of module 'urllib.request' added 'http.client,http.cookiejar'.                                                                                    
Nuitka-Plugins:INFO: implicit-imports: Implicit dependencies of module 'socket' added '_socket'.                                                                                                               
Nuitka-Plugins:INFO: implicit-imports: Implicit dependencies of module 'shelve' added 'dbm.dumb'.                                                                                                              
Nuitka-Plugins:INFO: implicit-imports: Implicit dependencies of module 'pathlib' added 'ntpath,posixpath'.                                                                                                     
Nuitka-Plugins:INFO: implicit-imports: Implicit dependencies of module 'importlib.metadata' added 'email'.                                                                                                     
Nuitka-Plugins:INFO: implicit-imports: Implicit dependencies of module 'ctypes' added '_ctypes'.                                                                                                               
Nuitka-Plugins:INFO: implicit-imports: Implicit dependencies of module '_asyncio' added 'asyncio'.                                                                                                             
Nuitka-Plugins:INFO: implicit-imports: Implicit dependencies of module 'encodings' added 'encodings.idna,encodings.base64_codec,encodings.hex_codec,encodings.rot_13'.                                         
Nuitka-Plugins:INFO: anti-bloat: Handling module 'pydoc' with 1 change(s) for: remove module ability to display GUI with tkinter and topics data.                                                              
Nuitka:INFO: Completed Python level compilation and optimization.                                                                                                                                              
Nuitka:INFO: Generating source code for C backend compiler.
Nuitka:INFO: Running data composer tool for optimal constant value handling.                                                                                                                                   
Nuitka:INFO: Running C compilation via Scons.
Nuitka-Scons:INFO: Backend C compiler: clang (clang).
Nuitka-Scons:INFO: Backend linking program (no progress information available).                                                                                                                                
Nuitka-Scons:INFO: Compiled 23 C files using ccache.
Nuitka-Scons:INFO: Cached C files (using ccache) with result 'cache miss': 22
Nuitka-Postprocessing:INFO: Creating single file from dist folder, this may take a while.                                                                                                                      
Nuitka-Onefile:INFO: Running bootstrap binary compilation via Scons.
Nuitka-Scons:INFO: Onefile C compiler: clang (clang).
Nuitka-Scons:INFO: Onefile linking program (no progress information available).                                                                                                                                
Nuitka-Scons:INFO: Compiled 1 C files using ccache.
Nuitka-Scons:INFO: Cached C files (using ccache) with result 'cache miss': 1
Nuitka-Onefile:INFO: Keeping onefile build directory 'cli.onefile-build'.
Nuitka-Onefile:INFO: Using compression for onefile payload.
Nuitka-Onefile:INFO: Onefile payload compression ratio (31.45%) size 22039423 to 6932443.                                                                                                                      
Nuitka:INFO: Keeping dist folder 'cli.dist' for inspection, no need to use it.                                                                                                                                 
Nuitka:INFO: Keeping build directory 'cli.build'.
Nuitka:INFO: Created binary that runs on macOS 10.9 (x86_64) or higher.
Nuitka:INFO: Successfully created 'cli.bin'.
(defopt) bede@yoshina defoptmwe % ./cli.bin hello
Traceback (most recent call last):
  File "/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_87254_1655902215_231121/cli.py", line 1, in <module>
  File "/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_87254_1655902215_231121/defopt.py", line 48, in <module defopt>
  File "/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_87254_1655902215_231121/sphinxcontrib/napoleon/__init__.py", line 13, in <module sphinxcontrib.napoleon>
  File "/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_87254_1655902215_231121/sphinxcontrib/napoleon/docstring.py", line 22, in <module sphinxcontrib.napoleon.docstring>
ImportError: cannot import name 'modify_iter' from 'pockets' (/var/folders/l0/r1d9n9sn2lv83_lpr4swjcv80000gn/T//onefile_87254_1655902215_231121/pockets/__init__.py)
(defopt) bede@yoshina defoptmwe %  defoptmwe hello
Hello, world

[anntzer]

I would suggest reporting this as a bug to nuitka, I guess?
Closing as not actionable (AFAICT) on my side.

[bede]

Fair enough – I asked in Pyinstaller and Nuitka circles first, but haven't heard anything yet. I just find this strange since defopt has been the only problem package out of like 50 deps in one of my projects which work fine with both tools.

I do think this will affect other people and might be worthy of consideration

I've tried dropping the packages straight into src in my MWE and I get the same issue. Super odd.

[bede]

Ok I figured it out, I think. Adding pockets as a local package did the trick when built with --include-package=pockets

These tools also require if __name__ == "__main__": main()

🙌