Issue when using importlib_metadata inside a standalone application

[jaraco]

In GitLab by @astrofrog on Jun 14, 2019, 11:43

I'm using PyInstaller to build standalone Python applications, but I've run into issues with importlib_metadata. Specfically, if I make a Python script called e.g. test_app.py with:

import importlib_metadata

then run PyInstaller with:

pyinstaller --onefile test_app.py 

The resulting application crashes when trying to get the version for PyInstaller:

$ dist/test_app 
Traceback (most recent call last):
  File "test_app.py", line 1, in <module>
    import importlib_metadata
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/Users/tom/python/numpy-only/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "importlib_metadata/__init__.py", line 471, in <module>
  File "importlib_metadata/__init__.py", line 438, in version
  File "importlib_metadata/__init__.py", line 411, in distribution
  File "importlib_metadata/__init__.py", line 184, in from_name
importlib_metadata.PackageNotFoundError: importlib_metadata
[12943] Failed to execute script test_app

Other users of PyInstaller seem to have run into this too (see e.g. this issue).

I've only ever run into this issue with importlib_metadata, and haven't had a chance to investigate what this could be due to.

[jaraco]

In GitLab by @mcdanieldave on Jun 25, 2019, 17:55

Hi, I ran into this issue with cx_freeze. The code that sets importlib_metadata.__version__ is calling some logic to discover the metadata of the package. It seems that this type of egg-info is not coming over when distributed as a frozen package. In cx_freeze's case it is looking in the library.zip or the lib directory contained in the frozen distribution. I did not test the use case of freezing to a standalone file, but I came up with some hacky workarounds for my use case. Maybe you will find this helpful in your use case too. By hacky, please recognize that these only work for me since I am not relying on the found metadata being used for anything in my frozen distribution, besides that one version call. Option 1 will certainly not work if your application relies on that found data and will likely break the functionality of the library.

Option 1:

Put a file called "EGG-INFO" in one of the directories that importlib-metadata is searching for meta data in. In the case of cx_freeze this is in the lib or the library.zip directories.

Option 2:

You could also drop the proper importlib_metadata.egg-info file into the directory such that the discovery logic finds it. This really shouldn't break anything, but keep in mind you may have to do this for other packages as well if you are actually relying on it somehow in your application.

Option 3:

modified   importlib_metadata/__init__.py
@@ -469,3 +469,6 @@ def requires(package):
 
 
-__version__ = version(__name__)
+try:
+    __version__ = version(__name__)
+except PackageNotFoundError:  # for frozen app support, enter correct version here
+    __version__ = '0.18'

For now I opted for the last option since it only impacts the version of importlib-metadata which I happen to have pinned in my requirements file.

[jaraco]

In GitLab by @jaraco on Jul 13, 2019, 05:10

This issue was encountered in jaraco/irc#159.

[jaraco]

In GitLab by @jaraco on Jul 13, 2019, 05:23

The underlying issue is that pyinstaller and cx_Freeze are discarding important package metadata when constructing their artifacts (executables, etc). Similar to what you propose in option 2, here's what I recommend: Either,

• packagers should include metadata in a format that's already recognized by importlib_metadata's MetadataPathFinder, or
• packagers should register and install another DistributionFinder per the docs.

Considering option 3, it's possible that importlib_metadata could be more lenient about discovering its own metadata, but I fear, as you mention, that will only postpone the inevitable and lead to failures later when metadata is needed by the consumer of importlib_metadata. Therefore, I recommend that these meta packagers address the concerns. Please feel free to reference this issue/comment when filing issues with those projects. I'll be happy to advise those projects to support what's needed.

[jaraco]

In GitLab by @jaraco on Sep 1, 2019, 16:05

packagers should include metadata in a format that's already recognized

I should have also recommended against using egg-info metadata, as dist-info metadata is more modern and likely to be supported longer. Additionally, you can use tools like pep517 to generate that metadata from source distributions, or you can extract that metadata directly from wheels.

[jaraco]

In GitLab by @jaraco on Sep 2, 2019, 14:39

I don't believe there's anything more for importlib_metadata to do here, but feel free to continue the conversation here or open a new issue as appropriate.

[jaraco]

In GitLab by @jaraco on Sep 2, 2019, 14:39

closed

[jaraco]

In GitLab by @lowlyocean on May 16, 2020, 21:09

Hi Jason, just wanted to flag this is also encountered by Nuitka when trying to use keyring module in standalone application:
Nuitka/Nuitka#710

Nuitka/Nuitka#711

[jaraco]

In GitLab by @jaraco on Jun 5, 2020, 17:22

Hi Daniel. Sorry you've encountered the issue. I don't see any reason why the recommendation for Nuitka is any different than for the other bundlers. The recommendation is for these bundlers to supply the metadata.

[jaraco]

In GitLab by @jaraco on Jun 5, 2020, 17:31

mentioned in merge request !120

[jaraco]

In GitLab by @pe2mbs on Jun 12, 2020, 10:00

Hi,
I've encountered the same problem, the problem comes from the package keyring. When importing the keyring package in a normal Python script application there is no problem. The problem occurs when building a stand alone application with PyInstaller, The build of the application goes without any issues. But when starting the application the import of the keyring throws an exception on the following:

try:
    from importlib import metadata
except ImportError:
    import importlib_metadata as metadata

Specially on this line in the 'importlib_metadata' module.

__version__ = version(__name__)

As it seems PyInstaller doesn't include the Package information in full.

I've patched import_metadata as described in the patch above, now it works just fine.

I like to appeal to include the patch into the package.

[jaraco]

In GitLab by @jaraco on Jun 14, 2020, 15:58

reopened

[jaraco]

In GitLab by @jaraco on Jun 14, 2020, 16:14

This issue has come up also in importlib_resources 94 and 100 and #124.

Rather than wrap this clunky code with a try/except block, I'm inclined to instead recommend that projects just not supply a __version__ and that any consumers that wish to resolve the version for a distribution package should just use the importlib-metadata APIs directly.

What this means is that the scope of the problem will be limited in environments like PyInstaller and cxFreeze and stash, avoiding errors when something tries to import importlib_metadata or importlib_resources, but they'll later encounter problems when an application like keyring attempts to load entry points or other metadata. The proper fix needs to be for these environments to support storing and providing metadata so that package authors can rely on the metadata for their package being available.

[jaraco]

In GitLab by @jaraco on Jun 14, 2020, 16:33

mentioned in commit 3aab89a1fcf30d31a7c6721d7d641f2bd70c5c5c

[jaraco]

In GitLab by @jaraco on Jun 14, 2020, 16:55

Is there a way for libraries like keyring to avoid this dependency on importlib_metadata?

Essentially no. Another way of phrasing the question would be "is there a way for libraries like keyring to load package metadata when a bundler omits that metadata?" The only way for something like that to work would be for all packages to have zero reliance on package metadata, which obviates the whole purpose of this project - to allow projects to have a meaningful relationship with their metadata and the metadata of other projects. Keyring (and any other project using entry points) relies heavily on the ability to detect entry points advertised by other projects.

I do think it's reasonable for bundlers to provide workarounds (patches) to accommodate these packages as a transition until the bundler can provide native support for supplying metadata, but I'd like to avoid these bundlers imposing that a workaround be implemented across the ecosystem and thus mask the underlying deficiency.

[jaraco]

In GitLab by @jaraco on Jun 14, 2020, 17:02

mentioned in commit af471bc

[jaraco]

In GitLab by @jaraco on Jun 26, 2020, 22:42

closed via merge request !125

[jaraco]

In GitLab by @jaraco on Jun 26, 2020, 22:42

mentioned in commit a274bcc