test_pkgutil:test_name_resolution fails on macOS HFS+ file systems #85326

RJ722 · 2020-06-29T06:56:41Z

BPO	41154
Nosy	@brettcannon, @vsajip, @ned-deily, @merwok, @zooba, @dstufft, @The-Compiler, @RJ722

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-02-03.18:05:45.937>
created_at = <Date 2020-06-29.06:56:41.028>
labels = ['tests', 'build', 'library', '3.9', '3.10']
title = 'test_pkgutil:test_name_resolution fails on macOS HFS+ file systems'
updated_at = <Date 2021-02-03.18:05:45.936>
user = 'https://github.com/RJ722'

bugs.python.org fields:

activity = <Date 2021-02-03.18:05:45.936>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-03.18:05:45.937>
closer = 'steve.dower'
components = ['Build', 'Distutils', 'Tests']
creation = <Date 2020-06-29.06:56:41.028>
creator = 'RJ722'
dependencies = []
files = []
hgrepos = []
issue_num = 41154
keywords = []
message_count = 8.0
messages = ['372549', '372550', '372551', '372603', '372624', '376141', '376165', '386244']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'vinay.sajip', 'ned.deily', 'eric.araujo', 'steve.dower', 'dstufft', 'The Compiler', 'RJ722']
pr_nums = []
priority = 'critical'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue41154'
versions = ['Python 3.9', 'Python 3.10']

RJ722 · 2020-06-29T06:56:40Z

After configuring and building using the command:

./configure --with-pydebug ** make -j

I ran the test suite (without changing anything) and saw that test_pkg has failed. Here is the output of ./python.exe -m test -v test_pkgutil:

== CPython 3.10.0a0 (heads/master:cd3c2bdd5d, Jun 28 2020, 13:29:09) [Clang 9.0.0 (clang-900.0.39.2)]
== macOS-10.12.6-x86_64-i386-64bit little-endian
== cwd: /Users/rahuljha/Documents/cpython/build/test_python_10678æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 2.01 Run tests sequentially
0:00:00 load avg: 2.01 [1/1] test_pkgutil
test_getdata_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_getdata_zipfile (test.test_pkgutil.PkgutilTests) ... ok
test_name_resolution (test.test_pkgutil.PkgutilTests) ... ERROR
test_unreadable_dir_on_syspath (test.test_pkgutil.PkgutilTests) ... ok
test_walk_packages_raises_on_string_or_bytes_input (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_zipfile (test.test_pkgutil.PkgutilTests)
Tests the same as test_walkpackages_filesys, only with a zip file. ... ok
test_alreadyloaded (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_getdata_pep302 (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_iter_importers (test.test_pkgutil.ExtendPathTests) ... ok
test_mixed_namespace (test.test_pkgutil.ExtendPathTests) ... ok
test_simple (test.test_pkgutil.ExtendPathTests) ... ok
test_nested (test.test_pkgutil.NestedNamespacePackageTest) ... ok
test_find_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_find_loader_missing_module (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_importer_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_None_in_sys_modules (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_loader_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_spec_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_spec_attribute_none (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_importer_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_iter_importers_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_loader_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok

======================================================================
ERROR: test_name_resolution (test.test_pkgutil.PkgutilTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/rahuljha/Documents/cpython/Lib/test/test_pkgutil.py", line 262, in test_name_resolution
    mod = importlib.import_module(uw)
  File "/Users/rahuljha/Documents/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'é'

Ran 24 tests in 0.186s

FAILED (errors=1)
test test_pkgutil failed
test_pkgutil failed

== Tests result: FAILURE ==

1 test failed:
test_pkgutil

Total duration: 482 ms
Tests result: FAILURE
Py vulture ~/Documents/cpython master !1 ?1 ❯ ./python.exe -m test -v test_pkgutil
== CPython 3.10.0a0 (heads/master:cd3c2bdd5d, Jun 28 2020, 13:29:09) [Clang 9.0.0 (clang-900.0.39.2)]
== macOS-10.12.6-x86_64-i386-64bit little-endian
== cwd: /Users/rahuljha/Documents/cpython/build/test_python_21819æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 2.69 Run tests sequentially
0:00:00 load avg: 2.69 [1/1] test_pkgutil
test_getdata_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_getdata_zipfile (test.test_pkgutil.PkgutilTests) ... ok
test_name_resolution (test.test_pkgutil.PkgutilTests) ... ERROR
test_unreadable_dir_on_syspath (test.test_pkgutil.PkgutilTests) ... ok
test_walk_packages_raises_on_string_or_bytes_input (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_zipfile (test.test_pkgutil.PkgutilTests)
Tests the same as test_walkpackages_filesys, only with a zip file. ... ok
test_alreadyloaded (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_getdata_pep302 (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_iter_importers (test.test_pkgutil.ExtendPathTests) ... ok
test_mixed_namespace (test.test_pkgutil.ExtendPathTests) ... ok
test_simple (test.test_pkgutil.ExtendPathTests) ... ok
test_nested (test.test_pkgutil.NestedNamespacePackageTest) ... ok
test_find_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_find_loader_missing_module (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_importer_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_None_in_sys_modules (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_loader_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_spec_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_spec_attribute_none (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_importer_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_iter_importers_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_loader_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok

======================================================================
ERROR: test_name_resolution (test.test_pkgutil.PkgutilTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/rahuljha/Documents/cpython/Lib/test/test_pkgutil.py", line 262, in test_name_resolution
    mod = importlib.import_module(uw)
  File "/Users/rahuljha/Documents/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'é'

Ran 24 tests in 0.184s

FAILED (errors=1)
test test_pkgutil failed
test_pkgutil failed

== Tests result: FAILURE ==

1 test failed:
test_pkgutil

Total duration: 635 ms
Tests result: FAILURE

ned-deily · 2020-06-29T07:24:19Z

This same failure is observed with the most recent 3.9beta (3.9.0b3) when running on macOS 10.9 with the binaries from the python.org macOS installer. However, no failure is seen when running the same Python on macOS 10.15. I have not investigated further with other releases. This test failure should be addressed prior t0 3.9.0rc1.

ned-deily · 2020-06-29T07:25:37Z

PS, the problem may have been introduced by the changes for bpo-12915.

vsajip · 2020-06-29T17:06:04Z

The problem appears to be in importlib. I put in a breakpoint in the test just before it tries to import the module é. (At this point it has successfully imported a module called वमस in an earlier loop iteration.)

Then, in a different terminal console, I cd'd to the test directory and tried the import directly:
$ ls -l
total 0
drwxr-xr-x  3 xxxxxxxx  xxxxx  102 29 Jun 17:55 é
drwxr-xr-x  4 xxxxxxxx  xxxxx  136 29 Jun 17:55 वमस
$ ~/projects/cpython/python.exe
Python 3.10.0a0 (heads/master:a3ad95d, Jun 29 2020, 17:50:09)
[Clang 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import वमस
>>> import é
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'é'

So for some reason, importlib is failing to import the é module. (The é/init.py is a zero-byte file.)

Adding Brett to nosy.

brettcannon · 2020-06-29T19:46:05Z

I wonder if it's a problem from Unicode representation since I don't think importlib tries to have a standard normalization form of Unicode. I could see that leading to how the file system encoded something differing from how it was encoded in the test file, leading to a claim the file doesn't exist.

And thanks to binary file paths that isn't necessarily a wrong result. :P

ned-deily · 2020-08-31T07:52:58Z

I wonder if it's a problem from Unicode representation

That does seem to be the issue. In particular, there are differences in Unicode representation between file names on traditional HFS+ versus the newer APFS. APFS is the default for system file systems as of macOS 10.13; unlike HFS+, APFS is by default file name normalization insensitive. By creating two disk image files systems on a current macOS system, it's easy to demonstrate that test_name_resolution passes on APFS but fails on HFS+. So the question becomes what to do about the test. Without digging into it further, I suppose we could add something to test whether the file system where test temp files are created is normalization insensitive and, if so, skip. Or use brute force and always skip on macOS. In any case, while annoying, it probably doesn't need to be a "deferred blocker".

brettcannon · 2020-08-31T19:57:06Z

I think if we can craft a skip check that is targeted enough then that's our best solution, but barring that then skipping for all of macOS wouldn't be the end of the world (maybe a slightly less brute force is a macOS version check?).

zooba · 2021-02-03T18:05:46Z

Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools

RJ722 mannequin added 3.10 only security fixes build The build process and cross-build stdlib Python modules in the Lib dir tests Tests in the Lib/test dir labels Jun 29, 2020

ned-deily added 3.9 only security fixes deferred-blocker labels Jun 29, 2020

ned-deily changed the title ~~test_pkgutil:test_name_resolution fails on master~~ test_pkgutil:test_name_resolution fails on some platforms Jun 29, 2020

ned-deily added 3.9 only security fixes deferred-blocker labels Jun 29, 2020

ned-deily changed the title ~~test_pkgutil:test_name_resolution fails on master~~ test_pkgutil:test_name_resolution fails on some platforms Jun 29, 2020

ned-deily changed the title ~~test_pkgutil:test_name_resolution fails on some platforms~~ test_pkgutil:test_name_resolution fails on macOS HFS+ file systems Aug 31, 2020

ned-deily removed the deferred-blocker label Aug 31, 2020

ned-deily changed the title ~~test_pkgutil:test_name_resolution fails on some platforms~~ test_pkgutil:test_name_resolution fails on macOS HFS+ file systems Aug 31, 2020

ned-deily removed the deferred-blocker label Aug 31, 2020

zooba closed this as completed Feb 3, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

dmacks mentioned this issue Feb 18, 2023

Python updates with openssl300 fink/fink-distributions#996

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_pkgutil:test_name_resolution fails on macOS HFS+ file systems #85326

test_pkgutil:test_name_resolution fails on macOS HFS+ file systems #85326

RJ722 mannequin commented Jun 29, 2020

RJ722 mannequin commented Jun 29, 2020

ned-deily commented Jun 29, 2020

ned-deily commented Jun 29, 2020

vsajip commented Jun 29, 2020

brettcannon commented Jun 29, 2020

ned-deily commented Aug 31, 2020

brettcannon commented Aug 31, 2020

zooba commented Feb 3, 2021

test_pkgutil:test_name_resolution fails on macOS HFS+ file systems #85326

test_pkgutil:test_name_resolution fails on macOS HFS+ file systems #85326

Comments

RJ722 mannequin commented Jun 29, 2020

RJ722 mannequin commented Jun 29, 2020

ned-deily commented Jun 29, 2020

ned-deily commented Jun 29, 2020

vsajip commented Jun 29, 2020

brettcannon commented Jun 29, 2020

ned-deily commented Aug 31, 2020

brettcannon commented Aug 31, 2020

zooba commented Feb 3, 2021