Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python updates with openssl300 #996

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

dhomeier
Copy link
Contributor

The (mostly) current Python versions have not been updated for about a year; upgrading to the current patch/security releases and switching from openssl110 to openssl300. Also fixes a number of utility scripts to use the correct Fink python-pyNN. Builds tested with both pkgconfig and pkgconf; a number of network tests are hanging (as with earlier releases) on 10.14.6; on 13.2 everything building cleanly for arm64 and x86_64.
As 3.10 is also already on the second-to-last regular release, adding python311 as current version.

@dhomeier dhomeier added new package Packages that do not yet exist in Fink. new upstream Package has an updated upstream version labels Feb 14, 2023
@dhomeier
Copy link
Contributor Author

dhomeier commented Feb 15, 2023

@dmacks concerning #920 (comment), which would obviously be superseded by this, the builds are using identical flags and order whether set up with pkg-config, pkgconf or ppkg-config. But regarding the circular dependency through meson and glib2, I have tested the scipy-py310 build with meson 1.0.0 and ppkg-config, and am seeing

found pkg-config '/opt/x64/bin/ppkg-config' but it is Strawberry Perl and thus broken. Ignoring...
Found Pkg-config: NO

just as noted in #918 (comment).
Scipy eventually seems to find its libraries some other way, but I am not sure if the build is complete (it is failing a number of tests).
I have not attempted to build any glib2 > 2.59, but it does not appear ppkg-config will resolve the circular deps issue.

@dmacks
Copy link
Member

dmacks commented Feb 17, 2023

Test-building as we speak. Tools/Scripts/pathfix.py is a script that can fix all the #! lines. And it's written in python, so it's a nice chickensnake-and-egg non-solution here:)

Debian's got some useful-looking patches for py311 (I looked only at that one because it's a totally new package). Especially test-no-random-order.diff would be a big help for us as packagers to see what changes during re-re-rebuild cycles (applicable on older pyXX as well).

@dmacks
Copy link
Member

dmacks commented Feb 18, 2023

python39 throws warnings configure: WARNING: unrecognized options: --with-pkgconfig

python39 and python310 (all I've tested so far) fail a self-test on 10.13:

======================================================================
ERROR: test_name_resolution (test.test_pkgutil.PkgutilTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sw/build.build/python310-3.10.10-1/Python-3.10.10/Lib/test/test_pkgutil.py", line 263, in test_name_resolution
    mod = importlib.import_module(uw)
  File "/sw/build.build/python310-3.10.10-1/Python-3.10.10/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'é'

which looks like python/cpython/issues/85326. Is it passing for you? Are you on a APFS disk rather than my HFS+? If so, then just need to have that test get skipped.

Update: For this test, 311 also fails, 38 is clean.

@dmacks
Copy link
Member

dmacks commented Feb 18, 2023

python38 looks good. In fact, the current (3.8.13-2) failed .deb validation:

Error: Compiled python module points to fink install dir.
	Offending file: /sw/lib/python3.8/__pycache__/_bootlocale.cpython-38.opt-2.pyc

which is resolved in this update.

There are a bunch of #!/usr/bin/env python3 (and a few #!/usr/bin/env python) still in lib/ but I don't know if any are actually user-facing vs just demos and tests.

@nieder
Copy link
Member

nieder commented Feb 18, 2023

Data point: 3.8-3.10 pass on 10.14.6. 3.11 hung on test_socket. When I disabled that test, 3.11 passed everything else

@dmacks
Copy link
Member

dmacks commented Feb 18, 2023

I see some SyntaxError messages while byte-compiling.

For python311:

+ perl -pi -e 's,/sw/build.build/python311-3.11.2-1/Python-3.11.2,/sw/lib/python3.11/config-3.11-darwin,;s/-lintl //' _sysconfigdata__darwin_darwin.py
+ DYLD_LIBRARY_PATH=/sw/build.build/root-python311-3.11.2-1/sw/lib
+ /sw/build.build/root-python311-3.11.2-1/sw/bin/python3.11 -m compileall -f -o 0 -o 1 -o 2 _sysconfigdata__darwin_darwin.py Tools
[...]
Compiling 'Tools/c-analyzer/c_parser/parser/_delim.py'...
***   File "Tools/c-analyzer/c_parser/parser/_delim.py", line 37
    ''')
       ^
SyntaxError: f-string: empty expression not allowed

For python310, the same one as python311, and also:

+ DYLD_LIBRARY_PATH=/sw/build.build/root-python310-3.10.10-1/sw/lib
+ /sw/build.build/root-python310-3.10.10-1/sw/bin/python3.10 -m compileall -o 0 -o 1 -o 2 Tools
[...]
Compiling 'Tools/test2to3/maintest.py'...
***   File "Tools/test2to3/maintest.py", line 8
    except ImportError, e:
           ^^^^^^^^^^^^^^
SyntaxError: multiple exception types must be parenthesized
[...]
Compiling 'Tools/test2to3/test/test_foo.py'...
***   File "Tools/test2to3/test/test_foo.py", line 7
    print 'In test_foo, using Python %s...' % (sys.version_info,)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?
[...]
Compiling 'Tools/test2to3/test2to3/hello.py'...
***   File "Tools/test2to3/test2to3/hello.py", line 3
    print "Hello, world"
    ^^^^^^^^^^^^^^^^^^^^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?

For python39, the same maintest.py and hello.py as python310, but test_foo.py is instead:

Compiling 'Tools/test2to3/test/test_foo.py'...
***   File "Tools/test2to3/test/test_foo.py", line 7
    print 'In test_foo, using Python %s...' % (sys.version_info,)
          ^
SyntaxError: invalid syntax

test2to3 is removed altogether as of 311ish (python/cpython/pull/26981) so we can ignore those (or maybe we should even nuke it?)

@dmacks
Copy link
Member

dmacks commented Feb 18, 2023

Debian has an interesting packaging approach, where they move the tkinter component (and a few others) to a separately-built package (https://packages.debian.org/source/sid/python3-stdlib-extensions). That gets a heavy dependency (tcltk) out of the main pythonXX package, which can both help resolve some dep-cycles. And it also makes the python-interp package lighter, which is important if we are increasing our reliance on it as a build-tool for other packages.

That shouldn't hold up things here, but it could be helpful in the future, and the jump to a new pythonXX is a cleaner time to start doing it than simply a new version or revision once we already have a given XX. I'll send it to Issues once this PR closes, but it relates to existing Debian patches, something already mentioned here.

@dhomeier
Copy link
Contributor Author

test2to3 is removed altogether as of 311ish (python/cpython/pull/26981) so we can ignore those (or maybe we should even nuke it?)

Yes, those are obviously not meant to be called directly without running them to 2to3 first, so for clarity that directory should perhaps be exempt from the compileall.

There are a bunch of #!/usr/bin/env python3 (and a few #!/usr/bin/env python) still in lib/ but I don't know if any are actually user-facing vs just demos and tests.

And even a #! /usr/local/bin/python |-D! I think their primary purpose is providing system library functions, but as they generally have __main__ entry points, in principle they could be called directly. No idea if that makes sense for any of them, but perhaps better to fix them like the ones in Tools/scripts.

@nieder
Copy link
Member

nieder commented Feb 18, 2023

On 12.6

  • 3.8 fails this:
test_mkfifo_dir_fd (test.test_posix.PosixTester) ... Fatal Python error: Segmentation fault

Current thread 0x0000000110c74600 (most recent call first):
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/test_posix.py", line 1260 in test_mkfifo_dir_fd
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/unittest/case.py", line 633 in _callTestMethod
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/unittest/case.py", line 676 in run
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/unittest/case.py", line 736 in __call__
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/unittest/suite.py", line 122 in run
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/unittest/suite.py", line 84 in __call__
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/unittest/suite.py", line 122 in run
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/unittest/suite.py", line 84 in __call__
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/unittest/runner.py", line 176 in run
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/support/__init__.py", line 2030 in _run_suite
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/support/__init__.py", line 2152 in run_unittest
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/test_posix.py", line 1892 in test_main
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/libregrtest/runtest.py", line 234 in _runtest_inner2
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/libregrtest/runtest.py", line 270 in _runtest_inner
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/libregrtest/runtest.py", line 153 in _runtest
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/libregrtest/runtest.py", line 193 in runtest
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/libregrtest/main.py", line 318 in rerun_failed_tests
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/libregrtest/main.py", line 694 in _main
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/libregrtest/main.py", line 637 in main
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/libregrtest/main.py", line 715 in main
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/__main__.py", line 2 in <module>
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/runpy.py", line 87 in _run_code
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/runpy.py", line 194 in _run_module_as_main
make: *** [Makefile:1142: test] Segmentation fault: 11
  • 3.9, 3.10, 3.11 pass tests but have SyntaxErrors like @dmacks (didn't pay attention if they're also present on 10.14.6)

@dmacks
Copy link
Member

dmacks commented Feb 18, 2023

The syntax error in _delim.py in 3.10 and 3.11 is python/cpython/issues/102033

@dmacks
Copy link
Member

dmacks commented Feb 18, 2023

On 12.6

* 3.8 fails this:
test_mkfifo_dir_fd (test.test_posix.PosixTester) ... Fatal Python error: Segmentation fault

Current thread 0x0000000110c74600 (most recent call first):
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/test_posix.py", line 1260 in test_mkfifo_dir_fd

Is this python/cpython/issues/97897 (whicih would specifcally affect OS X 12.x when using OS X 13.x SDK?) that seems like upstream fixes with 6d0a0191a4e5477bd843e62c24d7f3bcad4fd5fc ?

@nieder
Copy link
Member

nieder commented Feb 19, 2023

On 12.6

* 3.8 fails this:
test_mkfifo_dir_fd (test.test_posix.PosixTester) ... Fatal Python error: Segmentation fault

Current thread 0x0000000110c74600 (most recent call first):
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/test_posix.py", line 1260 in test_mkfifo_dir_fd

Is this python/cpython/issues/97897 (whicih would specifcally affect OS X 12.x when using OS X 13.x SDK?) that seems like upstream fixes with 6d0a0191a4e5477bd843e62c24d7f3bcad4fd5fc ?

Yes, that looks like the bug. Unfortunately, it hasn't been ported back to the 3.8 branch, and the patch doesn't apply cleanly (a lot of the #define HAVE_* logic doesn't exist in the 3.8 branch).

@dmacks
Copy link
Member

dmacks commented Feb 19, 2023

On 12.6

* 3.8 fails this:
test_mkfifo_dir_fd (test.test_posix.PosixTester) ... Fatal Python error: Segmentation fault

Current thread 0x0000000110c74600 (most recent call first):
  File "/opt/sw/build.build/python38-3.8.16-1/Python-3.8.16/Lib/test/test_posix.py", line 1260 in test_mkfifo_dir_fd

Is this python/cpython/issues/97897 (whicih would specifcally affect OS X 12.x when using OS X 13.x SDK?) that seems like upstream fixes with 6d0a0191a4e5477bd843e62c24d7f3bcad4fd5fc ?

Yes, that looks like the bug. Unfortunately, it hasn't been ported back to the 3.8 branch, and the patch doesn't apply cleanly (a lot of the #define HAVE_* logic doesn't exist in the 3.8 branch).

On my 10.13:

checking for mkfifoat... no
checking for mknod... yes
checking for mknodat... no

It sounds like the problem is autoconf is detecting (for you) something that exists but doesn't actually work, whereas for me it isn't even detected and so the source does something else.

Adding ac_cv_func_mkfifoat=no ac_cv_func_mknodat=no to ConfigureParams, that becomes

checking for mkfifoat... (cached) no
checking for mknod... yes
checking for mknodat... (cached) no

@nieder
Copy link
Member

nieder commented Feb 20, 2023

I tried to hack the upstream mkfifoat patch to work on the 3.8 branch. The test still fails.
On 12.6 (13.1 SDK), this is the configure output for 3.8 (and 3.9):

checking for mkfifoat... yes
checking for mknod... yes
checking for mknodat... yes

Adding ac_cv_func_mkfifoat=no ac_cv_func_mknodat=no to %c makes all tests pass for python38 on 12.6.
All the code around this mentions macOS 13, so it does look like something is being detected on macOS 12 when it shouldn't.

For CompileScript:

# https://github.com/python/cpython/issues/97897
if [ `uname -r | cut -f1 -d.` -eq 12 ]; then
	MKFIFOAT_CONFIG_PARAMS="ac_cv_func_mkfifoat=no ac_cv_func_mknodat=no
fi
	./configure %c $MKFIFOAT_CONFIG_PARAMS $DIST_CONFIG_PARAMS

$DIST_CONFIG_PARAMS exists, but is not defined anywhere. Would be interesting to see if macOS 11 has this issue to modify the conditional.

@dmacks
Copy link
Member

dmacks commented Feb 20, 2023

That wouldn't be the first time I've seen XCode SDK at some point start defaulting to "next higher version" According to the bug-report, there's confusion when it builds for what appears to be available but then tries to run with (obviously) the current version. The advance here is that at least there was a compile-time warning about it, albeit one that didn't seem as dire as it actually is.

@dmacks
Copy link
Member

dmacks commented Feb 20, 2023

DIST_CONFIG_PARAMS is used in some pythonXX builds, and I guess sits there inertly for when what it implemented was no longer needed. See ecbc060

@dmacks
Copy link
Member

dmacks commented Feb 22, 2023

The minimal fix for the _delim.py syntax-error in python310 and python311 is sobolevn/cpython@a930d37 (thanks upstream!)

@dmacks
Copy link
Member

dmacks commented Feb 24, 2023

This PR is dragging out a bit, so I'm forking the python311 portion to #1002, where we can work on splitting off tk before this new lang-version package lands in public repo. Please do not merge the py11 packages of this PR.

@nieder
Copy link
Member

nieder commented Feb 27, 2023

3.8 needs just ac_cv_func_mkfifoat=no ac_cv_func_mknodat=no on macOS 12.
3.9 seems fine as is everywhere.
3.10 would need sobolevn/cpython@a930d37.

I can cherry pick the individual commits from here, add the changes described. @dhomeier: thoughts?

@dhomeier
Copy link
Contributor Author

Thanks all for looking into the failures! I have tested all on 12.6; pushing the corresponding 3.11 fixes here as well so they may be later applied to #1002.

@@ -37,7 +37,7 @@ BuildDepends: <<
readline8,
sqlite3-dev (>= 3.30.1-1),
openssl300-dev (>= 3.0.1-1),
pkgconfig (>= 0.9.0-1),
pkgconf,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ppkg-config? And fix $PKG_CONFIG down below to match?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have not tested this, but python310 did not report any errors of the kind in #996 (comment), so it's probably safe to switch as well.

@TheSin-
Copy link
Member

TheSin- commented Feb 27, 2023

pkg-config is fixed now, the issue was glib2 which I fixed a while back.

@dhomeier
Copy link
Contributor Author

So no more circular deps??

@nieder
Copy link
Member

nieder commented Mar 1, 2024

Getting a test failure in test_xml_etree, even bumping to latest 3.10.13:

FAIL: test_simple_xml (test.test_xml_etree.XMLPullParserTest) (chunk_size=5)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sw/build.build/python310-3.10.13-1/Python-3.10.13/Lib/test/test_xml_etree.py", line 1414, in test_simple_xml
    self.assert_event_tags(parser, [
  File "/sw/build.build/python310-3.10.13-1/Python-3.10.13/Lib/test/test_xml_etree.py", line 1396, in assert_event_tags
    self.assertEqual([(action, elem.tag) for action, elem in events],
AssertionError: Lists differ: [('end', 'element')] != [('end', 'element'), ('end', 'empty-element')]

Second list contains 1 additional elements.
First extra element 1:
('end', 'empty-element')

- [('end', 'element')]
+ [('end', 'element'), ('end', 'empty-element')]

Wonder if it's related to a libxml2 change (haven't looked yet).

@dmacks
Copy link
Member

dmacks commented Mar 1, 2024

Getting a test failure in test_xml_etree, even bumping to latest 3.10.13:

FAIL: test_simple_xml (test.test_xml_etree.XMLPullParserTest) (chunk_size=5)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sw/build.build/python310-3.10.13-1/Python-3.10.13/Lib/test/test_xml_etree.py", line 1414, in test_simple_xml
    self.assert_event_tags(parser, [
  File "/sw/build.build/python310-3.10.13-1/Python-3.10.13/Lib/test/test_xml_etree.py", line 1396, in assert_event_tags
    self.assertEqual([(action, elem.tag) for action, elem in events],
AssertionError: Lists differ: [('end', 'element')] != [('end', 'element'), ('end', 'empty-element')]

Second list contains 1 additional elements.
First extra element 1:
('end', 'empty-element')

- [('end', 'element')]
+ [('end', 'element'), ('end', 'empty-element')]

Wonder if it's related to a libxml2 change (haven't looked yet).

Or new expat (per https://www.linuxfromscratch.org/blfs/view/svn/general/python3.html)

@dmacks
Copy link
Member

dmacks commented Mar 2, 2024

Python-upstream confirms there are test-failures due to the just-released expat-2.6.0 and has fixes. python/cpython#115133

@nieder
Copy link
Member

nieder commented Mar 2, 2024

What's left here (apart from expat fixes which are available 3.8-3.12 from upstream and python3.11 getting its own PR for Tk-icide) ? Some of pythons here use pkgconf (3.8, 3.9, 3.11) and others use ppkg-config (3.10). I never followed the discussion as to which was less dep heavy or what, but #923 suggests that others are moving towards pkgconf.

@dmacks
Copy link
Member

dmacks commented Mar 4, 2024

Offloaded a separate tracking ticket for the expat issue (which also ponders what versions of pythonXX we should continue to support at all).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new package Packages that do not yet exist in Fink. new upstream Package has an updated upstream version
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants