Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Python to 3.9.1, pip to 20.3.3 #30589

Closed
mkoeppe opened this issue Sep 17, 2020 · 53 comments
Closed

Upgrade Python to 3.9.1, pip to 20.3.3 #30589

mkoeppe opened this issue Sep 17, 2020 · 53 comments

Comments

@mkoeppe
Copy link
Member

mkoeppe commented Sep 17, 2020

This contains the upgrade to Python 3.9.1

To test, use

./configure --without-system-python3 --enable-download-from-upstream-url

Depends on #31050

CC: @slel @jhpalmieri @fchapoton @dimpase @vbraun @kiwifb

Component: packages: standard

Author: Matthias Koeppe

Branch/Commit: 41d7e7a

Reviewer: John Palmieri, Samuel Lelièvre

Issue created by migration from https://trac.sagemath.org/ticket/30589

@mkoeppe
Copy link
Member Author

mkoeppe commented Sep 17, 2020

Dependencies: #30184

@mkoeppe

This comment has been minimized.

@mkoeppe

This comment has been minimized.

@mkoeppe
Copy link
Member Author

mkoeppe commented Sep 17, 2020

Branch: u/mkoeppe/upgrade-python-3.9

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 17, 2020

Commit: e188c5e

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 17, 2020

Branch pushed to git repo; I updated commit sha1. New commits:

e188c5esrc/sage/cpython/debugimpl.c: Disable _type_debug for Python 3.9

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 18, 2020

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

d3dd329Merge branch 't/30595/remove_deprecated_sage_libs_ppl' into t/30184/support_python_3_9
68ef4c5src/sage/cpython/debugimpl.c: Disable _type_debug for Python 3.9
f3cb571src/sage/all.py: Filter out Python 3.9 deprecation warnings from ast
e16ded2src/sage/symbolic/ring.pyx (isidentifier): Remove py2-only code, use of deprecated parser module
27a57bcsrc/sage/misc/sageinspect.py: Adjust doctest for sage_getsourcelines for Python 3.9 - decorators are now part of the definition
39416fbsrc/sage/misc/sageinspect.py (SageArgSpecVisitor): Relax doctest pattern for python 3.9
5548c0dsrc/sage/misc/latex.py (has_latex_attr): Relax doctest pattern for python 3.9
4a9362esrc/sage/combinat/root_system/pieri_factors.py: Use sorted in doctest for python 3.9
49eccd9src/sage/geometry/polyhedron/palp_database.py: Relax doctest pattern for python 3.9
5e89ab2Merge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 18, 2020

Changed commit from e188c5e to 5e89ab2

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 24, 2020

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

b21ed97build/pkgs/python3/spkg-configure.m4: Ensure that PYTHON_FOR_VENV is a full path
49aa061Revert "build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'"
a8b77cdRevert "only test python3, to be 3.7 or 3.8"
9403629build/bin/sage-system-python: Work around LC_ALL=C
ff0dbc6build/sage_bootstrap/uncompress/tar_file.py: Fix encoding to utf-8
f05a110Merge branch 't/30008/after__30053__sphinx_3_1_2_does_not_build_on_ubuntu__trusty_xenial_bionic___debian_jessie__centos_7__again_' into t/30576/public/30576
945c8c5build/bin/sage-site (--docbuild): Set an English locale suuch as C.utf-8, but not the C locale, to ensure correct operation on Python 3.6
8906897Merge branch 't/30576/public/30576' into t/30546/build/py3x_conf
60137bcMerge branch 't/30546/build/py3x_conf' into t/30184/support_python_3_9
316ae6dMerge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 24, 2020

Changed commit from 5e89ab2 to 316ae6d

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 24, 2020

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

ca58693build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
5a747c4build/bin/sage-pip-{install,uninstall}: Remove pip2 support
9ee2110build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
d7aac84src/doc/en/developer/packaging.rst: Update sdh_... documentation
9b7c7a0build/bin/sage-pip-{install,uninstall}: Fix typo in comment
4135e8bbuild/bin/sage-pip-install: Remove an outdated comment
f2e7075Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels
a2dbb8dMerge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30546/build/py3x_conf
8c756edMerge branch 't/30546/build/py3x_conf' into t/30184/support_python_3_9
5cf1bafMerge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 24, 2020

Changed commit from 316ae6d to 5cf1baf

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 25, 2020

Branch pushed to git repo; I updated commit sha1. New commits:

f19d9a4allow absolute names as arguments to --with-python
4f7d588Merge branch 't/30546/build/py3x_conf' into t/30184/support_python_3_9
b3c1dc1Merge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Sep 25, 2020

Changed commit from 5cf1baf to b3c1dc1

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Oct 6, 2020

Changed commit from b3c1dc1 to 6d3a0fa

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Oct 6, 2020

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

255d22bbuild/pkgs/python3: Update to 3.9.0rc2
6d3a0fabuild/pkgs/python3: Update to 3.9.0

@mkoeppe
Copy link
Member Author

mkoeppe commented Oct 6, 2020

Changed dependencies from #30184 to none

@mkoeppe

This comment has been minimized.

@mkoeppe mkoeppe changed the title Test ticket: Python 3.9.0rc2 Test ticket: Python 3.9.0 Oct 6, 2020
@mkoeppe
Copy link
Member Author

mkoeppe commented Oct 6, 2020

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 15, 2020

comment:13

3.9.1 is out https://www.python.org/downloads/release/python-391/
Release Date: Dec. 7, 2020

"3.9.1 is the first version of Python to support macOS 11 Big Sur. With Xcode 11 and later it is now possible to build “Universal 2” binaries which work on Apple Silicon. We are providing such an installer as the macos11.0 variant. This installer can be deployed back to older versions, tested down to OS X 10.9. As we are waiting for an updated version of pip, please consider the macos11.0 installer experimental."

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 15, 2020

comment:14

Not sure which updated version of pip they are referring to. 20.3 (2020-11-30) lists "Add support for MacOS Big Sur compatibility tags. (#9138)" https://pip.pypa.io/en/stable/news/#id15

No new news items regarding Big Sur in 20.3.3 (2020-12-15)

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 16, 2020

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

29aee87build/pkgs/python3: Update to 3.9.0rc2
e61d464build/pkgs/python3: Update to 3.9.0
f8bb56dbuild/pkgs/python3: Update to 3.9.1

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 16, 2020

Changed commit from 6d3a0fa to f8bb56d

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 16, 2020

@mkoeppe

This comment has been minimized.

@jhpalmieri
Copy link
Member

comment:28

Right now homebrew's python3 is shadowing Xcode's, so I don't know. I would rather not uninstall homebrew's since it is used by other homebrew packages that I want to keep.

I have bigger issues right now: I can make almost no progress building Sage with 9.3.beta4. For example, with #31050 merged with beta4, gmpy2 fails to build:

  Compiling with an SDK that doesn't seem to exist: /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk
  Please check your Xcode installation
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -O2 -g -I./src -I/usr/local/include -I/usr/local/opt/[email protected]/include -I/usr/local/opt/sqlite/include -I/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/include -I/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/gmpy2.c -o build/temp.macosx-11-x86_64-3.9/src/gmpy2.o -DSHARED=1
  clang: warning: no such sysroot directory: '/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk' [-Wmissing-sysroot]
  In file included from src/gmpy2.c:445:
  /usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
  #include <stdio.h>

Is my command-line tools installation broken? It is certainly true that the named sysroot directory does not exist, but instead it's .../MacOSX11.1.sdk. Where is that path stored? I've tried uninstalling and reinstalling the command-line tools, uninstalling and reinstalling homebrew and its packages, but no luck.

In contrast, with just #31050 (which was based on beta3), I see the following in the gmpy2 log file:

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I./src -I/usr/local/include -I/usr/local/opt/[email protected]/include -I/usr/local/opt/sqlite/include -I/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/include -I/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/gmpy2.c -o build/temp.macosx-11-x86_64-3.9/src/gmpy2.o -DSHARED=1

(same wrong path given to -I) but gmpy2 builds successfully.

Separately, on a machine running Catalina, I'm having problems with cypari...

@dimpase
Copy link
Member

dimpase commented Dec 18, 2020

comment:29

In a (working) Homebrew installation on Catalina, I don't have these /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk paths used by the tools - ie.
gcc/clang is installed in
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
and not in /Library/....

This can be seen in config.log

@jhpalmieri
Copy link
Member

comment:30

config.log is a good place to look. I see that gfortran -v prints that it is configured using --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk. I reinstalled homebrew's gcc but it didn't help. Any ideas how this is path determined? What does gfortran -v say for you?

(For gcc/clang, I see InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin.)

@jhpalmieri
Copy link
Member

comment:31

Well, this is probably not the right thing to do in the long term, but I created a symlink connecting MacOSX11.1.sdk to MacOSX11.0.sdk. The build still fails on scipy with

  /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/[email protected]/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so
  gfortran: warning: couldn't understand version 11
  ld: library not found for -lgcc_s.10.4
  collect2: error: ld returned 1 exit status
  error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/[email protected]/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy

but the other problems seem to be solved.

@dimpase
Copy link
Member

dimpase commented Dec 21, 2020

comment:32

Replying to @jhpalmieri:

Well, this is probably not the right thing to do in the long term, but I created a symlink connecting MacOSX11.1.sdk to MacOSX11.0.sdk. The build still fails on scipy with

  /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/[email protected]/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so
  gfortran: warning: couldn't understand version 11
  ld: library not found for -lgcc_s.10.4
  collect2: error: ld returned 1 exit status
  error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/[email protected]/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy

but the other problems seem to be solved.

scipy probably needs an update, and/or a hack:
scipy/scipy#13102 (comment)

@jhpalmieri
Copy link
Member

comment:33

I'm not clever enough to figure out how to hack scipy so it builds. If I issue the command export SYSTEM_VERSION_COMPAT=1 globally before building Sage, then cython doesn't build. If I just issue it in the scipy spkg-install script, then scipy doesn't build with the same error as in comment:31.

Edit: the error with cython after executing export SYSTEM_VERSION_COMPAT=1:

ERROR: Cython-0.29.21-cp39-cp39-macosx_11_0_x86_64.whl is not a supported wheel on this platform.

This is with this ticket and an updated setuptools and setuptools_scm.

@jhpalmieri
Copy link
Member

comment:34

I guess we should move my derailment of the discussion to #30651.

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 27, 2020

comment:35

Does someone have time to review this ticket? The tests have already run successfully.

Note that an update to latest 3.8.x is not a good alternative:

https://www.python.org/downloads/release/python-387/
Python 3.8.7 - Release Date: Dec. 21, 2020
macOS 11 Big Sur not fully supported

Python 3.8.7 is not yet fully supported on macOS 11 Big Sur. It will install on macOS 11 Big Sur and will run on Apple Silicon Macs using Rosetta 2 translation. However, a few features do not work correctly, most noticeably those involving searching for system libraries (vs user libraries) such as ctypes.util.find_library() and in Distutils. This limitation affects both Apple Silicon and Intel processors. We are looking into improving the situation for Python 3.8.8.

@slel
Copy link
Member

slel commented Jan 3, 2021

comment:38

On macOS 10.14.6 with lots of Homebrew packages,
successfully ran

$ git trac checkout 30589
$ source .homebrew-build-env
$ ./bootstrap -q
$ ./configure --without-system-python3 --enable-download-from-upstream-url -q
$ make -s V=0

resulting in a build of Sage 9.3.beta4 based on Python 3.9.1.

But something that worked for another Sage install currently at 9.3.beta5:

$ V=0 sage -i pynormaliz

failed here while attempting to build e-antic.

@jhpalmieri
Copy link
Member

comment:39

On OS X Big Sur + various homebrew packages, if I use

% ./configure --with-system-python3=no --enable-download-from-upstream-url 

then Sage builds (if I also incorporate various other changes from #30651), but I get doctest failures of the form

File "src/sage/misc/sagedoc.py", line 646, in sage.misc.sagedoc.format
Failed example:
    cython('\n'.join(cython_code))
Expected nothing
Got:
    ld: warning: dylib (/usr/local/lib/libmpfr.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgmpxx.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgsl.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libntl.dylib) was built for newer macOS version (11.0) than being linked (10.9)

We could (a) not worry about these for now, (b) filter out these warnings, (c) tinker with MACOSX_DEPLOYMENT_TARGET, or (d) something else. I don't have an opinion about which way to go.

@mkoeppe
Copy link
Member Author

mkoeppe commented Jan 7, 2021

comment:40

These linker warnings are annoying and I think we should filter them out for the doctests. I think it should be a separate ticket. I think I have seen them on other platforms too.

@mkoeppe
Copy link
Member Author

mkoeppe commented Jan 7, 2021

comment:41

This is now #31204

@mkoeppe
Copy link
Member Author

mkoeppe commented Jan 7, 2021

comment:42

Replying to @slel:

But something that worked for another Sage install currently at 9.3.beta5:

$ V=0 sage -i pynormaliz

failed here while attempting to build e-antic.

Is this on homebrew too?

@mkoeppe
Copy link
Member Author

mkoeppe commented Jan 11, 2021

comment:43

In any case, let's worry about optional packages like e-antic later.

Let's get this in please.

@jhpalmieri
Copy link
Member

comment:44

Sure, let's proceed.

@jhpalmieri
Copy link
Member

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/426748847 to John Palmieri, Samuel Lelièvre

@vbraun
Copy link
Member

vbraun commented Jan 24, 2021

Changed branch from u/mkoeppe/upgrade-python-3.9 to 41d7e7a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants