tox.ini: Add local-macos-nohomebrew environments, deployment targets 10.14, 10.15, 11.1 #31567

mkoeppe · 2021-03-26T21:36:43Z

This will do a "best effort" isolation to avoid using a homebrew installation in /usr/local for anything except bootstrapping and unpacking xz tarballs + a little trick for liblzma.h, which is missing on macOS.

To test:

$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode

It can then be checked that no libraries from /usr/local leak in, using

find prefix/lib -name "*.so" -o -name "*.dylib*" -exec otool -L {} \;

(Building R is disabled because the R package build still pokes around in /usr/local and finds libintl and liblzma.)

To set specific deployment targets, use:

$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.14-nohomebrew-python3_xcode -- config.status
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.15-nohomebrew-python3_xcode -- config.status
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-11.1-nohomebrew-python3_xcode -- config.status

... and with python3.7 from XCode:

$ tox -e local-macos-10.14-nohomebrew-python3_xcode-python3.7 -- config.status

(Remove config.status from the end of these lines to have the actual build done.)

These environments allow building Sage for older macOS versions, for example for building a 10.14 distribution on a 10.15 machine. The wheels are then correctly tagged, for example pplpy-0.8.6-cp38-cp38-macosx_10_14_x86_64.whl.

Cannot go in the opposite direction (building for 11 on a 10.15 machine) -- because pip will refuse to install a previously built wheel that is tagged for 11.)

Depends on #31552
Depends on #31562
Depends on #31409
Depends on #31584

CC: @jhpalmieri @zlscherr

Component: porting

Author: Matthias Koeppe

Branch/Commit: ed1a089

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/31567

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-03-26T23:33:03Z

Branch: u/mkoeppe/tox_ini__add_local_macos_nohomebrew_environments

sagetrac-git · 2021-03-27T00:16:27Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`87e2dba`	`tox.ini: Add local-macos-nohomebrew and configuration facors macos-{10.14,10.15,11.1}`

sagetrac-git · 2021-03-27T00:16:27Z

Commit: 87e2dba

mkoeppe · 2021-03-27T00:28:06Z

Author: Matthias Koeppe

sagetrac-git · 2021-03-27T02:21:32Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1ed9302`	`tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8}`

sagetrac-git · 2021-03-27T02:21:32Z

Changed commit from 87e2dba to 1ed9302

sagetrac-git · 2021-03-27T02:21:57Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`0458f23`	`tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8}`

sagetrac-git · 2021-03-27T02:21:57Z

Changed commit from 1ed9302 to 0458f23

mkoeppe · 2021-03-27T02:49:50Z

comment:8

This is best tested together with #31552, #31562, which do some of the necessary isolation work.

More isolation work is needed at least for pillow and freetype - as observed in #31396 comment:32

mkoeppe · 2021-03-27T03:00:51Z

Dependencies: #31552, #31562

sagetrac-git · 2021-03-27T03:01:40Z

Changed commit from 0458f23 to 9d9c562

sagetrac-git · 2021-03-27T03:01:40Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8b5b273`	`build/pkgs/singular/checksums.ini: Use 4.2.0p1+2021-03-24+sage-2`
`39629dd`	`Merge #31552`
`1ef46d0`	`build/pkgs/giac: Add another patch in patches/autotools`
`89f61d2`	`build/pkgs/giac: Update to 1.6.0.47p3`
`9d9c562`	`Merge #31562`

sagetrac-git · 2021-03-28T00:54:45Z

Branch pushed to git repo; I updated commit sha1. New commits:

`236ea3c`	`tox.ini (macos-nohomebrew): Pass compiler configuration to configure using variables other than CC, CXX.`

sagetrac-git · 2021-03-28T00:54:45Z

Changed commit from 9d9c562 to 236ea3c

sagetrac-git · 2021-03-28T00:55:03Z

Changed commit from 236ea3c to 92db4e4

sagetrac-git · 2021-03-28T00:55:03Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`92db4e4`	`tox.ini (macos-nohomebrew): Pass compiler configuration to configure using variables other than CC, CXX.`

mkoeppe · 2021-03-28T01:47:31Z

Changed dependencies from #31552, #31562 to #31552, #31562, #31409

sagetrac-git · 2021-03-28T01:54:24Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`737b21c`	`src/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2`
`4403924`	`src/sage/interfaces/r.py: Mark all tests # optional - rpy2`
`405ebb9`	`More # optional - rpy2`
`b7c1576`	`Merge tag '9.3.beta8' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional`
`ce3b35f`	`More # optional - r`
`49c10c1`	`Revert "build/pkgs/{r,rpy2}: Downgrade to optional"`
`a67300b`	`Merge #30383`
`f0a5fb5`	`configure.ac: Add option --disable-r`
`f04c134`	`Merge branch 't/30383/new_package_type__optional_enabled_by_default' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional`
`7966b66`	`Merge #31409`

sagetrac-git · 2021-03-28T06:43:34Z

Branch pushed to git repo; I updated commit sha1. New commits:

`dffcdbc`	`tox.ini (macos-nohomebrew): Fix up pillow build by setting ZLIB_ROOT`

sagetrac-git · 2021-03-28T06:43:34Z

Changed commit from b5edde2 to dffcdbc

mkoeppe · 2021-03-29T01:26:21Z

comment:22

pillow still finds various libraries in /usr/local because of very creative discovery code that finds everything in /usr/local because we told it that it can have /usr/local/opt/xz/include - see _add_directory (https://github.com/python-pillow/Pillow/blob/8.1.x/setup.py#L197)

sagetrac-git · 2021-03-29T01:59:42Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3779a25`	`tox.ini (macos-nohomebrew): Disable more libraries to remove dependencies on homebrew`

sagetrac-git · 2021-03-29T01:59:42Z

Changed commit from dffcdbc to 3779a25

mkoeppe · 2021-03-29T02:08:31Z

comment:24

Fixed pillow. Now I can build a wheel in #31396 that does not have any dependencies on shared libraries in /usr/local.

jhpalmieri · 2021-03-30T22:02:25Z

comment:25

For what it's worth, I see a few suspicious items in the log files. (This is on a system with homebrew and with source .homebrew-build-env in my shell init file. That may be causing some of these.)

cvxopt includes flags -L/usr/local/lib and -I/usr/local/include when calling gcc
pillow says Appending path /usr/local/Cellar/xz/5.2.5/include and then the gcc flag -I/usr/local/Cellar/xz/5.2.5/include.

mkoeppe · 2021-03-30T22:12:43Z

comment:26

The xz include is on purpose -- see comment added in tox.ini

mkoeppe · 2021-03-30T22:39:10Z

comment:27

Replying to @jhpalmieri:

For what it's worth, I see a few suspicious items in the log files. (This is on a system with homebrew and with source .homebrew-build-env in my shell init file. That may be causing some of these.)

Yes, this a good setting for testing this ticket. The build using tox is designed to be isolated from the effects of .homebrew-build-env.

cvxopt includes flags -L/usr/local/lib and -I/usr/local/include when calling gcc

This is probably https://github.com/cvxopt/cvxopt/blob/master/setup.py#L55 in combination with a bug in our build system - #31584.

mkoeppe · 2021-03-30T22:46:45Z

Changed dependencies from #31552, #31562, #31409 to #31552, #31562, #31409, #31584

sagetrac-git · 2021-03-30T22:56:12Z

Changed commit from 3779a25 to ed1a089

sagetrac-git · 2021-03-30T22:56:12Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7ff26df`	`git grep -l SAGE_SUITESPARSE \| xargs sed -i .bak s/SAGE_SUITESPARSE_LOCALINSTALL/SAGE_SUITESPARSE_PREFIX/g`
`52537d1`	`build/pkgs/cvxopt/spkg-install.in: Actually use the value of SAGE_SUITESPARSE_PREFIX`
`ed1a089`	`Merge #31584`

jhpalmieri · 2021-03-31T02:04:23Z

Reviewer: John Palmieri

jhpalmieri · 2021-03-31T02:04:23Z

comment:30

Okay, this looks good to me.

mkoeppe · 2021-03-31T02:11:32Z

comment:31

Thanks!

jhpalmieri · 2021-03-31T02:35:50Z

comment:32

By the way, should commands like make distclean also clean the .tox directory?

mkoeppe · 2021-03-31T02:59:16Z

comment:33

I think it should not because .tox is not a "build artifact" - if this reasoning makes sense.

jhpalmieri · 2021-03-31T16:49:25Z

comment:34

That makes sense. Maybe I need to learn more about tox. What command should I use if I want to restart EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode from scratch? Just delete .tox/local-macos-nohomebrew-python3?

mkoeppe · 2021-03-31T16:52:46Z

comment:35

Replying to @jhpalmieri:

That makes sense. Maybe I need to learn more about tox. What command should I use if I want to restart EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode from scratch? Just delete .tox/local-macos-nohomebrew-python3?

That works, but you can also pass the -r option to tox, which will start from scratch.

jhpalmieri · 2021-03-31T17:21:28Z

comment:36

Great, thanks!

vbraun · 2021-05-27T20:28:46Z

Changed branch from u/mkoeppe/tox_ini__add_local_macos_nohomebrew_environments to ed1a089

mkoeppe added this to the sage-9.3 milestone Mar 26, 2021

mkoeppe added c: porting labels Mar 26, 2021

mkoeppe changed the title ~~tox.ini: Add local-macos-nohomebrew environments~~ tox.ini: Add local-macos-nohomebrew environments, deployment targets 10.14, 10.15, 11.1 Mar 27, 2021

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Mar 27, 2021

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs work label Mar 28, 2021

mkoeppe added s: needs review and removed s: needs work labels Mar 29, 2021

jhpalmieri added s: positive review and removed s: needs review labels Mar 31, 2021

mkoeppe modified the milestones: sage-9.3, sage-9.4 Apr 18, 2021

vbraun removed the s: positive review label May 27, 2021

vbraun closed this as completed in 9e7b0bd May 27, 2021

tox.ini: Add local-macos-nohomebrew environments, deployment targets 10.14, 10.15, 11.1 #31567

tox.ini: Add local-macos-nohomebrew environments, deployment targets 10.14, 10.15, 11.1 #31567

Comments

mkoeppe commented Mar 26, 2021

mkoeppe commented Mar 26, 2021

sagetrac-git mannequin commented Mar 27, 2021

sagetrac-git mannequin commented Mar 27, 2021

mkoeppe commented Mar 27, 2021

This comment has been minimized.

This comment has been minimized.

sagetrac-git mannequin commented Mar 27, 2021

sagetrac-git mannequin commented Mar 27, 2021

sagetrac-git mannequin commented Mar 27, 2021

sagetrac-git mannequin commented Mar 27, 2021

This comment has been minimized.

mkoeppe commented Mar 27, 2021

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Mar 27, 2021

sagetrac-git mannequin commented Mar 27, 2021

sagetrac-git mannequin commented Mar 27, 2021

sagetrac-git mannequin commented Mar 28, 2021

sagetrac-git mannequin commented Mar 28, 2021

sagetrac-git mannequin commented Mar 28, 2021

sagetrac-git mannequin commented Mar 28, 2021

This comment has been minimized.

mkoeppe commented Mar 28, 2021

sagetrac-git mannequin commented Mar 28, 2021

sagetrac-git mannequin commented Mar 28, 2021

sagetrac-git mannequin commented Mar 28, 2021

mkoeppe commented Mar 29, 2021

sagetrac-git mannequin commented Mar 29, 2021

sagetrac-git mannequin commented Mar 29, 2021

mkoeppe commented Mar 29, 2021

jhpalmieri commented Mar 30, 2021

mkoeppe commented Mar 30, 2021

mkoeppe commented Mar 30, 2021

mkoeppe commented Mar 30, 2021

sagetrac-git mannequin commented Mar 30, 2021

sagetrac-git mannequin commented Mar 30, 2021

jhpalmieri commented Mar 31, 2021

jhpalmieri commented Mar 31, 2021

mkoeppe commented Mar 31, 2021

jhpalmieri commented Mar 31, 2021

mkoeppe commented Mar 31, 2021

jhpalmieri commented Mar 31, 2021

mkoeppe commented Mar 31, 2021

jhpalmieri commented Mar 31, 2021

vbraun commented May 27, 2021