pyzmq: Fix build errors; zeromq, pyzmq: Update; setuptools: upgrade to 59.2.0 #32828

mkoeppe · 2021-11-05T17:09:37Z

pyzmq fails to build on some platforms. We fix it by

upgrading setuptools to a version that includes CCompiler.has_function: Do not fail if self.outputdir is set pypa/distutils#65
correcting the pyzmq build configuration

We also upgrade pyzmq and zeromq to the latest versions.

For example on opensuse-tumbleweed-minimal (https://github.com/mkoeppe/sage/runs/4085305194?check_suite_focus=true)

Setting up build directory for pyzmq-22.2.1
Finished extraction
No patch files found in ../patches
****************************************************
Host system:
Linux dd2167ea4425 5.11.0-1020-azure #21~20.04.1-Ubuntu SMP Mon Oct 11 18:54:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib64/gcc/x86_64-suse-linux/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-suse-linux
Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,ada,go,d,jit --enable-offload-targets=nvptx-none,amdgcn-amdhsa, --without-cuda-driver --enable-host-shared --enable-checking=release --disable-werror --with-gxx-include-dir=/usr/include/c++/11 --enable-ssp --disable-libssp --disable-libvtv --enable-cet=auto --disable-libcc1 --enable-plugin --with-bugurl=https://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --with-slibdir=/lib64 --with-system-zlib --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-libphobos --enable-version-specific-runtime-libs --with-gcc-major-version-only --enable-linker-build-id --enable-linux-futex --enable-gnu-indirect-function --program-suffix=-11 --without-system-libunwind --enable-multilib --with-arch-32=x86-64 --with-tune=generic --with-build-config=bootstrap-lto-lean --enable-link-mutex --build=x86_64-suse-linux --host=x86_64-suse-linux
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.2.1 20210816 [revision 056e324ce46a7924b5cf10f61010cf9dd2ca10e9] (SUSE Linux) 
****************************************************
Package 'pyzmq' is currently not installed
No legacy uninstaller found for 'pyzmq'; nothing to do
Installing pyzmq-22.2.1
Processing /sage/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/pyzmq-22.2.1/src
    Preparing wheel metadata: started
    Running command /sage/local/var/lib/sage/venv-python3.9.7/bin/python3 /sage/local/var/lib/sage/venv-python3.9.7/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpn08jd7qu
    INFO: Disabling color, you really want to install colorlog.
    Disabling color, you really want to install colorlog.
    running dist_info
    creating /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info
    writing /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/requires.txt
    writing top-level names to /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/top_level.txt
    writing manifest file '/tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/SOURCES.txt'
    running configure
    Settings obtained from pkg-config: {'library_dirs': [], 'include_dirs': [], 'libraries': ['zmq']}
    {'libraries': ['zmq'], 'include_dirs': [], 'library_dirs': [], 'runtime_library_dirs': [], 'extra_link_args': []}
    C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -fPIC

    compile options: '-c'
    gcc: build/temp.linux-x86_64-3.9/scratch/check_sys_un.c
    gcc build/temp.linux-x86_64-3.9/scratch/check_sys_un.o -o build/temp.linux-x86_64-3.9/scratch/check_sys_un
    Configure: Autodetecting ZMQ settings...
        Custom ZMQ dir:
    Checking for timer_create
    ** Errors about missing timer_create are a normal part of this process **
    ************************************************
    C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -fPIC

    creating build/temp.linux-x86_64-3.9/scratch/tmp
    compile options: '-c'
    gcc: /tmp/timer_create7kgvq7_0.c
    /tmp/timer_create7kgvq7_0.c: In function 'main':
    /tmp/timer_create7kgvq7_0.c:2:5: warning: implicit declaration of function 'timer_create' [-Wimplicit-function-declaration]
        2 |     timer_create();
          |     ^~~~~~~~~~~~
    gcc build/temp.linux-x86_64-3.9/scratch/tmp/timer_create7kgvq7_0.o -o build/temp.linux-x86_64-3.9/scratch/a.out

    error: [Errno 2] No such file or directory: 'a.out'
    Failed with default libzmq, trying again with /usr/local
    {'libraries': ['zmq'], 'include_dirs': ['/usr/local/include'], 'library_dirs': ['/usr/local/lib'], 'runtime_library_dirs': ['/usr/local/lib'], 'extra_link_args': []}
    Configure: Autodetecting ZMQ settings...
        Custom ZMQ dir:       /usr/local
    Checking for timer_create
    ** Errors about missing timer_create are a normal part of this process **
    ************************************************
    C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -fPIC

    compile options: '-c'
    gcc: /tmp/timer_createl5aoyxgr.c
    /tmp/timer_createl5aoyxgr.c: In function 'main':
    /tmp/timer_createl5aoyxgr.c:2:5: warning: implicit declaration of function 'timer_create' [-Wimplicit-function-declaration]
        2 |     timer_create();
          |     ^~~~~~~~~~~~
    Assembler messages:
    Fatal error: can't create build/temp.linux-x86_64-3.9/scratch/tmp/timer_createl5aoyxgr.o: No such file or directory
    ** The above error about timer_create is normal and not a problem! **
    no timer_create, linking librt
    C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -fPIC

    compile options: '-I/usr/local/include -Izmq/utils -c'
    gcc: build/temp.linux-x86_64-3.9/scratch/vers.c
    gcc build/temp.linux-x86_64-3.9/scratch/vers.o -L/usr/local/lib -Wl,--enable-new-dtags,-R/usr/local/lib -lzmq -lrt -o build/temp.linux-x86_64-3.9/scratch/vers
    Error running version detection script:

    build/temp.linux-x86_64-3.9/scratch/vers: error while loading shared libraries: libzmq.so.5: cannot open shared object file: No such file or directory


    error: Error running version detection script:

    build/temp.linux-x86_64-3.9/scratch/vers: error while loading shared libraries: libzmq.so.5: cannot open shared object file: No such file or directory


    Warning: Couldn't find an acceptable libzmq on the system.

    If you expected pyzmq to link against an installed libzmq, please check to make sure:

        * You have a C compiler installed
        * A development version of Python is installed (including headers)
        * A development version of ZMQ >= 3.2 is installed (including headers)
        * If ZMQ is not in a default location, supply the argument --zmq=<path>
        * If you did recently install ZMQ to a default location,
          try rebuilding the ld cache with `sudo ldconfig`
          or specify zmq's location with `--zmq=/usr/local`

    You can skip all this detection/waiting nonsense if you know
    you want pyzmq to bundle libzmq as an extension by passing:

        `--zmq=bundled`

    I will now try to build libzmq as a Python extension
    unless you interrupt me (^C) in the next 10 seconds...

    ************************************************

    10...
     9...
     8...
     7...
     6...
     5...
     4...
     3...
     2...
     1...
    Using bundled libzmq
    already have bundled/zeromq
    attempting ./configure to generate platform.hpp
    Warning: failed to configure libzmq:
    [Errno 2] No such file or directory: './configure'
    staging platform.hpp from: /sage/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/pyzmq-22.2.1/src/buildutils/include_linux
    checking for timer_create

Also reported in #31855 comment:22

Upstream: Fixed upstream, in a later stable release.

CC: @enriqueartal @jhpalmieri @dimpase

Component: packages: standard

Author: Matthias Koeppe

Branch: b430bcf

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/32828

The text was updated successfully, but these errors were encountered:

dimpase · 2021-11-05T18:35:37Z

comment:3

    Configure: Autodetecting ZMQ settings...
        Custom ZMQ dir:       /usr/local

weird.

dimpase · 2021-11-05T18:51:48Z

comment:4

I'd have tried updating zeromq to the latest version, 4.3.4.

enriqueartal · 2021-11-05T19:01:05Z

comment:5

Replying to @dimpase:

I'd have tried updating zeromq to the latest version, 4.3.4.

It didn't work for me. I tried also to use pyzmq-22.3.0; it fails to download https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.zip and stops (downloading directly works)

Compiling pyzmq outside Sage gives no error; maybe some local variable? Is is possible to use system package?

mkoeppe · 2021-11-05T19:40:27Z

comment:6

I think the attempted configuration with setup.cfg does not work at all. This was masked so far by SAGE_VENV and SAGE_LOCAL being the same, which has changed.

This is another case where the locations discovered by the configure tests in spkg-configure.m4 are not passed on

dimpase · 2021-11-05T20:02:06Z

comment:7

Sorry, but the error happens in the minimal build, spkg-configure.m4 is not involved, right?

mkoeppe · 2021-11-05T21:01:49Z

comment:8

You are right, it isn't; I should have written "this is also another case..."

enriqueartal · 2021-11-05T21:15:34Z

comment:9

Is there a temptative solution to test?

Replying to @mkoeppe:

I think the attempted configuration with setup.cfg does not work at all. This was masked so far by SAGE_VENV and SAGE_LOCAL being the same, which has changed.

This is another case where the locations discovered by the configure tests in spkg-configure.m4 are not passed on

enriqueartal · 2021-11-06T09:02:19Z

comment:10

Dirty trick to test the other changes in Fedora 35:

sage -pip install pyzmq==22.0.3

and comment out the install line in spkg-install.in. In other installation I did the same thing with the last version of pyzmq.

I am running tests now.

enriqueartal · 2021-11-06T22:19:15Z

comment:11

I finished two compilations in Fedora 35, in both cases using a package of cysignals from https://github.com/enriqueartal/cysignals/commit/b404fbef69ddd5cfd0085ab15885e9f0315f9471 which is actually https://github.com/kliem/cysignals/tree/fix_non_constant_MINSIGSTKSZ up to a right parenthesis and installing pymzq as in comment:10.

One compilation is for 9.4. The test gave the following failures:

sage -t --long --warn-long 212.7 --random-seed=0 src/sage/schemes/elliptic_curves/ell_rational_field.py  # 9 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/doctest/test.py  # 7 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/symbolic/integration/integral.py  # 1 doctest failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # Timed out
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/tests/cmdline.py  # 2 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/modular/abvar/abvar.py  # 1 doctest failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/rings/polynomial/multi_polynomial_ideal.py  # 2 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/modular/hecke/submodule.py  # 1 doctest failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/lfunctions/sympow.py  # 13 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/schemes/elliptic_curves/lseries_ell.py  # 3 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/libs/giac/__init__.py  # 13 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/lfunctions/lcalc.py  # 2 doctests failed

Some of them seem to be harmless (right permissions), other ones may reflect some problems between giac and cysignals.

The second one is for the branch #30766 merged with develop, with the above cysignals and pyzmq and also gmpy2-2.1.0rc1. Besides the errors below, I did not find jupyter in $SAGE_ROOT/local/share but in $SAGE_ROOT/local/var/lib/sage/venv-python3.10/share:

sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/schemes/elliptic_curves/ell_rational_field.py  # 9 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # Timed out
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/schemes/elliptic_curves/isogeny_small_degree.py  # Timed out
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/rings/function_field/function_field.py  # Timed out
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/doctest/test.py  # 25 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/tests/cmdline.py  # 4 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/misc/sageinspect.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/schemes/riemann_surfaces/riemann_surface.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/modular/abvar/abvar.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/rings/polynomial/multi_polynomial_ideal.py  # 2 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/modular/hecke/submodule.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/lfunctions/sympow.py  # 13 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/schemes/elliptic_curves/lseries_ell.py  # 3 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/plot/colors.py  # 5 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage_docbuild/utils.py  # 2 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/libs/giac/__init__.py  # 13 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/structure/unique_representation.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/lfunctions/lcalc.py  # 2 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/modular/modform/numerical.py  # 3 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/symbolic/callable.py  # 5 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/cpython/atexit.pyx  # 6 doctests failed

mkoeppe · 2021-11-06T22:22:05Z

comment:12

I think the above would be better to discuss in #30767.
Let's please keep this ticket focused on the pyzmq build failures.

mkoeppe · 2021-11-06T22:22:50Z

comment:13

It looks like pyzmq is incompatible with our use of SETUPTOOLS_USE_DISTUTILS=local.

enriqueartal · 2021-11-06T22:56:55Z

comment:14

Replying to @mkoeppe:

I think the above would be better to discuss in #30767.
Let's please keep this ticket focused on the pyzmq build failures.

OK, sorry for the noise (#32576 is also involved)

mkoeppe · 2021-11-06T23:02:07Z

comment:15

Replying to @mkoeppe:

It looks like pyzmq is incompatible with our use of SETUPTOOLS_USE_DISTUTILS=local.

The recent setuptools versions vendor a version of distutils that has a bug described in pypa/distutils#64

mkoeppe · 2021-11-07T01:57:54Z

Branch: u/mkoeppe/pyzmq__build_errors

sagetrac-git · 2021-11-07T02:10:40Z

Commit: 89e1dea

sagetrac-git · 2021-11-07T02:10:40Z

Branch pushed to git repo; I updated commit sha1. New commits:

`fb48e08`	`build/pkgs/pyzmq/spkg-install.in: Do not overwrite library_dirs, include_dirs here, instead use zmq_prefix`
`89e1dea`	`build/pkgs/setuptools: Instead of bumping the patch level, upgrade to 58.5.3`

mkoeppe · 2021-11-07T02:11:46Z

Author: Matthias Koeppe

sagetrac-git · 2021-11-07T02:15:25Z

Branch pushed to git repo; I updated commit sha1. New commits:

`129d0fb`	`build/pkgs/pyzmq/spkg-install.in: Fix up`

sagetrac-git · 2021-11-07T02:15:25Z

Changed commit from 89e1dea to 129d0fb

sagetrac-git · 2021-11-07T02:19:22Z

Changed commit from 129d0fb to fc11612

sagetrac-git · 2021-11-07T02:19:22Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2240cc1`	`build/pkgs/pyzmq: Update to 22.3.0`
`fc11612`	`build/pkgs/zeromq: Update to 4.3.4, add upstream_url`

sagetrac-git · 2021-11-07T02:26:15Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a8fe9da`	`build/pkgs/zeromq/patches/getrandom.patch: Remove`

sagetrac-git · 2021-11-07T02:26:15Z

Changed commit from fc11612 to a8fe9da

mkoeppe · 2021-11-09T23:57:21Z

Upstream: Fixed upstream, but not in a stable release.

mkoeppe · 2021-11-09T23:57:21Z

comment:32

pypa/distutils#65 has been merged

mkoeppe · 2021-11-13T18:11:16Z

comment:33

... and is included in https://setuptools.pypa.io/en/latest/history.html#v59-0-0 (not yet on PyPI)

mkoeppe · 2021-11-15T00:26:37Z

comment:34

setuptools 59.0.1 is out, but it looks like we should wait for 59.1.0 so that we can avoid the setup_requires deprecation done in 58.3.0 and reverted in 59.1.0

sagetrac-git · 2021-11-15T02:19:18Z

Branch pushed to git repo; I updated commit sha1. New commits:

`078e37b`	`build/pkgs/setuptools: Update to 59.1.0, remove patch`

sagetrac-git · 2021-11-15T02:19:18Z

Changed commit from a8fe9da to 078e37b

mkoeppe · 2021-11-15T02:20:09Z

Changed upstream from Fixed upstream, but not in a stable release. to Fixed upstream, in a later stable release.

mkoeppe · 2021-11-16T20:23:12Z

comment:38

Let's get this in please

sagetrac-git · 2021-11-19T05:25:09Z

Changed commit from 078e37b to b430bcf

sagetrac-git · 2021-11-19T05:25:09Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ee4f6b4`	`Merge tag '9.5.beta7' into t/32828/pyzmq__build_errors`
`b430bcf`	`build/pkgs/setuptools: Update to 59.2.0`

dimpase · 2021-11-19T11:00:15Z

Reviewer: Dima Pasechnik

dimpase · 2021-11-19T11:00:15Z

comment:42

lgtm

mkoeppe · 2021-11-19T17:05:52Z

comment:43

Thanks!

tornaria · 2021-11-22T18:24:24Z

comment:44

Works for me on void linux using musl libc which used to fail (I was carrying a different patch before; I've replaced that patch by the changes in this ticket and it works).

mkoeppe · 2021-11-22T18:35:15Z

comment:45

Thanks for testing!

vbraun · 2021-12-05T11:15:41Z

Changed branch from u/mkoeppe/pyzmq__build_errors to b430bcf

mkoeppe · 2021-12-21T19:09:42Z

Changed commit from b430bcf to none

mkoeppe · 2021-12-21T19:09:42Z

comment:47

Follow-ups in #32944, #33059

mkoeppe added this to the sage-9.5 milestone Nov 5, 2021

mkoeppe added c: packages: standard labels Nov 5, 2021

mkoeppe changed the title ~~pyzmq: Build errors~~ pyzmq: Fix build errors, upgrade and patch setuptools Nov 7, 2021

mkoeppe added the s: needs review label Nov 7, 2021

mkoeppe changed the title ~~pyzmq: Fix build errors, upgrade and patch setuptools~~ pyzmq: Fix build errors; zeromq, pyzmq: Update; setuptools: upgrade to 58.5.3, add patch Nov 7, 2021

mkoeppe changed the title ~~pyzmq: Fix build errors; zeromq, pyzmq: Update; setuptools: upgrade to 58.5.3, add patch~~ pyzmq: Fix build errors; zeromq, pyzmq: Update; setuptools: upgrade to 59.1.0 Nov 15, 2021

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~pyzmq: Fix build errors; zeromq, pyzmq: Update; setuptools: upgrade to 59.1.0~~ pyzmq: Fix build errors; zeromq, pyzmq: Update; setuptools: upgrade to 59.2.0 Nov 19, 2021

dimpase added s: positive review and removed s: needs review labels Nov 19, 2021

vbraun removed the s: positive review label Dec 5, 2021

vbraun closed this as completed in e12c3ec Dec 5, 2021

This was referenced Jan 7, 2023

Meta-ticket: Eliminate use of patches for Python packages #31543

Open

Update pip to 21.3.1, distlib to 0.3.3 #32671

Closed

pyzmq: Fix build error on cygwin #32944

Closed

fedora-{26,27,28,...}-minimal: pyzmq build failure #33145

Closed

pyzmq: Fix build errors; zeromq, pyzmq: Update; setuptools: upgrade to 59.2.0 #32828

pyzmq: Fix build errors; zeromq, pyzmq: Update; setuptools: upgrade to 59.2.0 #32828

Comments

mkoeppe commented Nov 5, 2021

dimpase commented Nov 5, 2021

dimpase commented Nov 5, 2021

enriqueartal commented Nov 5, 2021

mkoeppe commented Nov 5, 2021

dimpase commented Nov 5, 2021

mkoeppe commented Nov 5, 2021

enriqueartal commented Nov 5, 2021

enriqueartal commented Nov 6, 2021

enriqueartal commented Nov 6, 2021

mkoeppe commented Nov 6, 2021

mkoeppe commented Nov 6, 2021

enriqueartal commented Nov 6, 2021

mkoeppe commented Nov 6, 2021

mkoeppe commented Nov 7, 2021

sagetrac-git mannequin commented Nov 7, 2021

sagetrac-git mannequin commented Nov 7, 2021

mkoeppe commented Nov 7, 2021

sagetrac-git mannequin commented Nov 7, 2021

sagetrac-git mannequin commented Nov 7, 2021

sagetrac-git mannequin commented Nov 7, 2021

sagetrac-git mannequin commented Nov 7, 2021

sagetrac-git mannequin commented Nov 7, 2021

sagetrac-git mannequin commented Nov 7, 2021

mkoeppe commented Nov 9, 2021

mkoeppe commented Nov 9, 2021

mkoeppe commented Nov 13, 2021

mkoeppe commented Nov 15, 2021

sagetrac-git mannequin commented Nov 15, 2021

sagetrac-git mannequin commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

This comment has been minimized.

mkoeppe commented Nov 16, 2021

This comment has been minimized.

sagetrac-git mannequin commented Nov 19, 2021

sagetrac-git mannequin commented Nov 19, 2021

dimpase commented Nov 19, 2021

dimpase commented Nov 19, 2021

mkoeppe commented Nov 19, 2021

tornaria commented Nov 22, 2021

mkoeppe commented Nov 22, 2021

vbraun commented Dec 5, 2021

mkoeppe commented Dec 21, 2021

mkoeppe commented Dec 21, 2021