Allow poetry-core 1.5.0

[icp1994]

Following up from #14931

To force building with poetry-core-1.5.0, I ran the following on a fresh synapse checkout with the pr patch

#!/bin/bash

set -euxo pipefail

poetry install --no-root --extras "saml2 oidc url-preview redis test"
poetry add build installer poetry-core==1.5.0
poetry run python -m build --no-isolation --wheel
poetry run python -m installer --dest=$(poetry env info --path) --prefix=/ dist/matrix_synapse-*.whl
poetry run python -m twisted.trial --jobs=$(nproc) tests
poetry remove build installer poetry-core

Test summary PASSED (skips=60, successes=2574) with the following skipped categories

[SKIPPED]
`BaseFederationServlet` does not support cancellation yet.

[SKIPPED]
Requires Postgres

[SKIPPED]
Requires jaeger_client

[SKIPPED]
Test only applies when postgres is used as the database

[SKIPPED]
Requires PyICU

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Pull request includes a sign off (requested via email)
• Code style is correct
  (run the linters)

[icp1994]

Regarding the auditwheel failures, the wheel built with poetry-core-1.5.0 doesn't contain synapse_rust.abi3.so resulting in the detection error by auditwheel. I built three separate wheels with versions poetry-core versions 1.3.2, 1.4.0 & 1.5.0 and the diffs of the unpacked wheels are as follows:

[icp@snow tmp]$ diff -qr synapse-poetry-core-132/ synapse-poetry-core-140/
Files synapse-poetry-core-132/matrix_synapse-1.76.0rc2.dist-info/METADATA and synapse-poetry-core-140/matrix_synapse-1.76.0rc2.dist-info/METADATA differ
Files synapse-poetry-core-132/matrix_synapse-1.76.0rc2.dist-info/RECORD and synapse-poetry-core-140/matrix_synapse-1.76.0rc2.dist-info/RECORD differ
Files synapse-poetry-core-132/matrix_synapse-1.76.0rc2.dist-info/WHEEL and synapse-poetry-core-140/matrix_synapse-1.76.0rc2.dist-info/WHEEL differ

[icp@snow tmp]$ diff -qr synapse-poetry-core-140/ synapse-poetry-core-150/
Files synapse-poetry-core-140/matrix_synapse-1.76.0rc2.dist-info/METADATA and synapse-poetry-core-150/matrix_synapse-1.76.0rc2.dist-info/METADATA differ
Files synapse-poetry-core-140/matrix_synapse-1.76.0rc2.dist-info/RECORD and synapse-poetry-core-150/matrix_synapse-1.76.0rc2.dist-info/RECORD differ
Files synapse-poetry-core-140/matrix_synapse-1.76.0rc2.dist-info/WHEEL and synapse-poetry-core-150/matrix_synapse-1.76.0rc2.dist-info/WHEEL differ
Only in synapse-poetry-core-140/synapse: synapse_rust.abi3.so

[icp@snow tmp]$ diff -qr synapse-poetry-core-150/ synapse-poetry-core-132/
Files synapse-poetry-core-150/matrix_synapse-1.76.0rc2.dist-info/METADATA and synapse-poetry-core-132/matrix_synapse-1.76.0rc2.dist-info/METADATA differ
Files synapse-poetry-core-150/matrix_synapse-1.76.0rc2.dist-info/RECORD and synapse-poetry-core-132/matrix_synapse-1.76.0rc2.dist-info/RECORD differ
Files synapse-poetry-core-150/matrix_synapse-1.76.0rc2.dist-info/WHEEL and synapse-poetry-core-132/matrix_synapse-1.76.0rc2.dist-info/WHEEL differ
Only in synapse-poetry-core-132/synapse: synapse_rust.abi3.so

So seems to me as either a regression or an intended change in the poetry-core-1.5.0.

[squahtx]

If you have the time for it, the regression needs investigating and fixing before we can merge this PR.

[nekopsykose]

this is because poetry-core 1.5 no longer allows the build= key. the new syntax is:

[tool.poetry.build]      
script = "build_rust.py" 

(instead of build = "build_rust.py"). the specific cause is python-poetry/poetry-core@4f7d44a

i don't know if the above change works with <1.5 too, but it works with 1.5, so distros can backport that [] change.

this also fixes #14966

[nekopsykose]

actually, even that doesn't work, so nevermind

[mgorny]

I see you hit it too while I've been trying to figure the other bug report out. I've filed python-poetry/poetry#7474 to check if it's intentional or a bug in poetry-core.

[DMRobertson]

the specific cause is python-poetry/poetry-core@4f7d44a

The underlying PR is python-poetry/poetry-core#318

actually, even that doesn't work, so nevermind

The PR description reads:

With this change, projects that require a setup.py to be
generated when a build script is used needs to explicitly set
tool.poetry.build.generate-setup-file, introduced in python-poetry/poetry-core#26, to true in pyproject.toml.

Which sounds like it's worth a shot.

[mgorny]

According to python-poetry/poetry#7470, this can be worked around by using:

[tool.poetry.build]
script = "build_rust.py"
generate-setup-file = true

That said, this is a short-time solution. Apparently poetry is going to remove setup.py support eventually, so this is going to stop working sooner or later. Poetry docs pretty explicitly say they don't support non-pure wheels, so hacking it around is not the way forward.

[nekopsykose]

I see you hit it too while I've been trying to figure the other bug report out.

: )

this can be worked around by using:

that is exactly what i was trying (after above; was incomplete) but i apparently typo'd something (classic). thanks, seems to work now instead..

[DMRobertson]

• Pull request includes a sign off (requested via email)

Received, many thanks.

[icp1994]

I have included the upstream workaround in pyproject.toml and built synapse wheels with poetry-core-1.0.0 (current lower bound), poetry-core-1.3.2 (current upper bound), and poetry-core-1.5.0 (proposed upper bound). All the built wheels contain identical files (except the dist-info of course) - all including synapse_rust.abi3.so and passes tests for me locally. I am force pushing this branch to check if CI is happy.

[DMRobertson]

That said, this is a short-time solution. Apparently poetry is going to remove setup.py support eventually, so this is going to stop working sooner or later.

Poetry docs pretty explicitly say they don't support non-pure wheels, so hacking it around is not the way forward.

@mgorny (or anyone else): do you have any links for these two points? I would like to better understand the situation here.

[mgorny]

This is the first step in removing setup file generation from Poetry projects.

Well, I might be misunderstanding but I'd assume that if there's the "first step", there will be next steps, and the eventual result will be the removal of it.

From: python-poetry/poetry-core#318

Note that, at the moment, only pure python wheels are supported.

From: https://python-poetry.org/docs/cli/#build

[DMRobertson]

Related: python-poetry/poetry#2740 (thanks @clokep)

[squahtx]

I have included the upstream workaround in pyproject.toml and built synapse wheels with poetry-core-1.0.0 (current lower bound), poetry-core-1.3.2 (current upper bound), and poetry-core-1.5.0 (proposed upper bound). All the built wheels contain identical files (except the dist-info of course) - all including synapse_rust.abi3.so and passes tests for me locally. I am force pushing this branch to check if CI is happy.

Sounds great. Let's get this merged.

[DMRobertson]

(Many thanks to all, and @icp1994 especially, for bearing with us and for putting in the effort to check that this change was safe to make.)

[DMRobertson]

Hmm. Develop CI is failing to build wheels on OSX:

https://github.com/matrix-org/synapse/actions/runs/4107246533/jobs/7086484359

  
  + pip install virtualenv -c /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/cibuildwheel/resources/constraints-python37.txt
  Requirement already satisfied: virtualenv in /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/build/venv/lib/python3.7/site-packages (20.16.3)
  Requirement already satisfied: filelock<4,>=3.4.1 in /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/build/venv/lib/python3.7/site-packages (from virtualenv) (3.8.0)
  Requirement already satisfied: distlib<1,>=0.3.5 in /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/build/venv/lib/python3.7/site-packages (from virtualenv) (0.3.5)
  Requirement already satisfied: platformdirs<3,>=2.4 in /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/build/venv/lib/python3.7/site-packages (from virtualenv) (2.5.2)
  Requirement already satisfied: importlib-metadata>=4.8.3 in /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/build/venv/lib/python3.7/site-packages (from virtualenv) (4.12.0)
  Requirement already satisfied: zipp>=0.5 in /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/build/venv/lib/python3.7/site-packages (from importlib-metadata>=4.8.3->virtualenv) (3.8.1)
  Requirement already satisfied: typing-extensions>=3.6.4 in /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/build/venv/lib/python3.7/site-packages (from importlib-metadata>=4.8.3->virtualenv) (4.3.0)
  + python -m virtualenv --no-download /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/venv-test
  created virtual environment CPython3.7.9.final.0-64 in 1032ms
    creator CPython3Posix(dest=/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/venv-test, clear=False, no_vcs_ignore=False, global=False)
    seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/runner/Library/Application Support/virtualenv)
      added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
    activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
  + which python
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/venv-test/bin/python
  + pip install /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/repaired_wheel/matrix_synapse-1.76.0-cp37-abi3-macosx_11_0_x86_64.whl
  ERROR: matrix_synapse-1.76.0-cp37-abi3-macosx_11_0_x86_64.whl is not a supported wheel on this platform.
  Error: Command ['pip', 'install', '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-b0_mzyp3/cp37-macosx_x86_64/repaired_wheel/matrix_synapse-1.76.0-cp37-abi3-macosx_11_0_x86_64.whl'] failed with code 1. None

@erikjohnston does this mean anything to you? Smells like it's related to https://github.com/matrix-org/synapse/blob/1e73effebf4f307f499c8cd4be58f805959beb38/.ci/scripts/auditwheel_wrapper.py

[icp1994]

Related I assume: pypa/cibuildwheel#1410
Is building *-cp37-* wheels necessary? README badge shows 3.8 as the minimum python version.

[clokep]

Related I assume: pypa/cibuildwheel#1410 Is building *-cp37-* wheels necessary? README badge shows 3.8 as the minimum python version.

Python 3.7 is the minimum supported version:

synapse/pyproject.toml

Line 150 in 5b55c32

python = "^3.7.1"

[icp1994]

Sorry this took so much time/effort post-merge to be fixed properly and thanks for handling that!