Support CPython 3.11, 3.12, and aarch64 processors #2331

ddelange · 2023-01-20T14:54:17Z

Hoi 👋

linux-aarch64 makes up for almost 10% of all platforms ref giampaolo/psutil#2103

aarch64 has already surpassed windows in terms of downloads for this package. Oracle, Amazon, Google, and Microsoft are all offering aarch64 cloud instances at an undeniable price point compared to amd/intel, so the demand will undoubtedly only grow

this PR is adapted from Add arm64 mac and linux wheels MagicStack/asyncpg#954
uses QEMU emulation for linux arm64 wheels: manylinux takes around 2.5hrs per wheel and alpine arm64 up to 4 hrs 😅
manylinux2014 wheels are built with GCC 10, which I think does not guarantee proper functioning of pybind11 (docs).
- so with this PR, linux wheels are built with GCC 12 (manylinux_2_28).
- pip will only install these wheels on linux operating systems with glibc >= 2.28 (mostly all 2020+ linux distributions like debian 10 buster, ubuntu 20.04 focal, almalinux/rhel 8, ...).

the wheels from this PR can be installed with:

# comma separated list for --find-links
export PIP_FIND_LINKS=https://github.com/ddelange/vaex/releases/expanded_assets/core-v4.17.1.post4
pip install --force-reinstall vaex

fixes #2366, fixes #2368, fixes #2397, closes #2427, fixes #2384

maartenbreddels · 2023-01-21T08:32:29Z

Hoi 👋

exciting, will take a look early next week!

manylinux takes around 2.5hrs per wheel and alpine arm64 up to 4 hrs

that worries me a bit.. :)

groeten,

Maarten

ddelange · 2023-01-21T09:40:18Z

here are all timings: https://github.com/ddelange/vaex/actions/runs/3965720337/usage

depending on how often a month you release vaex, this could eat into the 2k free minutes of GH...

as the parallelization is maximised and they're pushed to PyPI as soon as they're built, most of the wheels will be available soon upon release regardless

here are all the wheels: distributions.zip

ddelange · 2023-01-21T10:32:44Z

interestingly, that was 8260 minutes ^

apparently that's OK? then I don't understand their explanation 🤔 https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#included-storage-and-minutes

ddelange · 2023-01-21T10:40:57Z

ah there is a fair amount of duplication in that usage table for whatever reason 🤯

ddelange · 2023-01-22T10:54:32Z

a diff of current PyPI vs the zip above:

 vaex_core-4.16.1-cp310-cp310-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp310-cp310-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp310-cp310-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp310-cp310-win_amd64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_10_9_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_11_0_arm64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-win_amd64.whl
 vaex_core-4.16.1-cp36-cp36m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp36-cp36m-win_amd64.whl
 vaex_core-4.16.1-cp37-cp37m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp37-cp37m-win_amd64.whl
 vaex_core-4.16.1-cp38-cp38-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp38-cp38-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp38-cp38-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp38-cp38-win_amd64.whl
 vaex_core-4.16.1-cp39-cp39-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp39-cp39-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp39-cp39-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp39-cp39-win_amd64.whl

ddelange · 2023-01-23T07:30:45Z

packages/vaex-core/src/hash_object.cpp

-namespace std {
-    template<>
-    struct hash<PyObject*> {
-        size_t operator()(const PyObject *const &o) const {
-            return PyObject_Hash((PyObject*)o);
-        }
-    };
-}


This was my best guess...

pybind11 ref https://github.com/pybind/pybind11/blob/769fd3b889fef6cddb060f2a0be26aee62b4da05/include/pybind11/pytypes.h#L859

failure log https://github.com/ddelange/vaex/actions/runs/3965609112/jobs/6795506653#step:6:2110

@maartenbreddels any thoughts on this (incl me updating the pybind11 submodule)?

ddelange · 2023-01-23T07:31:30Z

packages/vaex-core/setup.py

@@ -183,12 +183,14 @@ def __str__(self):
      include_package_data=True,
      ext_modules=([extension_vaexfast] if on_rtd else [extension_vaexfast, extension_strings, extension_superutils, extension_superagg]) if not use_skbuild else [],
      zip_safe=False,
+      python_requires=">=3.6",


cibuildwheel parses this to determine which wheels to build

cc @franz101

see also the diff above

ddelange · 2023-02-27T07:19:30Z

I'm guessing this is blocked by #2339

maartenbreddels · 2023-03-06T13:11:53Z

Just letting you know i'm very busy and had a vacation.
Yes, I'll try to get #2339 green first!

ddelange · 2023-05-25T23:20:01Z

fwiw there are now third party free minutes on native arm64 machines, to get rid of the slow qemu builds

maartenbreddels · 2023-07-10T20:25:06Z

Could you try rebasing this?

ddelange · 2023-07-10T20:48:15Z

@maartenbreddels already merged in master 👍

ddelange · 2023-07-11T06:55:46Z

    ERROR: Could not find a version that satisfies the requirement vaex-core<4.17,>=4.17.0 (from vaex)
    ERROR: No matching distribution found for vaex-core<4.17,>=4.17.0

maartenbreddels · 2023-07-11T15:50:10Z

Yeah, a bug/artifact or our release script. Should be good now.

ddelange · 2023-08-03T20:50:04Z

hoi @maartenbreddels 👋

I pulled master and fixed merge conflicts, but it looks like CI is still not very happy. Seeing errors like hdf file missing on disk, and TypeError: train() got an unexpected keyword argument 'early_stopping_rounds'.

Do you think it might be related to this PR?

franz101 · 2023-08-17T20:51:39Z

Just wondering here on the Python packaging. Python 3.6 and 3.7 are now deprecated on the other hand we can bump to 3.10 and 3.11?

to-bee · 2023-08-28T07:48:57Z

Do we have any updates on this MR?

ddelange · 2023-09-01T08:07:30Z

HI @maartenbreddels 👋

Was your s3 account deleted by any chance?

vaex.open('s3://vaex/taxi/yellow_taxi_2009_2015_f32.hdf5?anon=true')

raises

FileNotFoundError: [Errno 2] Path does not exist 'vaex/taxi/yellow_taxi_2009_2015_f32.hdf5'. Detail: [errno 2] No such file or directory

ddelange · 2024-08-30T16:50:36Z

looks like not allowed. have a feeling it might be windows 3.8 threading implementation memory leak

ddelange · 2024-08-30T16:57:27Z

you could simply skip publish of that vaex-core wheel, and don't upload tar.gz for vaex-core. it's the tensorflow approach

done in the commits below

.github/workflows/wheel.yml

ddelange · 2024-08-30T19:08:00Z

.github/workflows/wheel-universal.yml

+    # https://github.com/pypa/gh-action-pypi-publish#trusted-publishing
+    - name: Publish package distributions to PyPI
+      uses: pypa/[email protected]
+      if: startsWith(github.ref, 'refs/tags')
+      with:
+        skip-existing: true


all you need to do for this to work is add vaexio/vaex as trusted publisher here: https://pypi.org/manage/project/vaex/settings/publishing (and to the other vaex pypi projects)

This now fails, because it will try to release all packages (even though I only pushed the vaex-core tag).

I guess if we only do the cp commands when the specific tag is used, we can make this work. What do you think?

ddelange · 2024-08-31T15:02:31Z

that did it. we're green 🏁

fwiw, pandas deprecated python 3.8 in August 2023 (v2.1.0)

maartenbreddels · 2024-09-05T14:53:33Z

Whow, amazing. You did most of the work @ddelange thanks a lot for the work and support !

I'll set up the trusted publisher (probably next week) and see if we can make a release 🥳

EwoutH · 2024-09-11T11:03:25Z

Wow it got merged! Congratulations!!

erwanp · 2024-09-13T09:17:36Z

Well done @ddelange @maartenbreddels @EwoutH ; that's a great news for our codes!

HajimeKawahara · 2024-09-13T09:30:45Z

Great news! Thanks a lot. @ddelange @maartenbreddels @EwoutH

franz101 · 2024-09-18T01:36:35Z

@Ben-Epstein wake up it's christmas

ddelange · 2024-09-18T06:05:13Z

fyi followup PR #2434 should land soon, so release is imminent:)

ddelange · 2024-09-27T15:25:00Z

packages/vaex-hdf5/setup.py

@@ -1,19 +1,19 @@
 import os
-import imp
+from importlib.machinery import SourceFileLoader


@maartenbreddels imp was removed and will fail to import on cp312. so this PR touches all the universal packages to achieve cp312 support.

it might make sense to bump the minimum required versions accordingly in vaex: so cp312 users don't end up with older (broken) versions of the universal packages in their env

I'm not sure if I should take action here. Can you elaborate?

here it is: #2438

EwoutH · 2024-10-01T13:54:12Z

Now that this upgrade process is still fresh in everybody’s mind, would it make sense to directly push to Python 3.13?

maartenbreddels · 2024-10-01T13:57:36Z

I doubt the dependencies (e.g. arrow) are already shipping, but I think we should indeed do this soon!

erwanp · 2024-10-25T05:58:14Z

Hello Vaex team Python 3.12 compatibility is merged (congrats) but it does not seem to be released on Conda, can you confirm?
(@minouHub for information)

ddelange · 2024-10-25T08:27:37Z

Hi @erwanp 👋

you can fetch vaex-core 4.18.1 from PyPI
it was yanked because windows support is still an open issue [BUG-REPORT] vaex causes a segmentation fault on windows #2442 but it's fine to use on linux and mac
the other vaex packages (vaex, vaex-ml etc) still need to be released to PyPI with the right version bumps introduced in this PR for Python 3.12 support, I think @maartenbreddels will want to wait with that until the windows bug is fixed and release all packages together.
I don't know if conda release is planned or automated -- the wheels are self contained so you can just fetch them from PyPI

ddelange mentioned this pull request Jan 21, 2023

Maximize wheel build parallellization ddelange/vaex#1

Closed

ddelange commented Jan 23, 2023

View reviewed changes

ddelange force-pushed the build-matrix branch from beff33a to 7bf26cc Compare February 27, 2023 07:15

This was referenced Jun 30, 2023

[FEATURE-REQUEST] Support Python 3.11.2 #2368

Closed

[BUG-REPORT] Failed to build vaex-core (pybind11 issue?) #2366

Closed

ddelange changed the title ~~Build aarch64 wheels~~ Build aarch64 wheels and support python 3.11 Jul 10, 2023

maartenbreddels force-pushed the master branch from 8d44942 to d4bf017 Compare July 11, 2023 13:55

CarrieFilion mentioned this pull request Aug 2, 2023

Install with Python 3.11 fails athob/py-ananke#1

Closed

ddelange referenced this pull request in rapidfuzz/RapidFuzz Aug 10, 2023

upgrade cython

dfea552

ddelange mentioned this pull request Aug 30, 2023

Discussion: support bundling libmagic ahupp/python-magic#233

Closed

ddelange force-pushed the build-matrix branch 3 times, most recently from 5680eb9 to 2136629 Compare September 4, 2023 08:28

ddelange commented Aug 30, 2024

View reviewed changes

.github/workflows/wheel.yml Outdated Show resolved Hide resolved

ddelange added 2 commits August 30, 2024 20:33

Add back tags event trigger for core wheel upload

bac3efb

Switch to trusted publishing

93ce2c0

ddelange commented Aug 30, 2024

View reviewed changes

ddelange added 5 commits August 30, 2024 21:10

Add trusted publishing for vaex-core

f6c8b79

Exclude windows 3.8 from pythonpackage.yml

7988bb4

Don't upload vaex-core source distribution

b23f5a7

Disable cp38-win vaex-core wheel build

b594d77

Add permission

faaf23e

This was referenced Sep 9, 2024

[BUG-REPORT] PydanticImportError: BaseSettings has been moved #2384

Closed

vaex does not work with numpy==2.0.0 #2425

Open

maartenbreddels merged commit e3e1842 into vaexio:master Sep 11, 2024
51 checks passed

ddelange mentioned this pull request Sep 11, 2024

Fix pythonpackage.yml failing on macos-13 #2432

Merged

erwanp mentioned this pull request Sep 13, 2024

support for python 3.11 HajimeKawahara/exojax#502

Closed

minouHub mentioned this pull request Sep 18, 2024

Vaex compatible up to python<=3.12 radis/radis#698

Draft

4 tasks

ddelange commented Sep 27, 2024

View reviewed changes

ddelange mentioned this pull request Sep 30, 2024

Update vaex version ranges #2438

Closed

Support CPython 3.11, 3.12, and aarch64 processors #2331

Support CPython 3.11, 3.12, and aarch64 processors #2331

Conversation

ddelange commented Jan 20, 2023 • edited Loading

the wheels from this PR can be installed with:

maartenbreddels commented Jan 21, 2023

ddelange commented Jan 21, 2023 • edited Loading

ddelange commented Jan 21, 2023

ddelange commented Jan 21, 2023 • edited Loading

ddelange commented Jan 22, 2023

ddelange Jan 23, 2023 • edited Loading

Choose a reason for hiding this comment

ddelange Feb 27, 2023

Choose a reason for hiding this comment

ddelange Jan 23, 2023

Choose a reason for hiding this comment

ddelange Aug 17, 2023

Choose a reason for hiding this comment

ddelange commented Feb 27, 2023

maartenbreddels commented Mar 6, 2023

ddelange commented May 25, 2023

maartenbreddels commented Jul 10, 2023

ddelange commented Jul 10, 2023

ddelange commented Jul 11, 2023

maartenbreddels commented Jul 11, 2023

ddelange commented Aug 3, 2023

franz101 commented Aug 17, 2023

to-bee commented Aug 28, 2023

ddelange commented Sep 1, 2023 • edited Loading

ddelange commented Aug 30, 2024

ddelange commented Aug 30, 2024 • edited Loading

ddelange Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

maartenbreddels Sep 27, 2024

Choose a reason for hiding this comment

ddelange commented Aug 31, 2024

maartenbreddels commented Sep 5, 2024

EwoutH commented Sep 11, 2024

erwanp commented Sep 13, 2024

HajimeKawahara commented Sep 13, 2024

franz101 commented Sep 18, 2024

ddelange commented Sep 18, 2024

ddelange Sep 27, 2024 • edited Loading

Choose a reason for hiding this comment

maartenbreddels Oct 1, 2024

Choose a reason for hiding this comment

ddelange Oct 1, 2024

Choose a reason for hiding this comment

EwoutH commented Oct 1, 2024

maartenbreddels commented Oct 1, 2024

erwanp commented Oct 25, 2024

ddelange commented Oct 25, 2024

ddelange commented Jan 20, 2023 •

edited

Loading

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange Jan 23, 2023 •

edited

Loading

ddelange commented Sep 1, 2023 •

edited

Loading

ddelange commented Aug 30, 2024 •

edited

Loading

ddelange Aug 30, 2024 •

edited

Loading

ddelange Sep 27, 2024 •

edited

Loading