Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEST: arrow v18.0.0rc0 #1432

Draft
wants to merge 34 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
e0c1cad
TST: bring in pyarrow builds
h-vetinari Jul 28, 2024
371682a
re-introduce subpackage dependencies; test for only one python on aar…
h-vetinari Jul 9, 2024
a3f7e7a
arrow v18.0.0rc0
h-vetinari Oct 16, 2024
742d7cb
match migrations from pyarrow feedstock
h-vetinari Oct 16, 2024
8e38b35
add & set PARQUET_TEST_DATA
h-vetinari Oct 17, 2024
5614f2e
clean up patches & skips where possible
h-vetinari Oct 17, 2024
638898e
use a swapfile to avoid getting OOM-killed
h-vetinari Oct 17, 2024
5214c0a
remove deps for obsolete python versions
h-vetinari Oct 17, 2024
d2f5813
add python 3.13 migration
h-vetinari Oct 17, 2024
ad5c55d
skip test deps that are not available yet for 3.13
h-vetinari Oct 17, 2024
81128f5
remove track_features for cuda
h-vetinari Oct 17, 2024
70e7b46
use default CMAKE_ARGS; absorb EXTRA_CMAKE_ARGS
h-vetinari Oct 17, 2024
cffd409
backport patch for tzdata handling
h-vetinari Oct 22, 2024
eb35adb
backport patch for ARROW_GRPC_CPP_PLUGIN
h-vetinari Oct 23, 2024
e0c6d2d
MNT: Re-rendered with conda-build 24.9.0, conda-smithy 3.43.0, and co…
h-vetinari Oct 23, 2024
5b24463
DEBUG: remove package validation
h-vetinari Oct 16, 2024
c8a675f
remove sed for grpc_cpp_plugin
h-vetinari Oct 23, 2024
d7ea4ab
undo infra changes compared to last passing windows build
h-vetinari Oct 23, 2024
4e89f1e
use micromamba again
jaimergp Oct 28, 2024
d8ae9a6
Merge branch 'main' of github.com:conda-forge/arrow-cpp-feedstock int…
jaimergp Oct 28, 2024
2bf0d0e
Use micromamba 2.0.2-2
jaimergp Oct 28, 2024
c363dd6
skip unix for now
jaimergp Oct 28, 2024
4667b6b
set GIT_SSL_NO_VERIFY=true
jaimergp Oct 28, 2024
4217105
use C:
jaimergp Oct 28, 2024
9555ad3
set GIT_SSL_NO_VERIFY again
jaimergp Oct 28, 2024
f0f4310
Merge remote-tracking branch 'upstream/main' into e2e
h-vetinari Oct 28, 2024
13dc18c
DEBUG: disable output validation
h-vetinari Oct 28, 2024
8b71767
MNT: Re-rendered with conda-build 24.9.0, conda-smithy 3.43.0, and co…
h-vetinari Oct 28, 2024
71fe125
set GIT_SSL_NO_VERIFY=true
jaimergp Oct 28, 2024
c965271
fix windows variables
h-vetinari Oct 28, 2024
2dc025c
Use micromamba 2.0.2-2
jaimergp Oct 28, 2024
4408ffa
Do not move cache
jaimergp Oct 29, 2024
65d0df6
set CONDA_PKGS_DIRS=%MINIFORGE_HOME%\pkgs instead
jaimergp Oct 29, 2024
d007a0e
Rerender with https://github.com/conda-forge/conda-smithy/pull/2107
jaimergp Oct 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions .azure-pipelines/azure-pipelines-linux.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 1 addition & 3 deletions .azure-pipelines/azure-pipelines-osx.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions .azure-pipelines/azure-pipelines-win.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,26 @@ libprotobuf:
- 5.27.5
lz4_c:
- 1.9.3
numpy:
- '2.0'
- '2.0'
- '2.0'
- '2'
- '2.0'
openssl:
- '3'
orc:
- 2.0.2
pin_run_as_build:
python:
min_pin: x.x
max_pin: x.x
python:
- 3.10.* *_cpython
- 3.11.* *_cpython
- 3.12.* *_cpython
- 3.13.* *_cp313
- 3.9.* *_cpython
re2:
- 2024.07.02
snappy:
Expand All @@ -76,6 +92,8 @@ zip_keys:
- cuda_compiler
- cuda_compiler_version
- docker_image
- - python
- numpy
zlib:
- '1'
zstd:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,26 @@ libprotobuf:
- 5.27.5
lz4_c:
- 1.9.3
numpy:
- '2.0'
- '2.0'
- '2.0'
- '2'
- '2.0'
openssl:
- '3'
orc:
- 2.0.2
pin_run_as_build:
python:
min_pin: x.x
max_pin: x.x
python:
- 3.10.* *_cpython
- 3.11.* *_cpython
- 3.12.* *_cpython
- 3.13.* *_cp313
- 3.9.* *_cpython
re2:
- 2024.07.02
snappy:
Expand All @@ -76,6 +92,8 @@ zip_keys:
- cuda_compiler
- cuda_compiler_version
- docker_image
- - python
- numpy
zlib:
- '1'
zstd:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,26 @@ libprotobuf:
- 5.27.5
lz4_c:
- 1.9.3
numpy:
- '2.0'
- '2.0'
- '2.0'
- '2'
- '2.0'
openssl:
- '3'
orc:
- 2.0.2
pin_run_as_build:
python:
min_pin: x.x
max_pin: x.x
python:
- 3.10.* *_cpython
- 3.11.* *_cpython
- 3.12.* *_cpython
- 3.13.* *_cp313
- 3.9.* *_cpython
re2:
- 2024.07.02
snappy:
Expand All @@ -80,6 +96,8 @@ zip_keys:
- cuda_compiler
- cuda_compiler_version
- docker_image
- - python
- numpy
zlib:
- '1'
zstd:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,26 @@ libprotobuf:
- 5.27.5
lz4_c:
- 1.9.3
numpy:
- '2.0'
- '2.0'
- '2.0'
- '2'
- '2.0'
openssl:
- '3'
orc:
- 2.0.2
pin_run_as_build:
python:
min_pin: x.x
max_pin: x.x
python:
- 3.10.* *_cpython
- 3.11.* *_cpython
- 3.12.* *_cpython
- 3.13.* *_cp313
- 3.9.* *_cpython
re2:
- 2024.07.02
snappy:
Expand All @@ -80,6 +96,8 @@ zip_keys:
- cuda_compiler
- cuda_compiler_version
- docker_image
- - python
- numpy
zlib:
- '1'
zstd:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,26 @@ libprotobuf:
- 5.27.5
lz4_c:
- 1.9.3
numpy:
- '2.0'
- '2.0'
- '2.0'
- '2'
- '2.0'
openssl:
- '3'
orc:
- 2.0.2
pin_run_as_build:
python:
min_pin: x.x
max_pin: x.x
python:
- 3.10.* *_cpython
- 3.11.* *_cpython
- 3.12.* *_cpython
- 3.13.* *_cp313
- 3.9.* *_cpython
re2:
- 2024.07.02
snappy:
Expand All @@ -76,6 +92,8 @@ zip_keys:
- cuda_compiler
- cuda_compiler_version
- docker_image
- - python
- numpy
zlib:
- '1'
zstd:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,26 @@ libprotobuf:
- 5.27.5
lz4_c:
- 1.9.3
numpy:
- '2.0'
- '2.0'
- '2.0'
- '2'
- '2.0'
openssl:
- '3'
orc:
- 2.0.2
pin_run_as_build:
python:
min_pin: x.x
max_pin: x.x
python:
- 3.10.* *_cpython
- 3.11.* *_cpython
- 3.12.* *_cpython
- 3.13.* *_cp313
- 3.9.* *_cpython
re2:
- 2024.07.02
snappy:
Expand All @@ -76,6 +92,8 @@ zip_keys:
- cuda_compiler
- cuda_compiler_version
- docker_image
- - python
- numpy
zlib:
- '1'
zstd:
Expand Down
9 changes: 0 additions & 9 deletions .ci_support/migrations/aws_crt_cpp0290.yaml

This file was deleted.

74 changes: 74 additions & 0 deletions .ci_support/migrations/numpy2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
__migrator:
build_number: 1
kind: version
commit_message: |
Rebuild for numpy 2.0

TL;DR: The way we build against numpy has changed as of numpy 2.0. This bot
PR has updated the recipe to account for the changes (see below for details).
The numpy 2.0 package itself is currently only available from a special release
channel (`conda-forge/label/numpy_rc`) and will not be available on the main
`conda-forge` channel until the release of numpy 2.0 GA.

The biggest change is that we no longer need to use the oldest available numpy
version at build time in order to support old numpy version at runtime - numpy
will by default use a compatible ABI for the oldest still-supported numpy versions.

Additionally, we no longer need to use `{{ pin_compatible("numpy") }}` as a
run requirement - this has been handled for more than two years now by a
run-export on the numpy package itself. The migrator will therefore remove
any occurrences of this.

However, by default, building against numpy 2.0 will assume that the package
is compatible with numpy 2.0, which is not necessarily the case. You should
check that the upstream package explicitly supports numpy 2.0, otherwise you
need to add a `- numpy <2.0dev0` run requirement until that happens (check numpy
issue 26191 for an overview of the most important packages).

Note that the numpy release candidate promises to be ABI-compatible with the
final 2.0 release. This means that building against 2.0.0rc1 produces packages
that can be published to our main channels.

If you already want to use the numpy 2.0 release candidate yourself, you can do
```
conda config --add channels conda-forge/label/numpy_rc
```
or add this channel to your `.condarc` file directly.

### To-Dos:
* [ ] Match run-requirements for numpy (i.e. check upstream `pyproject.toml` or however the project specifies numpy compatibility)
* If upstream is not yet compatible with numpy 2.0, add `numpy <2.0dev0` upper bound under `run:`.
* If upstream is already compatible with numpy 2.0, nothing else should be necessary in most cases.
* If upstream requires a minimum numpy version newer than 1.19, you can add `numpy >=x.y` under `run:`.
* [ ] Remove any remaining occurrences of `{{ pin_compatible("numpy") }}` that the bot may have missed.

PS. If the build does not compile anymore, this is almost certainly a sign that
the upstream project is not yet ready for numpy 2.0; do not close this PR until
a version compatible with numpy 2.0 has been released upstream and on this
feedstock (in the meantime, you can keep the bot from reopening this PR in
case of git conflicts by marking it as a draft).

migration_number: 1
exclude:
# needs local overrides that get stomped on by the migrator, which then fails
- scipy
# already done, but thinks its unsolvable
- pandas
ordering:
# prefer channels including numpy_rc (otherwise smithy doesn't
# know which of the two values should be taken on merge)
channel_sources:
- conda-forge
- conda-forge/label/numpy_rc,conda-forge

# needs to match length of zip {python, python_impl, numpy}
# as it is in global CBC in order to override it
numpy:
- 1.22 # no py38 support for numpy 2.0
- 2.0
- 2.0
- 2.0
- 2.0
channel_sources:
- conda-forge/label/numpy_rc,conda-forge
migrator_ts: 1713572489.295986
Loading