pyarrow breaks when updating from 15 to 16

[visr]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

If I add pyarrow version 15 or 16 to an empty project, either version works fine. If I update from pyarrow 15 to 16 as shown below, pyarrow stops working.

pixi init
pixi add pyarrow=15
pixi run python -c "from pyarrow import array"
# works
pixi add pyarrow=16
pixi run python -c "from pyarrow import array"
# ImportError: cannot import name 'array' from 'pyarrow' (unknown location)

Issue description

I also don't get this issue if I pixi rm pyarrow in between pixi add pyarrow=15 and pixi add pyarrow=16. Both paths lead to identical pixi.lock files with pyarrow 16, but I noticed in the broken project all files under .pixi/envs/default/Lib/site-packages/pyarrow have names like array.pxi__clobber-from-pyarrow-core.

When updating from 16.0 to 16.1, or 16 to 15, there is also no issue.

What I don't know if this is really an issue with pixi or the pyarrow-feedstock.

I use pixi 0.25.0 with:

channels = ["conda-forge"]
platforms = ["win-64"]

Possibly related to #1194.

Expected behavior

I expect pixi run python -c "from pyarrow import array" to work regardless of how I got there.

[Hofer-Julian]

Hey Martijn 👋🏻

I can reproduce it on my (Linux) machine. No idea yet what is going wrong here, though.

[ruben-arts]

I can reproduce and the issue seems to be with what gets left behind when you update the package.

pixi init
pixi add pyarrow=15
pixi run python -c "from pyarrow import array"
# works
pixi add pyarrow=16
pixi run python -c "from pyarrow import array"
# ImportError: cannot import name 'array' from 'pyarrow' (unknown location)

# Now this works:
pixi clean
pixi run python -c "from pyarrow import array"
# works

Is see that almost everything gets a clobber warning:

Lib\site-packages\pyarrow
❯ ls
╭─────┬──────────────────────────────────────────────────────────────────┬──────┬───────────┬─────╮
│   # │                               name                               │ type │   size    │ ... │
├─────┼──────────────────────────────────────────────────────────────────┼──────┼───────────┼─────┤
│   0 │ __init__.pxd__clobber-from-pyarrow-core                          │ file │   2.1 KiB │ ... │
│   1 │ __init__.py__clobber-from-pyarrow-core                           │ file │  17.4 KiB │ ... │
│   2 │ __pycache__                                                      │ dir  │   8.0 KiB │ ... │
│   3 │ _acero.cp312-win_amd64.pyd__clobber-from-pyarrow-core            │ file │ 198.5 KiB │ ... │
│   4 │ _acero.pxd__clobber-from-pyarrow-core                            │ file │   1.4 KiB │ ... │
│   5 │ _acero.pyx__clobber-from-pyarrow-core                            │ file │  20.8 KiB │ ... │
│   6 │ _azurefs.pyx                                                     │ file │   5.8 KiB │ ... │
│   7 │ _compute.cp312-win_amd64.pyd__clobber-from-pyarrow-core          │ file │ 799.5 KiB │ ... │
│   8 │ _compute.pxd__clobber-from-pyarrow-core                          │ file │   1.9 KiB │ ... │
│   9 │ _compute.pyx__clobber-from-pyarrow-core                          │ file │ 103.7 KiB │ ... │

[baszalmstra]

The reason this happens seems to be that pyarrow=16 introduced a new "sub" package called pyarrow-core. The new pyarrow-core package actually includes all the files previously in pyarrow. This causes a conflict. Ill take a closer look on Thursday!

[ruben-arts]

Ah thanks!

[baszalmstra]

I worked on a fix in conda/rattler#776 which seems to fix this issue!