[CI][Packaging] Python 3.9 conda builds fail to install GCS testbench

[Tom-Newton]

Describe the bug, including details regarding any error messages, version, and platform.

 > [5/5] RUN /arrow/ci/scripts/install_gcs_testbench.sh default:
441.7   note: This error originates from a subprocess, and is likely not a problem with pip.
441.7   ERROR: Failed building wheel for grpcio
441.7   Running setup.py clean for grpcio
442.8   Building wheel for MarkupSafe (setup.py): started
443.6   Building wheel for MarkupSafe (setup.py): finished with status 'done'
443.6   Created wheel for MarkupSafe: filename=MarkupSafe-2.1.5-py3-none-any.whl size=9887 sha256=3e8d7ae11ba676a23e13b4efd471cc234b1d092881a171980eb2b72ecad2535f
443.6   Stored in directory: /root/.cache/pip/wheels/39/87/45/5b4e1bc0eeda469ebeb98b9fef379d4b16b43d4370b7823514
443.6 Successfully built googleapis-storage-testbench MarkupSafe
443.6 Failed to build grpcio
443.6 ERROR: Could not build wheels for grpcio, which is required to install pyproject.toml-based projects
------
conda-python.dockerfile:38
--------------------
  36 |     # but we changed the installed Python version above, so we need to reinstall it.
  37 |     COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts
  38 | >>> RUN /arrow/ci/scripts/install_gcs_testbench.sh default
  39 |     
  40 |     ENV ARROW_ACERO=ON \
--------------------
ERROR: failed to solve: process "/bin/bash -c -l /arrow/ci/scripts/install_gcs_testbench.sh default" did not complete successfully: exit code: 1
Service 'conda-python' failed to build : Build failed
Error: `docker-compose --file /home/runner/work/arrow/arrow/docker-compose.yml build --build-arg BUILDKIT_INLINE_CACHE=1 conda-python` exited with a non-zero exit code 1, see the process log above.

Example failure: https://github.com/apache/arrow/actions/runs/7943013070/job/21686899595?pr=40080
This is on a branch but the failure is reproducible on main with

docker-compose build conda && docker-compose build conda-cpp && PYTHON=3.9 docker-compose build conda-python

(I'm sure there is an archery command that is equivalent to this but I don't know what it is)

This failure is during the docker build step, so if it is completed successfully once I believe CI will cache it and keep returning that result. Without a cache to rely on I can reproduce this locally from main (91bf1c9c170c1917ad47bb0dbb38aa5c9fbbbfb2).

To be precise the error actually comes from pip install grpcio==1.59.0 which is a dependency of GCS testbench https://github.com/googleapis/storage-testbench/blob/43a1a82c39f6349598ec2b77f19cc336eca4ae58/setup.py#L46C10-L46C24

Component(s)

Continuous Integration

[Tom-Newton]

It looks like there was previously a similar issue with Python 3.11 #33352. I also see the same problem appearing in builds where conda stuff was not modified https://github.com/apache/arrow/actions/runs/7951551953/job/21705115595?pr=40119

[Tom-Newton]

I think I found a fix by restricting the version of cffi we build with #40120

[pitrou]

Is a bug reported upstream for this?

[xhochy]

Sam questions from me in #40120 (comment)

We can fix that on the conda-forge side so that you don't have to restrict anything here. Do you have an upstream bug report I could reference to argue for retroactively adding the pin?

Happy to fix this so that you don't need a workaround here, but I need a good reason for it.

[Tom-Newton]

I can't find any upstream bug report. I also have not reproduced yet outside of arrow (I know approximately nothing about conda).

[pitrou]

I have no idea why cffi actually causes this, since the grpcio wheel does not seem to depend on cffi...

[pitrou]

Ok, here is the reason: PyPy is being installed by mamba instead of CPython!

3.045 pip version: pip 24.0 from /opt/conda/envs/arrow/lib/pypy3.9/site-packages/pip (python 3.9)
3.046 sys.version: 3.9.18 | packaged by conda-forge | (9c4f8ef1, Jan 30 2024, 18:27:40)
3.046 [PyPy 7.3.15 with GCC 12.3.0]
3.046 sys.executable: /opt/conda/envs/arrow/bin/python3
3.046 sys.getdefaultencoding: utf-8
3.047 sys.getfilesystemencoding: ascii
3.047 locale.getpreferredencoding: UTF-8
3.047 sys.platform: linux
3.048 sys.implementation:
3.048   name: pypy

[pitrou]

(WTF)

[pitrou]

mamba log:

#8 12.11 Transaction
#8 12.11 
#8 12.11   Prefix: /opt/conda/envs/arrow
#8 12.11 
#8 12.11   Updating specs:
#8 12.11 
#8 12.11    - boto3
#8 12.11    - cffi
#8 12.11    - cython[version='>=0.29.31']
#8 12.11    - cloudpickle
#8 12.11    - fsspec
#8 12.11    - hypothesis
#8 12.11    - numpy[version='>=1.16.6']
#8 12.11    - pytest[version='<8']
#8 12.11    - pytest-faulthandler
#8 12.11    - s3fs[version='>=2023.10.0']
#8 12.11    - setuptools
#8 12.11    - setuptools_scm[version='<8.0.0']
#8 12.11    - python=3.9
#8 12.11    - nomkl
#8 12.11    - ca-certificates
#8 12.11    - openssl
#8 12.11 
#8 12.11 
#8 12.13   Package                 Version  Build                Channel           Size
#8 12.13 ────────────────────────────────────────────────────────────────────────────────
#8 12.13   Install:
#8 12.13 ────────────────────────────────────────────────────────────────────────────────
#8 12.13 
#8 12.13   + nomkl                     1.0  h5ca1d4c_0           conda-forge        4kB
#8 12.13   + python_abi                3.9  4_pypy39_pp73        conda-forge        6kB
#8 12.13   + xorg-xextproto          7.3.0  h0b41bf4_1003        conda-forge       30kB
#8 12.13   + expat                   2.5.0  hcb278e6_1           conda-forge      137kB
#8 12.13   + gdbm                     1.18  h0a1914f_2           conda-forge      195kB
#8 12.13   + sqlite                 3.45.1  h2c6b66d_0           conda-forge      848kB
#8 12.13   + libopenblas            0.3.26  pthreads_h413a1c8_0  conda-forge        6MB
#8 12.13   + xorg-libxau            1.0.11  hd590300_0           conda-forge       14kB
#8 12.13   + xorg-kbproto            1.0.7  h7f98852_1002        conda-forge       27kB
#8 12.13   + xorg-xproto            7.0.31  h7f98852_1007        conda-forge       75kB
#8 12.13   + pthread-stubs             0.4  h36c2ea0_1001        conda-forge        6kB
#8 12.13   + xorg-libxdmcp           1.1.3  h7f98852_0           conda-forge       19kB
#8 12.13   + libblas                 3.9.0  21_linux64_openblas  conda-forge       15kB
#8 12.13   + libxcb                   1.15  h0b41bf4_0           conda-forge      384kB
#8 12.13   + libcblas                3.9.0  21_linux64_openblas  conda-forge       15kB
#8 12.13   + liblapack               3.9.0  21_linux64_openblas  conda-forge       15kB
#8 12.13   + xorg-libx11             1.8.7  h8ee46fc_0           conda-forge      829kB
#8 12.13   + pypy3.9                7.3.15  h9557127_0           conda-forge       33MB
#8 12.13   + idna                      3.6  pyhd8ed1ab_0         conda-forge     Cached
#8 12.13   + pysocks                 1.7.1  pyha2e5f31_6         conda-forge     Cached
#8 12.13   + six                    1.16.0  pyh6c4a22f_0         conda-forge       14kB
#8 12.13   + jmespath                1.0.1  pyhd8ed1ab_0         conda-forge       21kB
#8 12.13   + sortedcontainers        2.4.0  pyhd8ed1ab_0         conda-forge       26kB
#8 12.13   + attrs                  23.2.0  pyh71513ae_0         conda-forge       55kB
#8 12.13   + click                   8.1.7  unix_pyh707e725_0    conda-forge       84kB
#8 12.13   + iniconfig               2.0.0  pyhd8ed1ab_0         conda-forge       11kB
#8 12.13   + exceptiongroup          1.2.0  pyhd8ed1ab_2         conda-forge       21kB
#8 12.13   + pluggy                  1.4.0  pyhd8ed1ab_0         conda-forge       23kB
#8 12.13   + colorama                0.4.6  pyhd8ed1ab_0         conda-forge     Cached
#8 12.13   + tomli                   2.0.1  pyhd8ed1ab_0         conda-forge       16kB
#8 12.13   + packaging                23.2  pyhd8ed1ab_0         conda-forge     Cached
#8 12.13   + typing_extensions       4.9.0  pyha770c72_0         conda-forge       36kB
#8 12.13   + fsspec               2024.2.0  pyhca7485f_0         conda-forge      129kB
#8 12.13   + cloudpickle             3.0.0  pyhd8ed1ab_0         conda-forge       25kB
#8 12.13   + cffi                   1.16.0  1_pypy39             conda-forge        7kB
#8 12.13   + python-dateutil         2.8.2  pyhd8ed1ab_0         conda-forge      246kB
#8 12.13   + pytest                  7.4.4  pyhd8ed1ab_0         conda-forge      245kB
#8 12.13   + aioitertools           0.11.0  pyhd8ed1ab_0         conda-forge       23kB
#8 12.13   + typing-extensions       4.9.0  hd8ed1ab_0           conda-forge       10kB
#8 12.13   + pytest-faulthandler     2.0.1  py_0                 conda-forge        5kB
#8 12.13   + async-timeout           4.0.3  pyhd8ed1ab_0         conda-forge       11kB
#8 12.13   + setuptools-scm          7.1.0  pyhd8ed1ab_0         conda-forge       39kB
#8 12.13   + setuptools_scm          7.1.0  hd8ed1ab_0           conda-forge        6kB
#8 12.13   + backports.zoneinfo      0.2.1  py39h4162558_8       conda-forge        7kB
#8 12.13   + brotli-python           1.1.0  py39hc10206b_1       conda-forge      351kB
#8 12.13   + multidict               6.0.5  py39hf860d4a_0       conda-forge       35kB
#8 12.13   + frozenlist              1.4.1  py39hf860d4a_0       conda-forge       19kB
#8 12.13   + wrapt                  1.16.0  py39hf860d4a_0       conda-forge       41kB
#8 12.13   + numpy                  1.26.4  py39h6dedee3_0       conda-forge        7MB
#8 12.13   + cython                  3.0.8  py39hc10206b_0       conda-forge        2MB
#8 12.13   + yarl                    1.9.4  py39hf860d4a_0       conda-forge       64kB
#8 12.13   + hypothesis             6.98.8  pyha770c72_0         conda-forge      319kB
#8 12.13   + urllib3               1.26.18  pyhd8ed1ab_0         conda-forge      115kB
#8 12.13   + aiosignal               1.3.1  pyhd8ed1ab_0         conda-forge       13kB
#8 12.13   + botocore              1.34.34  pyhd8ed1ab_0         conda-forge        7MB
#8 12.13   + s3transfer             0.10.0  pyhd8ed1ab_0         conda-forge       63kB
#8 12.13   + boto3                 1.34.34  pyhd8ed1ab_0         conda-forge       81kB
#8 12.13   + aiohttp                 3.9.3  py39hf860d4a_0       conda-forge      522kB
#8 12.13   + aiobotocore            2.11.2  pyhd8ed1ab_0         conda-forge       66kB
#8 12.13   + s3fs                 2024.2.0  pyhd8ed1ab_0         conda-forge       32kB
#8 12.13 
#8 12.13   Reinstall:
#8 12.13 ────────────────────────────────────────────────────────────────────────────────
#8 12.13 
#8 12.13   o wheel                  0.42.0  pyhd8ed1ab_0         conda-forge     Cached
#8 12.13   o setuptools             69.1.0  pyhd8ed1ab_0         conda-forge     Cached
#8 12.13   o pip                      24.0  pyhd8ed1ab_0         conda-forge     Cached
#8 12.13 
#8 12.13   Downgrade:
#8 12.13 ────────────────────────────────────────────────────────────────────────────────
#8 12.13 
#8 12.13   - python                3.10.13  hd12c33a_1_cpython   conda-forge     Cached
#8 12.13   + python                 3.9.18  1_73_pypy            conda-forge        7kB
#8 12.13 
#8 12.13   Summary:
#8 12.13 
#8 12.13   Install: 60 packages
#8 12.13   Reinstall: 3 packages
#8 12.13   Downgrade: 1 packages
#8 12.13 
#8 12.13   Total download: 59MB

[Tom-Newton]

@pitrou do you have a suggestion for a fix? Is it worth merging #40120 at least as a short term fix?

[xhochy]

If pypy is the problem, you should add python=*=*_cpython as an additional dependency to the conda requirements files. This will ensure that pypy is never picked.

There must be a problem with dependencies on cpython in the above example. The pypy package is configured in a way that it should never be automatically selected if there is a way to build the environment with CPython.

[pitrou]

Ok, I've opened #40131 with that suggestion.

[pitrou]

@xhochy It does not work:

 > [3/5] RUN mamba install -q -y         --file arrow/ci/conda_env_python.txt         $([ "3.8" == $(gdb --batch --eval-command 'python import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")') ] && echo "gdb")         "python=3.8=*_cpython"         nomkl &&     mamba clean --all:
10.38 Could not solve for environment specs
10.38 The following package could not be installed
10.38 └─ python 3.8 *_cpython does not exist (perhaps a typo or a missing channel).