Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-25118: [Python] Make NumPy an optional runtime dependency #41904

Merged
merged 66 commits into from
Sep 2, 2024
Merged

GH-25118: [Python] Make NumPy an optional runtime dependency #41904

merged 66 commits into from
Sep 2, 2024

Commits on Sep 2, 2024

  1. Minimal changes to make possible to import pyarrow and create a pyarr… 

    …ow.array without numpy
    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    8d7db71 View commit details
    Browse the repository at this point in the history

  2. Add minimal CI test

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    bf729e0 View commit details
    Browse the repository at this point in the history

  3. Add minor linting fixes

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    0763dce View commit details
    Browse the repository at this point in the history

  4. Minor fix for function call instead of dict access

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    50ebbe1 View commit details
    Browse the repository at this point in the history

  5. Refactor tests to be able to run test_array.py with and without numpy

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    6e4fbe1 View commit details
    Browse the repository at this point in the history

  6. Fix linting for C++ files

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    9ca7a82 View commit details
    Browse the repository at this point in the history

  7. Fix a bunch more tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    98af6e4 View commit details
    Browse the repository at this point in the history

  8. Several changes related to initial code review comments

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    60bd002 View commit details
    Browse the repository at this point in the history

  9. Fix wrong mark

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    70b6237 View commit details
    Browse the repository at this point in the history

  10. Minor lint

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    a3aea68 View commit details
    Browse the repository at this point in the history

  11. Add HAS_NUMPY in order to avoid 'numpy' in sys.modules every time

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    0b92561 View commit details
    Browse the repository at this point in the history

  12. Add last bits that crashed some tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    e2a2172 View commit details
    Browse the repository at this point in the history

  13. Some more improvements

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    19f013a View commit details
    Browse the repository at this point in the history

  14. Use PYTEST_PYARGS to define module to test

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    3bd78cb View commit details
    Browse the repository at this point in the history

  15. Add test_without_numpy test module and nonumpy marker. Add automarker… 

    … if numpy/pandas/ndarray is part of the test name
    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    d1d74a6 View commit details
    Browse the repository at this point in the history

  16. Add tensor test

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    b0e26c0 View commit details
    Browse the repository at this point in the history

  17. Add missing license

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    8e92e7c View commit details
    Browse the repository at this point in the history

  18. Minor tests refactor

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    05c9a71 View commit details
    Browse the repository at this point in the history

  19. Add some more tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    6fbff8e View commit details
    Browse the repository at this point in the history

  20. Rollback change on pyarrow/tests/test_adhoc_memory_leak.py

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    fd15246 View commit details
    Browse the repository at this point in the history

  21. Remove HAS_NUMPY helper variable

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    c5e1567 View commit details
    Browse the repository at this point in the history

  22. Some other import modules and test changes

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    f292b1f View commit details
    Browse the repository at this point in the history

  23. Remove stray import

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    d7a8de7 View commit details
    Browse the repository at this point in the history

  24. Remove without_numpy from test_array and mark the numpy tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    0d2886b View commit details
    Browse the repository at this point in the history

  25. Remove without_numpy from test_compute and mark the numpy tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    890f6d4 View commit details
    Browse the repository at this point in the history

  26. Remove without_numpy from test_dataset and mark the numpy tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    b885984 View commit details
    Browse the repository at this point in the history

  27. Remove without_numpy from test_flight

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    80a0abe View commit details
    Browse the repository at this point in the history

  28. Collect from test_cpp_internals only tests that do not require numpy … 

    …or pandas if numpy is not present
    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    ba4805f View commit details
    Browse the repository at this point in the history

  29. Mark numpy tests for pyarrow/tests/test_csv.py

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    2ca233c View commit details
    Browse the repository at this point in the history

  30. Add pytest.mark.numpy to all test_cuda and test_cuda_numba_interop tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    82cbdd8 View commit details
    Browse the repository at this point in the history

  31. Fix test collection and add corresponding marks for numpy

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    a31e55a View commit details
    Browse the repository at this point in the history

  32. Fix linting

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    92966bf View commit details
    Browse the repository at this point in the history

  33. Fix if on Digest and Quantile Options

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    a166d51 View commit details
    Browse the repository at this point in the history

  34. Remove unnecessary test fixture and last review comments

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    1bba8b7 View commit details
    Browse the repository at this point in the history

  35. Fix tests that without numpy where an ImportError is raised instead o… 

    …f ValueError
    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    e6ff932 View commit details
    Browse the repository at this point in the history

  36. Review comments: Update base image from no-numpy, use mamba uninstall… 

    …, update Numpy to NumPy and add correct markers for test_cpp_internals
    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    b3341b7 View commit details
    Browse the repository at this point in the history

  37. Some more review comments to remove usage of numpy on some tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    fcd37c3 View commit details
    Browse the repository at this point in the history

  38. Remove numpy from CSV tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    d28f9c8 View commit details
    Browse the repository at this point in the history

  39. Remove unnecessary numpy usages from test_dataset

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    e231a05 View commit details
    Browse the repository at this point in the history

  40. Fix several tests to do not require numpy on test_io

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    9c8d5c8 View commit details
    Browse the repository at this point in the history

  41. Fix tests and remove usages of numpy mark on test_ipc, fix fixture

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    1f9077d View commit details
    Browse the repository at this point in the history

  42. Revert back and use fixture for np types using strings

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    b7e2a56 View commit details
    Browse the repository at this point in the history

  43. Simplify if by using SUPPORTED_INPUT_ARR_TYPES

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    190e1f6 View commit details
    Browse the repository at this point in the history

  44. Add numpy or pandas mark to wrapper when collection C++ injected tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    02911d4 View commit details
    Browse the repository at this point in the history

  45. Remove skip when numpy on test_convert_builtin.py and add individual … 

    …test skips
    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    2cb7556 View commit details
    Browse the repository at this point in the history

  46. Remove empty stray lines

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    9b44792 View commit details
    Browse the repository at this point in the history

  47. Fix stray change

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    3e8c97d View commit details
    Browse the repository at this point in the history

  48. Remove stray import

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    8766b84 View commit details
    Browse the repository at this point in the history

  49. Mark hypothesis tests that require numpy

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    90455d3 View commit details
    Browse the repository at this point in the history

  50. Use available SEQUENCE_TYPES to inner sequence on nested arrays

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    ea22c30 View commit details
    Browse the repository at this point in the history

  51. Fix test collection for tests/interchange/test_conversion.py

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    630b607 View commit details
    Browse the repository at this point in the history

  52. Fix test collection for tests/interchange/test_interchange_spec.py

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    7963828 View commit details
    Browse the repository at this point in the history

  53. Remove numpy dependency from pyarrow/tests/test_dataset_encryption.py

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    547d98e View commit details
    Browse the repository at this point in the history

  54. Apply suggestions from code review 

    Co-authored-by: Antoine Pitrou <[email protected]>
    @raulcd @pitrou
    raulcd and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    51636e6 View commit details
    Browse the repository at this point in the history

  55. Fix test collection for pyarrow/tests/test_dlpack.py

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    ecd2e4f View commit details
    Browse the repository at this point in the history

  56. Fix test collection for pyarrow/tests/test_extension_type.py

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    d88a3f5 View commit details
    Browse the repository at this point in the history

  57. Fix test collection for pyarrow/tests/scalars.py

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    32304f8 View commit details
    Browse the repository at this point in the history

  58. Add comment to why cython tests require NumPy (still a build dependency)

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    f5420c2 View commit details
    Browse the repository at this point in the history

  59. Fix test collection for pyarrow/tests/pandas.py

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    6365ef2 View commit details
    Browse the repository at this point in the history

  60. Fixes to numpy tests

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    3d99652 View commit details
    Browse the repository at this point in the history

  61. Update python/pyarrow/tests/test_pandas.py 

    Co-authored-by: Joris Van den Bossche <[email protected]>
    @raulcd @jorisvandenbossche @pitrou
    2 people authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    6d54ffa View commit details
    Browse the repository at this point in the history

  62. Update python/pyarrow/tests/test_csv.py 

    Co-authored-by: Antoine Pitrou <[email protected]>
    @raulcd @pitrou
    raulcd and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    8560239 View commit details
    Browse the repository at this point in the history

  63. Update python/pyarrow/tests/test_csv.py 

    Co-authored-by: Antoine Pitrou <[email protected]>
    @raulcd @pitrou
    raulcd and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    a4e094c View commit details
    Browse the repository at this point in the history

  64. Remove numpy mark and add np guard to test

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    eaabab7 View commit details
    Browse the repository at this point in the history

  65. Rebase main and add mark to skip new tests that require numpy

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    1277b84 View commit details
    Browse the repository at this point in the history

  66. Add comment on why we duplicate test_basics_np_required for float16

    @raulcd @pitrou
    raulcd authored and pitrou committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    08da867 View commit details
    Browse the repository at this point in the history