Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-40079: [CI][Packaging] Enable Azure in more tests and builds #40080

Merged
merged 11 commits into from
Feb 22, 2024
Merged

GH-40079: [CI][Packaging] Enable Azure in more tests and builds #40080

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
11cfcb6
Add missing export that prevented any pyarrow tests from running
Tom-Newton Feb 14, 2024
8064499
Enable pyarrow tests on sdist_test
Tom-Newton Feb 14, 2024
1f455b3
Enable Azure on python macos builds
Tom-Newton Feb 14, 2024
cf8ce6a
Enable azure in conda builds and install dependencies
Tom-Newton Feb 14, 2024
5f1e50b
Install node from conda instead of apt
Tom-Newton Feb 14, 2024
79b1c5d
Update comment
Tom-Newton Feb 15, 2024
2e3e51c
Pin specific versions to avoid segfault
Tom-Newton Feb 17, 2024
cbe5c46
Less restrictive pin and add a comment about the issue
Tom-Newton Feb 18, 2024
7c27978
Correct comment typo
Tom-Newton Feb 18, 2024
5e3f846
Add retry mitigation suggested for GH-40121
Tom-Newton Feb 19, 2024
53b9d2c
Remove conda version restriction now that latest release works
Tom-Newton Feb 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ jobs:
timeout-minutes: 60
env:
ARROW_HOME: /usr/local
ARROW_AZURE: ON
ARROW_DATASET: ON
ARROW_FLIGHT: ON
ARROW_GANDIVA: ON
Expand Down
7 changes: 7 additions & 0 deletions ci/conda_env_cpp.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@
# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that is a relevant link. AFAIU there is not really a problem with the 1.11.0 conda package itself. It's that conda did not pin azure-core-cpp in other packages (eg building tiledb with azure-core-cpp against 1.10, and then running that in an env with 1.11 installed, then we got segfaults).

But for the purpose of our CI, we are both building and running against the same azure-core-cpp version, right? In that case, I would expect that building+running with azure-core-cpp would work just fine. Or if not, that means our azurefs implementation might not yet be compatible with the latest azure-core-cpp?

Copy link
Contributor Author

@Tom-Newton Tom-Newton Feb 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I tried using 1.11.0 building from source with cmake and everything worked. With 1.10.0 1.11.0 from conda we got segfaults.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean 1.11 from conda? (not 1.10) Do you remember if there was more information in the logs about the crash?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#40080 (comment) summarises the info we have.

It looks like 1.11.1 is now available from conda though and that works, so I can remove this version restriction.

azure-core-cpp>=1.10.3,<1.11.0
azure-identity-cpp>=1.6.0
azure-storage-blobs-cpp>=12.10.0
azure-storage-common-cpp>=12.5.0
azure-storage-files-datalake-cpp>=12.9.0
benchmark>=1.6.0
boost-cpp>=1.68.0
brotli
Expand All @@ -34,6 +40,7 @@ libutf8proc
lz4-c
make
ninja
nodejs
orc
pkg-config
python
Expand Down
8 changes: 8 additions & 0 deletions ci/docker/conda-cpp.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,13 @@ RUN mamba install -q -y \
valgrind && \
mamba clean --all

# Ensure npm, node and azurite are on path. npm and node are required to install azurite, which will then need to
# be on the path for the tests to run.
ENV PATH=/opt/conda/envs/arrow/bin:$PATH

COPY ci/scripts/install_azurite.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_azurite.sh

# We want to install the GCS testbench using the same Python binary that the Conda code will use.
COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts
RUN /arrow/ci/scripts/install_gcs_testbench.sh default
Expand All @@ -50,6 +57,7 @@ COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ENV ARROW_ACERO=ON \
ARROW_AZURE=ON \
ARROW_BUILD_TESTS=ON \
ARROW_DATASET=ON \
ARROW_DEPENDENCY_SOURCE=CONDA \
Expand Down
1 change: 1 addition & 0 deletions ci/scripts/cpp_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ ctest \
--label-regex unittest \
--output-on-failure \
--parallel ${n_jobs} \
--repeat until-pass:3 \
assignUser marked this conversation as resolved.
Show resolved Hide resolved
--timeout ${ARROW_CTEST_TIMEOUT:-300} \
"${ctest_options[@]}" \
"$@"
Expand Down
1 change: 1 addition & 0 deletions ci/scripts/python_sdist_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ export PARQUET_TEST_DATA=${arrow_dir}/cpp/submodules/parquet-testing/data
export PYARROW_CMAKE_GENERATOR=${CMAKE_GENERATOR:-Ninja}
export PYARROW_BUILD_TYPE=${CMAKE_BUILD_TYPE:-debug}
export PYARROW_WITH_ACERO=${ARROW_ACERO:-ON}
export PYARROW_WITH_AZURE=${ARROW_AZURE:-OFF}
export PYARROW_WITH_S3=${ARROW_S3:-OFF}
export PYARROW_WITH_ORC=${ARROW_ORC:-OFF}
export PYARROW_WITH_CUDA=${ARROW_CUDA:-OFF}
Expand Down
1 change: 1 addition & 0 deletions ci/scripts/python_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ fi
: ${PYARROW_TEST_S3:=${ARROW_S3:-ON}}

export PYARROW_TEST_ACERO
export PYARROW_TEST_AZURE
export PYARROW_TEST_CUDA
export PYARROW_TEST_DATASET
export PYARROW_TEST_FLIGHT
Expand Down
Loading