GH-40954: [CI] Fix use of obsolete docker-compose command on Github Actions #40949

pitrou · 2024-04-02T14:58:06Z

Rationale for this change

The docker-compose utility is progressively being removed from GHA-provided runners:
actions/runner-images#9557

What changes are included in this PR?

Use docker client CLI directly instead of docker-compose where possible.

Are these changes tested?

Yes, this should fix the sporadic CI failures because of the above migration.

Are there any user-facing changes?

No, except additional optional env var ARCHERY_DEBUG.

GitHub Issue: [CI] Failures on GHA due to missing docker-compose command #40954

github-actions · 2024-04-02T14:58:34Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

pitrou · 2024-04-02T16:38:01Z

@github-actions crossbow submit -g nightly

pitrou · 2024-04-02T16:42:04Z

@github-actions crossbow submit -g cpp

github-actions · 2024-04-02T16:44:37Z

Revision: 2304b1b

Submitted crossbow builds: ursacomputing/crossbow @ actions-81f3a4a053

Task	Status
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-14

pitrou · 2024-04-02T16:50:49Z

@github-actions crossbow submit -g linux

github-actions · 2024-04-02T16:53:42Z

Revision: 2304b1b

Submitted crossbow builds: ursacomputing/crossbow @ actions-2847c15c67

Task	Status
almalinux-8-amd64
almalinux-8-arm64
almalinux-9-amd64
almalinux-9-arm64
amazon-linux-2023-amd64
amazon-linux-2023-arm64
centos-7-amd64
centos-8-stream-amd64
centos-8-stream-arm64
centos-9-stream-amd64
centos-9-stream-arm64
debian-bookworm-amd64
debian-bookworm-arm64
debian-bullseye-amd64
debian-bullseye-arm64
debian-trixie-amd64
debian-trixie-arm64
ubuntu-focal-amd64
ubuntu-focal-arm64
ubuntu-jammy-amd64
ubuntu-jammy-arm64
ubuntu-noble-amd64
ubuntu-noble-arm64

github-actions · 2024-04-02T16:53:42Z

⚠️ GitHub issue #40954 has been automatically assigned in GitHub to PR creator.

pitrou · 2024-04-02T16:58:18Z

@github-actions crossbow submit cuda

github-actions · 2024-04-02T17:00:58Z

Revision: 6f89a99

Submitted crossbow builds: ursacomputing/crossbow @ actions-be8b558ace

Task	Status
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cuda-py3
conda-win-x64-cuda-py3
test-cuda-cpp
test-cuda-python

pitrou · 2024-04-03T07:48:57Z

@raulcd @assignUser @kou Any further comments here? Otherwise it would be nice to merge this quickly.

### Rationale for this change Same as PR #40949, but for Crossbow builds hosted on Azure Pipelines. ### Are these changes tested? Yes, they fix CI tests. ### Are there any user-facing changes? No. Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

conbench-apache-arrow · 2024-04-03T14:50:11Z

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit be3b789.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

We can't use multiple top-level `env:` in workflow. apacheGH-40949 introduced a top-level `env:` by `macros.github_header()`. It broke workflows that already have top-level `env:`.

…1091) ### Rationale for this change We can't use multiple top-level `env:` in workflow. GH-40949 introduced a top-level `env:` by `macros.github_header()`. It broke workflows that already have top-level `env:`. ### What changes are included in this PR? Omit top-level `env:` key and reuse the top-level `env:` key generated by `macros.github_header()` in workflows. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #41088 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…or (apache#41091) ### Rationale for this change We can't use multiple top-level `env:` in workflow. apacheGH-40949 introduced a top-level `env:` by `macros.github_header()`. It broke workflows that already have top-level `env:`. ### What changes are included in this PR? Omit top-level `env:` key and reuse the top-level `env:` key generated by `macros.github_header()` in workflows. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#41088 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…thub Actions (apache#40949) ### Rationale for this change The `docker-compose` utility is progressively being removed from GHA-provided runners: actions/runner-images#9557 ### What changes are included in this PR? Use `docker` client CLI directly instead of `docker-compose` where possible. ### Are these changes tested? Yes, this should fix the sporadic CI failures because of the above migration. ### Are there any user-facing changes? No, except additional optional env var `ARCHERY_DEBUG`. * GitHub Issue: apache#40954 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

### Rationale for this change Same as PR apache#40949, but for Crossbow builds hosted on Azure Pipelines. ### Are these changes tested? Yes, they fix CI tests. ### Are there any user-facing changes? No. Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…or (apache#41091) ### Rationale for this change We can't use multiple top-level `env:` in workflow. apacheGH-40949 introduced a top-level `env:` by `macros.github_header()`. It broke workflows that already have top-level `env:`. ### What changes are included in this PR? Omit top-level `env:` key and reuse the top-level `env:` key generated by `macros.github_header()` in workflows. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#41088 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…thub Actions (apache#40949) ### Rationale for this change The `docker-compose` utility is progressively being removed from GHA-provided runners: actions/runner-images#9557 ### What changes are included in this PR? Use `docker` client CLI directly instead of `docker-compose` where possible. ### Are these changes tested? Yes, this should fix the sporadic CI failures because of the above migration. ### Are there any user-facing changes? No, except additional optional env var `ARCHERY_DEBUG`. * GitHub Issue: apache#40954 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

Same as PR apache#40949, but for Crossbow builds hosted on Azure Pipelines. Yes, they fix CI tests. No. Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…thub Actions (apache#40949) ### Rationale for this change The `docker-compose` utility is progressively being removed from GHA-provided runners: actions/runner-images#9557 ### What changes are included in this PR? Use `docker` client CLI directly instead of `docker-compose` where possible. ### Are these changes tested? Yes, this should fix the sporadic CI failures because of the above migration. ### Are there any user-facing changes? No, except additional optional env var `ARCHERY_DEBUG`. * GitHub Issue: apache#40954 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

Same as PR apache#40949, but for Crossbow builds hosted on Azure Pipelines. Yes, they fix CI tests. No. Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…thub Actions (apache#40949) ### Rationale for this change The `docker-compose` utility is progressively being removed from GHA-provided runners: actions/runner-images#9557 ### What changes are included in this PR? Use `docker` client CLI directly instead of `docker-compose` where possible. ### Are these changes tested? Yes, this should fix the sporadic CI failures because of the above migration. ### Are there any user-facing changes? No, except additional optional env var `ARCHERY_DEBUG`. * GitHub Issue: apache#40954 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

Same as PR apache#40949, but for Crossbow builds hosted on Azure Pipelines. Yes, they fix CI tests. No. Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…thub Actions (apache#40949) ### Rationale for this change The `docker-compose` utility is progressively being removed from GHA-provided runners: actions/runner-images#9557 ### What changes are included in this PR? Use `docker` client CLI directly instead of `docker-compose` where possible. ### Are these changes tested? Yes, this should fix the sporadic CI failures because of the above migration. ### Are there any user-facing changes? No, except additional optional env var `ARCHERY_DEBUG`. * GitHub Issue: apache#40954 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

### Rationale for this change Same as PR apache#40949, but for Crossbow builds hosted on Azure Pipelines. ### Are these changes tested? Yes, they fix CI tests. ### Are there any user-facing changes? No. Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…or (apache#41091) ### Rationale for this change We can't use multiple top-level `env:` in workflow. apacheGH-40949 introduced a top-level `env:` by `macros.github_header()`. It broke workflows that already have top-level `env:`. ### What changes are included in this PR? Omit top-level `env:` key and reuse the top-level `env:` key generated by `macros.github_header()` in workflows. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#41088 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

[CI] Try to diagnose sporadic docker-compose issues

6d3c491

github-actions bot added the awaiting review Awaiting review label Apr 2, 2024

pitrou changed the title ~~[CI] Try to diagnose sporadic docker-compose issues~~ [CI] Try to fix sporadic docker-compose issues Apr 2, 2024

pitrou force-pushed the ci-docker-compose branch 5 times, most recently from b2604f0 to 1ddc67d Compare April 2, 2024 15:56

Try fix

6a11f94

pitrou force-pushed the ci-docker-compose branch from 1ddc67d to 6a11f94 Compare April 2, 2024 15:59

Generalize fix

3bc99c5

Fix Archery unit tests

2304b1b

pitrou changed the title ~~[CI] Try to fix sporadic docker-compose issues~~ GH-40954: [CI] Try to fix sporadic docker-compose issues Apr 2, 2024

pitrou marked this pull request as ready for review April 2, 2024 16:54

pitrou requested review from assignUser, kou and raulcd as code owners April 2, 2024 16:54

pitrou changed the title ~~GH-40954: [CI] Try to fix sporadic docker-compose issues~~ GH-40954: [CI] Fix use of obsolete docker-compose command on Github Actions Apr 2, 2024

pitrou added 2 commits April 2, 2024 18:55

Disable docker CLI on CUDA builds

b88fc75

Remove flag redundant with env var

6f89a99

apache deleted a comment from github-actions bot Apr 2, 2024

assignUser mentioned this pull request Apr 2, 2024

GH-40952: [Java][FlightSQL] Clean up flight-sql-jdbc-driver dependencies #40953

Merged

pitrou merged commit be3b789 into apache:main Apr 3, 2024
62 of 63 checks passed

pitrou removed the awaiting committer review Awaiting committer review label Apr 3, 2024

pitrou mentioned this pull request Apr 3, 2024

[CI] Failures on GHA due to missing docker-compose command #40954

Closed

pitrou deleted the ci-docker-compose branch April 3, 2024 08:33

pitrou mentioned this pull request Apr 3, 2024

MINOR: [CI] Avoid docker-compose on Azure #40972

Merged

assignUser approved these changes Apr 3, 2024

View reviewed changes

github-actions bot added the awaiting committer review Awaiting committer review label Apr 3, 2024

This was referenced Apr 4, 2024

[CI][Release] Verification jobs are failing due to curl failure #41007

Closed

[CI][Java] Crossbow verify-rc-source-java-macos-amd64 fails with a format error in configs #41088

Closed

kou mentioned this pull request Apr 9, 2024

GH-41088: [CI][Crossbow] Fix GitHub Actions workflow syntax error #41091

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-40954: [CI] Fix use of obsolete docker-compose command on Github Actions #40949

GH-40954: [CI] Fix use of obsolete docker-compose command on Github Actions #40949

pitrou commented Apr 2, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Apr 2, 2024

pitrou commented Apr 2, 2024

pitrou commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

pitrou commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

pitrou commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

pitrou commented Apr 3, 2024

conbench-apache-arrow bot commented Apr 3, 2024

GH-40954: [CI] Fix use of obsolete docker-compose command on Github Actions #40949

GH-40954: [CI] Fix use of obsolete docker-compose command on Github Actions #40949

Conversation

pitrou commented Apr 2, 2024 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Apr 2, 2024

pitrou commented Apr 2, 2024

pitrou commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

pitrou commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

pitrou commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

pitrou commented Apr 3, 2024

conbench-apache-arrow bot commented Apr 3, 2024

pitrou commented Apr 2, 2024 •

edited by github-actions bot

Loading