[CI][Archery] Some Azure jobs fail on pip install -e arrow/dev/archery[docker]

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

Some azure jobs on the maintenance branch for 16.0.0 are currently failing to install archery with:

pip install -e arrow/dev/archery[docker]
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/8c33e1f6-f05d-42a3-b425-f289fff01a2a.sh
ERROR: arrow/dev/archery[docker] is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).

[notice] A new release of pip is available: 23.0.1 -> 24.0
[notice] To update, run: pip install --upgrade pip

https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=62987&view=logs&j=50a69d0a-7972-5459-cdae-135ee6ebe312&t=e37d4d74-66f8-5ec4-c216-74b731c58e49&l=19
See maintenance validation PR with links to jobs here: #41118 (comment)

Component(s)

Archery, Continuous Integration

[kou]

Hmm. It seems that git clone failed:

https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=62987&view=logs&j=50a69d0a-7972-5459-cdae-135ee6ebe312&t=344e7f7d-9526-5715-882a-1f0e8858127b&l=13

fatal: couldn't find remote ref maint-16.0.0
error: pathspec 'FETCH_HEAD' did not match any file(s) known to git

Could you try this on maint-16.0.0?

diff --git a/dev/tasks/docker-tests/azure.linux.yml b/dev/tasks/docker-tests/azure.linux.yml
index b66bfbdfe9..a2c49d9f29 100644
--- a/dev/tasks/docker-tests/azure.linux.yml
+++ b/dev/tasks/docker-tests/azure.linux.yml
@@ -34,13 +34,7 @@ jobs:
     inputs:
       versionSpec: '3.8'
 
-  - script: |
-      git clone --no-checkout {{ arrow.remote }} arrow
-      git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
-      git -C arrow checkout FETCH_HEAD
-      git -C arrow submodule update --init --recursive
-      git -C arrow remote add upstream https://github.com/apache/arrow.git
-    displayName: Clone arrow
+  {{ macros.azure_checkout_arrow() }}
 
   - script: pip install -e arrow/dev/archery[docker]
     displayName: Setup Archery

[pitrou]

Also, we should ensure the build fails as soon as git clone fails, not later with an unrelated error message.

[kou]

Issue resolved by pull request 41153
#41153

[kou]

Also, we should ensure the build fails as soon as git clone fails, not later with an unrelated error message.

We use GitHub Actions instead of Azure Pipelines by #41153. So our jobs use the behavior now.