fetch specific commits even if the github fast path fails

[iliana]

What does this PR try to resolve?

This PR fixes #13555, which describes a regression from 1.64.0 to 1.65.0 where the inability to fetch commit information from api.github.com (the "GitHub fast path") silently changes Cargo's behavior.

Cargo can fetch a specific Git commit from a remote without having to fetch all refs. Prior to #10807, this functionality required a repository hosted on github.com and providing the full commit hash (usually available from the Cargo.lock); after that change, any revision (including abbreviated revisions) that could be resolved by GitHub's API could be fetched directly. However, this logic requires the "GitHub fast path", which was not intended to be robust, to successfully return the resolved commit hash; if a client is currently rate-limited by api.github.com (very common in CI and shared cloud / corporate environments) this fails and Cargo falls back to fetching all refs.

Usually this is not noticeable. However, GitHub allows fetching commits that are related to the repository but not actually part of any of its refs, including commits pushed to a fork. This results in the same command working fine in some environments where api.github.com is accessible, and not working in other environments that are rate-limited, which is very confusing and difficult to debug.

This change adds another branch to cover the regression case: if we are going through the GitHub fast path with a full commit hash, return early indicating that we need to fetch it. (Previously: when the GitHub fast path was unsuccessful, the user is not using the unstable shallow clone options, and we have a full commit hash and expect to be able to fetch it directly because we know it's a github.com repository.)

How should we test and review this PR?

I have been testing this PR by temporarily adding a 0.0.0.0 api.github.com entry to my /etc/hosts, which causes the GitHub fast path to always fail, then running:

target/release/cargo install --git https://github.com/haha-business/unstable-test-repo.git --rev c9040898c9183ddbb9402dcbf749ed06d6ea90ad

This refers to a particular commit on a fork of the repo which won't be found by the fallback path or current Cargo.

Note that you will need to delete ~/.cargo/git/checkouts/unstable-test-repo-* and ~/.cargo/git/db/unstable-test-repo-* after a successful run with this change in order to reproduce the broken behavior of the current release.

I am having trouble getting the test suite to run at all on my system so I haven't experimented with writing a specific test for this case, but I probably should.

Additional information

This uses the same logic as the unstable shallow clone support to detect if the revision is a full commit hash. This is not compatible with SHA-256 commit hashes; git2::Oid specifically expects a 40-character hexadecimal string. Given that the change introducing this bug was meant to future-proof SHA-256 support (despite only doing so for GitHub repositories), it might be good to make the logic more explicit within Cargo and allow either 40- or 64-character hex strings.

I wanted to keep this change focused on the regression fix, but in testing, pretty much every Git repository I could think of (including non-forges, like git.kernel.org and some repositories I host on my own infrastructure with cgit) supports fetching directly from a commit, so it would be ideal to eventually relax the GitHub requirement for this functionality. However, it would need some sort of fallback logic because I suspect the HTTP dumb protocol doesn't support commit references, and I haven't researched when this functionality was added to the smart protocol.

[rustbot]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @weihanglo (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[weihanglo]

Thank you for the nice finding and write-up!

I understand supporting other Git hosting would be awesome, though this is still a GitHub specific fast path. I wonder if we could move this patch to under github_fast_path. Maybe something like this?

                if let Some(local_object) = local_object {
                    if is_short_hash_of(rev, local_object) {
                        debug!("github fast path already has {local_object}");
                        return Ok(FastPathRev::UpToDate);
                    }
                }
+               if let Ok(oid) = rev.parse::<Oid>() {
+                    debug!("github fast path is already a full commit hash {rev}");
+                    return Ok(FastPathRev::NeedsFetch(oid))
+               }
                rev
            } else {
                debug!("can't use github fast path with `rev = \"{}\"`", rev);
                return Ok(FastPathRev::Indeterminate);
            }

[iliana]

Oh that's clever! Let me give that a shot.

[iliana]

I added a Co-authored-by header to my commit since you wrote the actual code! Let me know if that should be adjusted.

It looks like that approach also has the benefit of making the behavior around what the ref is fetched as more consistent; previously if the GitHub API request failed the ref would instead be fetched to refs/remotes/origin/HEAD instead of refs/commit/{commit_hash}. Given that was never supposed to happen, and the ref we fetch as is pretty meaningless in the context of these Cargo-managed repos, I think that's fine.

[weihanglo]

Looks great! Thank you for the contribution.

Let's move on and leave off the issue of handling 64-hex-digit commit hash, believing this part won't be the only place need to change at that time 😬.

[weihanglo]

@bors r+

[bors]

📌 Commit 5d7a06b has been approved by weihanglo

It is now in the queue for this repository.

[bors]

⌛ Testing commit 5d7a06b with merge 99075f2...

[bors]

☀️ Test successful - checks-actions
Approved by: weihanglo
Pushing 99075f2 to master...