crane: unable to copy between to registries (registry.gitlab.com and docker.io)

[ProbstDJakob]

Describe the bug

We copy images from gitlab.com to docker.io and until version 0.9.0 this worked fine. With version 0.10.0 this stopped working. The process has not been changed (written as pipeline job) and all authentication tokens still work. The error we receive is the following:

2022/07/04 08:31:37 Copying from registry.gitlab.com/bbbatscale/bbbatscale/bbbatscale:4.6.0 to docker.io/bbbatscale/bbbatscale:4.6.0
Error: failed to copy image: POST https://index.docker.io/v2/bbbatscale/bbbatscale/blobs/uploads/?from=bbbatscale%2Fbbbatscale%2Fbbbatscale&mount=sha256%3Ab85a868b505ffd0342a37e6a3b1c49f7c71878afe569a807e6238ef08252fcb7&origin=REDACTED: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:bbbatscale/bbbatscale Type:repository] map[Action:push Class: Name:bbbatscale/bbbatscale Type:repository] map[Action:pull Class: Name:bbbatscale/bbbatscale/bbbatscale Type:repository]]

When executing the same command with crane v0.9.0 this works (same environment).

To Reproduce

I am not quite sure if this is really the problem, but if so it could be reproduced by copying a non-existing image from a public registry to docker.io.

Expected behavior

When copying from one registry to another this should not result in an unauthorized error if the required permissions have been granted.

Additional context

My uneducated guess is that as the URL https://index.docker.io/[...]/uploads/?from=bbbatscale%2Fbbbatscale%2Fbbbatscale&[...] states, it tries to use the image bbbatscale/bbbatscale/bbbatscale which most likely will be resolved to docker.io/bbbatscale/bbbatscale/bbbatscale and this image does not exist, which therefore might cause the unauthorized error.

• Output of crane version: 0.10.0
• Registry used (e.g., GCR, ECR, Quay): registry.gitlab.com and docker.io

[imjasonh]

@jonjohnsonjr sounds like another case of #1321 and a spec-non-compliant destination registry.

[hobti01]

We're having the same issue since crane 0.10.0 and this really seems like a bug or at least a regression.

• crane copy docker.io/bash:4 registry.gitlab.com/redacted/bash:4
  • works if the image tag already exists
  • fails if all the layers (and manifest?) do not exist

$ crane copy docker.io/bash:5.2-rc registry.gitlab.com/redacted/bash:5.2-rc
2022/07/06 10:24:05 Copying from docker.io/bash:5.2-rc to registry.gitlab.com/redacted/bash:5.2-rc
2022/07/06 10:24:12 existing blob: sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139
Error: failed to copy index: POST https://registry.gitlab.com/v2/redacted/bash/blobs/uploads/?from=library%2Fbash&mount=sha256%3Ac6bb82abe643f745f4efe175532578eb009b3b3a3f938fba1a1de264575f4203&origin=REDACTED: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:redacted/bash Type:repository] map[Action:push Class: Name:redacted/bash Type:repository] map[Action:pull Class: Name:library/bash Type:repository]]

@imjasonh I'm trying to parse your statement above which seems to state that the registries are non-compliant. But it seems to me that crane:0.10.0 is now attempting to use non-spec-compliant cross-origin mounting and getting a response that is unexpected and failing.

The registry spec states that the registry SHOULD (not MUST) return a 202 response if it does not understand the cross-origin mount request. In this case the burden seems to be on crane to allow for responses other than 202 (e.g. 401) for errors since the spec does not mandate that the registry MUST return a 202.

[snowbldr]

We experienced the same issue copying from gcr to gitlab registry.

To work around the issue we downgraded to 0.9.0 with no other changes, and everything resumed working again.

I couldn't find any other viable work arounds for version 0.10.0.

[matutter]

Same issue copying an image from an internal Gitlab registry to public docker.io. Downgrading to 0.9.0 worked.