FR: Support cross-origin blob mounting

[rafibarash]

This FR proposes adding a origin URL parameter, for the reason outlined below.

crane & gcrane support efficient blob copies via cross-repo blob mounting by adding the 'mount' and 'from' url parameters on certain upload requests.

A CRBM http request via crane looks like so:

POST https://us-docker.pkg.dev/v2/<dest_proj>/<dest_repo>/<dest_pkg>/blobs/uploads/?from=<src_proj>/<src_repo>/<src_pkg>&mount=<src_sha256_digest>

For storage services that own multiple registries (e.g. Google owning gcr.io, us-docker.pkg.dev, us-west1-docker.pkg.dev, etc.), encoding the source registry hostname in a url parameter allows for efficient copies from a different origin.

This would look exactly the same as CRBM, but with an added origin parameter:

POST https://us-west1-docker.pkg.dev/v2/.../blobs/uploads?from=...&mount=...&origin=gcr.io

This change would be made in the remote pkg and propagate to crane & gcrane & any other clients using remote.

Also, I can't take credit for this proposal - @jonjohnsonjr originally proposed this.

[imjasonh]

Is this supported by registries today?

This behavior isn't described in the spec: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#mounting-a-blob-from-another-repository

[rafibarash]

Hey Jason,

Cross-repo blob mounting is currently supported by registries, but this proposal is talking about Cross-origin blob mounting, which involves mounting a blob from another origin (hostname).

In the screenshot of the link you posted, I highlighted the part of the spec that says CRBM should be on the same registry.

It is true Cross-origin blob mounting isn't described in the Docker Registry API. Google is planning on supporting this optimization because each Artifact Registry region has its own registry hostname (and GCR has multiple hostnames as well), and we want to update this client to support passing in this extra origin url parameter.

Perhaps there can be an effort to standardize this approach in the future.

[imjasonh]

Cross-repo blob mounting is currently supported by registries, but this proposal is talking about Cross-origin blob mounting, which involves mounting a blob from another origin (hostname).

In the screenshot of the link you posted, I highlighted the part of the spec that says CRBM should be on the same registry.

Yes, I understand. It's specifically the new cross-registry behavior I'm asking about, since as you note that's not mentioned in the spec.

It is true Cross-origin blob mounting isn't described in the Docker Registry API. Google is planning on supporting this optimization because each Artifact Registry region has its own registry hostname (and GCR has multiple hostnames as well), and we want to update this client to support passing in this extra origin url parameter.

Is this supported in production today so I can play along and make sure this works as expected?

Perhaps there can be an effort to standardize this approach in the future.

That would be ideal. I don't mind adding compatible registry-specific behavior to this library, but ideally it would only be as a step toward proving it works and is useful, so that other registries can implement it and take advantage of it too.

[rafibarash]

Is this supported in production today so I can play along and make sure this works as expected?

It will likely be a couple weeks until it is supported in prod, but it should soon be supported in staging

ideally it would only be as a step toward proving it works and is useful, so that other registries can implement it and take advantage of it too.

I don't have any experience with standardizing something like this, so if you have any tips that would be greatly appreciated :)

[imjasonh]

It will likely be a couple weeks until it is supported in prod, but it should soon be supported in staging

That sounds fine. Until then, I'd suggest hacking on your own internal fork of this until there's an accessible registry that can be used with this code. While the code exists in the codebase with no way to take advantage of it, it can only cause bugs and problems for other folks.

I don't have any experience with standardizing something like this, so if you have any tips that would be greatly appreciated :)

Once this works in one registry implementation and we can demonstrate its value, we can propose it as an optional feature in distribution-spec, argue over wording, and hopefully eventually get it merged. But that can't really happen until we show it working somewhere first.

At that point it might also be worth pinging folks in the OCI community who operate other multi-host registries, like ECR or ACR, since they might be interested in testing out a similar optimization, or at least they'd be able to share more about why they wouldn't want to support this.

In any case, the next step is to roll this out in GCR/AR and test it in your own private fork. After that, we can restart conversation on #1322 and get it merged into this repo.

[rafibarash]

Sounds good. Thanks Jason for the comments!

[rafibarash]

Hey @imjasonh, COBM has been working in prod for a couple weeks now (for both regular GCR and redirected-GCR sources). During our tests, COBM copies have taken constant time regardless of image size (~2s per image), whereas a regular copy takes a linear amount of time based on size.

I wrote up a user guide here. Let me know what you think and if this is enough justification to merge #1322.

[rafibarash]

Oh and feel free to request comment access