FR: loose mode allowing Docker mediaTypes

[thesayyn]

Is your feature request related to a problem? Please describe.
Right now only OCI standard mediaTypes are allowed and anything else gets rejected such as Docker V2 Manifest and ManifestList

Describe the solution you'd like
It would help a lot to have a loose mode where Docker V2 images are allowed to be stored inside the registry.

Describe alternatives you've considered
Converting docker v2 images to be have OCI mediaTypes and Config but that breaks provenance as signatures change due to mediaTypes and Config changes during conversion.

Additional context
I know the whole point of zot is to be vendor-neutral but I believe it would help a lot of people if we allowed this. Also, docker v2 images are mostly used image format in the wild.

PS: I noticed that zot does not enforce what mediaType config/blobs has as long as their descriptors use OCI standard mediaTypes.

Likely to prevent issues such as #622

[rchincha]

Describe the solution you'd like
It would help a lot to have a loose mode where Docker V2 images are allowed to be stored inside the registry.

Describe alternatives you've considered
Converting docker v2 images to be have OCI mediaTypes and Config but that breaks provenance as signatures change due to mediaTypes and Config changes during conversion.

@thesayyn thanks for taking a deep dive into zot.

This particular topic has been debated internally and our current position is that if you want to be in docker ecosystem, the best docker registry is really the "docker distribution" itself.

zot is positioned to be a good fit in the OCI ecosystem, which means one builds OCI natively [1], signs OCI [2][3], and pushes/pulls OCI to/from zot.

References:
[1] https://github.com/project-stacker/stacker
[2] https://github.com/sigstore/cosign
[3] https://github.com/notaryproject/notation

[thesayyn]

zot is positioned to be a good fit in the OCI ecosystem, which means one builds OCI natively [1], signs OCI [2][3], and pushes/pulls OCI to/from zot.

I get that but I don't see how allowing docker v2 images would break this given they are backward compatible with each other just lacking a bit of information between the two formats. Perhaps I am missing something here?

This particular topic has been debated internally and our current position is that if you want to be in the docker ecosystem, the best docker registry is really the "docker distribution" itself.

Using distribution is one option, but that wouldn't work for the use case I have in mind as it is supposed to be used with a container runtime using the official images. Also, provides only Linux binaries which I don't think going to change anytime soon.

As far as I can tell zot already supports Docker v2 if relevant lines are made conditional.

zot/pkg/storage/local.go

Line 552 in a702a23

if !IsSupportedMediaType(mediaType) {

zot/pkg/api/routes.go

Line 438 in a702a23

if !storage.IsSupportedMediaType(mediaType) {

@rchincha I'd be willing to work on this if it makes any difference.

[rchincha]

Using distribution is one option, but that wouldn't work for the use case I have in mind as it is supposed to be used with a container runtime using the official images.

@thesayyn Could you elaborate on this some more?

[thesayyn]

Using distribution is one option, but that wouldn't work for the use case I have in mind as it is supposed to be used with a container runtime using the official images.

@thesayyn Could you elaborate on this some more?

Using the distribution requires you to be on Linux as they only publish Linux images and binaries which is not feasible for my use case. I need to run a registry that is platform agnostic and can run on Linux and Darwin and optionally on windows.

[rchincha]

@thesayyn maybe fairly simple to just build docker distribution/registry binaries from source for your arch/os. Did you consider that?

[thesayyn]

Yes. This is not preferable as there is no guarantee that it wouldn’t break with a random commit given the lack of testing for the platforms that I am building. Also, requires me to publish prebuilt binaries with each version.

[thesayyn]

@rchincha Any thoughts? We'd really benefit from this as a backward compatibility feature in https://github.com/bazel-contrib/rules_oci. As far as you are concerned signing is completely unrelated to what media type is the image using. cosign does not care if you are using docker media types or oci media types

[rchincha]

@thesayyn Supporting docker format on-the-wire but converting to OCI layout is probably something we could consider. But this means signatures, etc become invalid. Would this be acceptable for your use case?

[thesayyn]

@rchincha We discussed this over at google/go-containerregistry#1293. signatures and sboms become more important each day so this implicit conversion might not be a good idea.

[rchincha]

@rchincha Any thoughts? We'd really benefit from this as a backward compatibility feature in https://github.com/bazel-contrib/rules_oci. As far as you are concerned signing is completely unrelated to what media type is the image using. cosign does not care if you are using docker media types or oci media types

For rules_oci, have you considered https://github.com/project-stacker/stacker?

[thesayyn]

@rchincha Any thoughts? We'd really benefit from this as a backward compatibility feature in https://github.com/bazel-contrib/rules_oci. As far as you are concerned signing is completely unrelated to what media type is the image using. cosign does not care if you are using docker media types or oci media types

For rules_oci, have you considered https://github.com/project-stacker/stacker?

Yes.

• AFAICT, requires a functioning runtime which we can't have.
• It only works with Linux. We need to be cross-platform. (it depends heavily on linux features such as squashfs etc.)
• I am not sure about its hermeticity/reproducibility traits.
• it does not play well with bazel.

[rchamarthy]

@thesayyn for cross-platform build you need the build system itself to run on another platform. Curious about what other platforms you need to be supported?

[thesayyn]

@thesayyn for cross-platform build you need the build system itself to run on another platform. Curious about what other platforms you need to be supported?

darwin/linux/windows. optionally other platforms such as freebsd.

[dtzar]

Is there a way to make docker build * via buildx or any other means be able to push to zot? @rchincha

I get this when trying to push a docker built image to the latest zot:

Removing intermediate container 3c7624421517
 ---> 25c5addfd02d
Warning:  One or more build-args [TEXT] were not consumed
Successfully built 25c5addfd02d
Successfully tagged localhost:5000/hello-azure-golang:9b57448bbfc9f22e3[67]
The push refers to repository [localhost:5000/hello-azure-golang]
5f58d1899b3a: Preparing
59ed5b393ba8: Preparing
9fce6bd02a21: Preparing
59ed5b393ba8: Pushed
9fce6bd02a21: Pushed
5f58d1899b3a: Pushed
manifest invalid: manifest invalid
Error: Process completed with exit code 1.

[rchincha]

Can docker cli push to OCI compliant registries by talking OCI dist-spec on the wire directly?

As a workaround, can do it in two steps.

1. Use docker/buildx to output OCI layout tar and then untar
   https://docs.docker.com/engine/reference/commandline/buildx_build/#output

2. Use skopeo to copy from copy this layout into zot

skopeo copy --format=oci --dest-tls-verify=false oci:<layout>:tag docker://localhost:5000

https://github.com/containers/skopeo

This is the reason we had to write stacker to be/do OCI-native.

[thesayyn]

bumping into this again. I'd like really like to see this supported here. a significant number of images out in the wild still use docker media types today (not v1 but v2) and this mode would allow those images to be stored inside zot. converting them on pulling seems like an option but these days lots of images ship with cosign signatures that are attached to it them via digests.
given that digests change during the conversion process, it's a bad decision from supply chain security standpoint. plus, it eliminates the chance of blobs being mounted across repositories.

I still don't see a problem allowing images with docker media types to be stored in zot behind a flag that is turned off by default.

@rchincha would you reconsider this?

[rchincha]

@thesayyn Pls. don't get us wrong, absolutely see why this would be needed.

This is likely going to a fully vertical integrated stack (HTTP APIs, layout, interaction with zot-specific extensions) etc - so basically a parallel implementation of "docker distribution"-lite.

That said, do you want to consider submitting a PR and let's see how it looks like.

[sudo-bmitch]

@rchincha

This is likely going to a fully vertical integrated stack (HTTP APIs, layout, interaction with zot-specific extensions) etc - so basically a parallel implementation of "docker distribution"-lite.

This confuses me. Tools working with the Docker media types are still pushing/pulling with the OCI distribution-spec. The only difference is the media type of the manifest, which has all the same fields as the OCI manifests (which is why conversion tools are easy to build). I haven't dug into the Zot internals in a while, but I don't believe the API would need to change for this.

[rbojan]

This is currently the most commented and 👍 open Issue in this repository. It would be really nice if you reconsider support for allowing Docker mediaTypes.

If not, it would be nice that it is documented somewhere that zot doesn't support pushing/storing Docker images. We spent some time automating the installation of zot only to find out that it doesn't support Docker.

[rchincha]

This is currently the most commented and 👍 open Issue in this repository. It would be really nice if you reconsider support for allowing Docker mediaTypes.

If not, it would be nice that it is documented somewhere that zot doesn't support pushing/storing Docker images. We spent some time automating the installation of zot only to find out that it doesn't support Docker.

https://zotregistry.io/latest/user-guides/user-guide-datapath/?#common-tasks-using-skopeo-for-oci-images

[ChristianCiach]

zot is positioned to be a good fit in the OCI ecosystem, which means one builds OCI natively [1], signs OCI [2][3], and pushes/pulls OCI to/from zot.

As it stands, Zot currently also cannot store in-toto attestations created by Cosign: sigstore/cosign#1397 (comment)

It looks like Cosign tries to store the attestations using docker-specific media-types.

[ChristianCiach]

After reading this and related issues and PRs, I want to challenge the definition of what it even means to be "vendor neutral".

It seems to be the interpretation of Zot that "vendor neutral" means to only support a very specific set of media types. This is "fine" (even though I don't fully agree with this definition for pragmatic reasons discussed here and elsewhere), but I don't think this is the only possible interpretation.

I think Zot could also be "vendor neutral" by being "vendor agnostic", as in "Zot doesn't care for the media types of the stored artifacts, except when providing additional features for specific known media types, like signatures".

(I really really want to move from Harbor to Zot, but not supporting Cosign attestations and Docker-CE is a total show-stopper).

[rchincha]

@ChristianCiach curious why cosign attestations are broken. zot already supports cosign sign/verify workflows.

[ChristianCiach]

@rchincha That's because cosign only implements (guarded behind an experimental flag) OCI support for signatures and sboms, but not for attestations. See sigstore/cosign#1397 (comment)

Unfortunately there has been absolutely no progress in cosign about this since it got very experimental OCI support for signatures and sboms earlier this year.

[ktarplee]

@rchincha Mentioned this in a comment above

That said, do you want to consider submitting a PR and let's see how it looks like.

Sounds like if we want Zot to support docker media types someone needs to open a PR. Then the maintainers could see how bad or easy it is to support Zot. Maybe it could be put behind a config option (as mentioned above, strict: true). I don't think any one cares to support docker v1 images in Zot so all we are talking about is v2 images and the only real difference is the media types and a few missing fields. But strict: false could even let the implementation ignore the subtle differences between the compatible media types. Maybe there is some bigger change that would be necessary that I am missing.

[rchincha]

A minimum viable candidate would be to get docker push working against authentication-enabled zot.

[luisdavim]

Looking at this #1385 I don't really understand the big deal around this issue, if that's all that it takes why not do it? Being vendor agnostic/independent doesn't mean blocking specific vendors.

[andaaron]

It's not just that. We would need to read the image mediatypes, and populate the data in our internal DB for features such as search and the CVE scanner to work.

Also it is not clear to me if the docker client respects the OCI distribution spec, If I remember correctly there were some issues when pushing blobs in multiple chunks.
We don't want to go down the rabbit hole of supporting clients which may not implement the OCI spec correctly.

Coding something around the spec specifically to make docker work, means the code would not be not vendor agnostic.

[luisdavim]

I see, what about some conversion midleware or proxy even, if you want to keep it separate?

[jangrewe]

This is getting ridiculous. I'm building an image with kaniko, which can produce OCI images, but as the base image (which i'm not going to rebuild) is not an OCI image, so the output is a also Docker image - which i then can't push.

Yes, you want to be OCI compliant. But The World™ isn't at that stage yet. You're shooting yourself in the foot, knee and balls by shutting out a large audience that can't just switch to 100% OCI compliance... because that's not how things work in the real world.

This is really frustrating, and checking every couple of months to see if you guy show any kind of understanding by now, but still getting the same stubbornness, doesn't help with considering Zot as a viable container registry candidate.

[rchincha]

@jangrewe does this workaround not work for you?

skopeo copy --format=oci baseimage into zot

In fact, point zot to the upstream using sync/mirroring, and this would be seamless (signatures etc notwithstanding of course)

Now, use kaniko pointing to the baseimage in zot

[aldinokemal]

https://github.com/aldinokemal/go-oci

Hey, I just set up something super simple so you guys can run push and feel like you're using Docker!

for example

docker push localhost:5000/go-whatsapp-web-multidevice:linux-arm64 \
        --insecure=true

we can do

go-oci image:push localhost:5000/go-whatsapp-web-multidevice:linux-arm64 \
        --insecure=true

Push process

Result

[rchincha]

IMHO, except for image signatures for originally docker images, there are two very viable workarounds:

1. setup zot as a sync/mirror pointing to upstream docker-only image registry
2. skopeo copy (or any other similar tool) into zot and then use

[jangrewe]

very viable workarounds

... but still not solutions to be able use Zot as a drop-in replacement which doesn't require changing dozens of CI images to include either of those tools, and then thousands of CI pipelines to use those new images.

So: No, "workarounds" are not an option for us, which means Zot isn't an option.

[rchincha]

#2714
^ no promises, but would encourage contributions/testing as we move this along

[rchincha]

This is getting ridiculous. I'm building an image with kaniko, which can produce OCI images, but as the base image (which i'm not going to rebuild) is not an OCI image, so the output is a also Docker image - which i then can't push.

GoogleContainerTools/kaniko#1836

A word of caution - it is not zot's charter to solve all the client tooling issues ^

Can folks build a zot with this PR #2714 and test pls.
Some basic docker push/pull tests are working for us.