Error "Manifest does not match provided manifest digest" when using @sha256:digest in docker-daemon

[airadier]

When running a skopeo inspect docker-daemon:repo@sha256:digest it fails with a "Manifest does not match provided manifest digest" error.

After some tests, the cause might be that the contents of the manifest are different (testing with skopeo inspect --raw for docker:// source and for docker-daemon: source. My guess in here is that for docker-daemon: source the manifest is automatically generated by Skopeo or the underlying libraries, while for docker:// source, the manifest is downloaded "as is". The docker:// version, when retrieved from the registry, is indented and formatted, so the hash differs with the hash of the locally generated manifest. See my tests:

airadier@ender:~$ docker pull alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65: Pulling from library/alpine
df20fa9351a1: Pull complete
Digest: sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
Status: Downloaded newer image for alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65

airadier@ender:~$ ./skopeo inspect docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
FATA[0000] Error parsing manifest for image: Manifest does not match provided manifest digest sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65

airadier@ender:~$ ./skopeo inspect --raw docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":1509,"digest":"sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":5844992,"digest":"sha256:50644c29ef5a27c9a40c393a73ece2479de78325cae7d762ef3cdc19bf42dd0a"}]}

airadier@ender:~$ ./skopeo inspect --raw docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 | shasum -a 256
0fe895ba713b184489c17550d67464dd67fade3f92f3f0c1904dc8df6750d609  -

airadier@ender:~$ ./skopeo inspect --raw docker://alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1509,
      "digest": "sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2797541,
         "digest": "sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c"
      }
   ]
}

airadier@ender:~$ ./skopeo inspect --raw docker://alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 | shasum -a 256
a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65  -

I would expect that it is possible to do a skopeo inspect docker-daemon:repo@sha256:digest instead of failing. So some proposals would be:

• Fix the manifest generation, so it matches the remote manifest format. I am not sure if this is in general a good idea. Is there an standard for manifest formatting?
• Skip the check that seems to be done in https://github.com/containers/image/blob/21244c96ad792ef415068dc1bc1ab82dffb68dc3/image/unparsed.go#L59, as the digest for the manifest generated on the fly won't probably match the real manifest in the remote repository.

[airadier]

The issue also applies for other commands like skopeo copy:

kopeo copy docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 oci:test.tar
FATA[0000] Error determining manifest MIME type for docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65: Manifest does not match provided manifest digest sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd6

and anything using FromUnparsedImage (

skopeo/vendor/github.com/containers/image/v5/image/sourced.go

Line 68 in 8151b89

func FromUnparsedImage(ctx context.Context, sys *types.SystemContext, unparsed *UnparsedImage) (types.Image, error) {

) which in turn uses:

https://github.com/containers/image/blob/21244c96ad792ef415068dc1bc1ab82dffb68dc3/image/unparsed.go#L59

[airadier]

Reported to containers/image in containers/image#1049

[rhatdan]

@mtrmac PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mtrmac]

(This was not actually completed, but we are tracking it in c/image.)