Support Singularity media types found in OCI registry

[tri-adam]

What happened:

When running syft packages against a registry: source that contains a SIF image, a panic occurs:

$ syft packages registry:core.harbor.domain/library/alpine:latest
panic: runtime error: index out of range [0] with length 0

goroutine 91 [running]:
github.com/anchore/stereoscope/pkg/image.newLayerMetadata({{0xc00056c730, 0x47}, 0x0, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {{0x0, ...}}, ...}, ...}, ...)
        /Users/runner/go/pkg/mod/github.com/anchore/[email protected]/pkg/image/layer_metadata.go:26 +0x1dd
github.com/anchore/stereoscope/pkg/image.(*Layer).Read(_, _, {{0xc00056c730, 0x47}, 0x0, {{0x0, 0x0}, {0x0, 0x0}, {0x0, ...}, ...}, ...}, ...)
        /Users/runner/go/pkg/mod/github.com/anchore/[email protected]/pkg/image/layer.go:85 +0xc8
github.com/anchore/stereoscope/pkg/image.(*Image).Read(0xc000544380)
        /Users/runner/go/pkg/mod/github.com/anchore/[email protected]/pkg/image/image.go:204 +0x4ed
github.com/anchore/stereoscope.GetImageFromSource({0x28bda40, 0xc0000500a0}, {0x7fffc6400386, 0x28}, 0x5, {0xc0001fe030, 0x1, 0x0?})
        /Users/runner/go/pkg/mod/github.com/anchore/[email protected]/client.go:92 +0x2d5
github.com/anchore/syft/syft/source.getImageWithRetryStrategy({{0x7fffc640037d, 0x31}, {0x24ad5cd, 0xb}, 0x5, {0x7fffc6400386, 0x28}, {0x0, 0x0}, 0x1}, ...)
        /Users/runner/work/syft/syft/syft/source/source.go:160 +0x29f
github.com/anchore/syft/syft/source.generateImageSource({{0x7fffc640037d, 0x31}, {0x24ad5cd, 0xb}, 0x5, {0x7fffc6400386, 0x28}, {0x0, 0x0}, 0x1}, ...)
        /Users/runner/work/syft/syft/syft/source/source.go:126 +0x58
github.com/anchore/syft/syft/source.New({{0x7fffc640037d, 0x31}, {0x24ad5cd, 0xb}, 0x5, {0x7fffc6400386, 0x28}, {0x0, 0x0}, 0x1}, ...)
        /Users/runner/work/syft/syft/syft/source/source.go:113 +0xf8
github.com/anchore/syft/cmd/syft/cli/packages.execWorker.func1()
        /Users/runner/work/syft/syft/cmd/syft/cli/packages/packages.go:72 +0x1e5
created by github.com/anchore/syft/cmd/syft/cli/packages.execWorker
        /Users/runner/work/syft/syft/cmd/syft/cli/packages/packages.go:69 +0x125

What you expected to happen:

Expected the same output as running syft packages against the SIF image locally:

$ singularity pull image.sif oras://core.harbor.domain/library/alpine:latest
INFO:    Downloading oras image
$ syft packages image.sif 
 ✔ Parsed image            
 ✔ Cataloged packages      [14 packages]
NAME                    VERSION      TYPE 
alpine-baselayout       3.2.0-r22    apk   
alpine-baselayout-data  3.2.0-r22    apk   
alpine-keys             2.4-r1       apk   
apk-tools               2.12.9-r3    apk   
busybox                 1.35.0-r17   apk   
ca-certificates-bundle  20220614-r0  apk   
libc-utils              0.7.2-r3     apk   
libcrypto1.1            1.1.1q-r0    apk   
libssl1.1               1.1.1q-r0    apk   
musl                    1.2.3-r0     apk   
musl-utils              1.2.3-r0     apk   
scanelf                 1.3.4-r0     apk   
ssl_client              1.35.0-r17   apk   
zlib                    1.2.12-r3    apk

How to reproduce it (as minimally and precisely as possible):

Using SingularityCE, push a SIF image to a supported OCI registry (I used Harbor), and process with Syft:

singularity pull docker://alpine
singularity push alpine_latest.sif oras://core.harbor.domain/library/alpine:latest
syft packages registry:core.harbor.domain/library/alpine:latest

Anything else we need to know?:

Environment:

$ syft version
Application:        syft
Version:            0.58.0
JsonSchemaVersion:  4.0.0
BuildDate:          2022-09-29T19:59:22Z
GitCommit:          b9b13d5525df89194d332467f692bc28bc68d07f
GitDescription:     v0.58.0
Platform:           linux/amd64
GoVersion:          go1.18.6
Compiler:           gc

$ singularity --version
singularity-ce version 3.10.2

$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

[kzantow]

Hey, @tri-adam -- sorry for the delay here. Do you happen to have a working example of this issue with a public repository we could look at? We'd definitely like to get any panics fixed if this is still happening.

Additionally, I created this issue to add support for ORAS: #253

[tri-adam]

Hey, @tri-adam -- sorry for the delay here. Do you happen to have a working example of this issue with a public repository we could look at? We'd definitely like to get any panics fixed if this is still happening.

Additionally, I created this issue to add support for ORAS: #253

Hi @kzantow, apologies for the slow response. I just pushed a SIF up to Dockerhub here: https://hub.docker.com/r/sylabsadam/alpine-sif.

It looks like things have improved since my original report. syft isn't able to process the SIF from the registry, but doesn't panic now:

$ syft packages registry:sylabsadam/alpine-sif
Command "packages" is deprecated, use `syft scan` instead
 ✔ Parsed image                                                           sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
could not determine source: an error occurred attempting to resolve 'sylabsadam/alpine-sif': oci-registry: unknown layer media type: application/vnd.sylabs.sif.layer.v1.sif

Running against the SIF locally still seems to work as expected:

$ singularity pull oras://sylabsadam/alpine-sif
INFO:    Downloading oras image
3.4MiB / 3.4MiB [==============================================================================================================] 100 % 1.7 MiB/s 0s
$ syft packages singularity:alpine-sif_latest.sif
Command "packages" is deprecated, use `syft scan` instead
 ✔ Parsed image                                                           sha256:bd640a41460ae7bfae0d608ad063c84cf5b61ef1b9647531bd59cbf57e519dde
 ✔ Cataloged contents                                                            ff77047d9cb035df6be6800b79aea4df63d2fb3e46c59125b2fc2d5235062496
   ├── ✔ Packages                        [14 packages]  
   ├── ✔ File digests                    [77 files]  
   ├── ✔ File metadata                   [77 locations]  
   └── ✔ Executables                     [17 executables]  
NAME                    VERSION      TYPE   
alpine-baselayout       3.6.5-r0     apk     
alpine-baselayout-data  3.6.5-r0     apk     
alpine-keys             2.4-r1       apk     
apk-tools               2.14.4-r0    apk     
busybox                 1.36.1-r29   apk     
busybox-binsh           1.36.1-r29   apk     
ca-certificates-bundle  20240226-r0  apk     
libcrypto3              3.3.1-r0     apk     
libssl3                 3.3.1-r0     apk     
musl                    1.2.5-r0     apk     
musl-utils              1.2.5-r0     apk     
scanelf                 1.3.7-r2     apk     
ssl_client              1.36.1-r29   apk     
zlib                    1.3.1-r1     apk

[kzantow]

I think we still probably have something to do here: support non-OCI image types from registry. There are a couple ways to do that today, one of which is detecting the media type somewhere around here: https://github.com/anchore/stereoscope/blob/main/pkg/image/oci/registry_provider.go#L92, but since that is the OCI provider, it might be more correct to add another registry provider for SIF and it just needs the right registry tags (see here), but this would result in checking the registry twice when really we just need to check the media type of the initial response, and the registry is still actually an OCI registry, just not an OCI image.

After a very short deliberation, I think my preferred approach would be to move the registry provider from the oci package into a registry package, and have this delegate reading images based on the media type to the appropriate oci or sif providers. Any feedback how to implement this otherwise would be appreciated!

I've retitled this issue to better reflect the ask, let me know if I've misrepresented anything @tri-adam !

[tri-adam]

I think we still probably have something to do here: support non-OCI image types from registry. There are a couple ways to do that today, one of which is detecting the media type somewhere around here: https://github.com/anchore/stereoscope/blob/main/pkg/image/oci/registry_provider.go#L92, but since that is the OCI provider, it might be more correct to add another registry provider for SIF and it just needs the right registry tags (see here), but this would result in checking the registry twice when really we just need to check the media type of the initial response, and the registry is still actually an OCI registry, just not an OCI image.

After a very short deliberation, I think my preferred approach would be to move the registry provider from the oci package into a registry package, and have this delegate reading images based on the media type to the appropriate oci or sif providers. Any feedback how to implement this otherwise would be appreciated!

I've retitled this issue to better reflect the ask, let me know if I've misrepresented anything @tri-adam !

Yeah, agreed that there's still some work to do here...

It seems to me that the logic you suggest near https://github.com/anchore/stereoscope/blob/main/pkg/image/oci/registry_provider.go#L92 is in the right area. In the case of the SingularityCE image I listed in my last comment, the config mediaType and/or the layer mediaType could tee things off to the sif package that knows how to build a *image.Image from the contents of the remote layer.

As for where that logic should live structurally, seems to me like there are good arguments for pulling the detection logic into its own package. Would you envision that logic calling into the oci/sif packages via a NewRegistryProvider interface that accepts the remote.Image retrieved by the detection logic?