image not known error despite image being listed for arm64 container on amd64 machine #10344

sdeoras · 2021-05-15T00:23:46Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
I am cross-compiling a go-binary for arm64 arch. and then building a container image for it using --arch flag in podman build command. This workflow is being done on a amd64 machine. I might have updated podman recently and now I am running into a strange issue where I seem to be building arm64 container image fine, but I am unable to push it to the registry or save to a tarball. In particular, here is the command output:

└─ $ ▶ podman images
REPOSITORY                                    TAG              IMAGE ID         CREATED         SIZE
registry/account/project/image           0.0.2.arm64  0f778141eb6e  17 minutes ago  45.4 MB
registry/account/project/image           0.0.2.amd64  119c62288fc4  17 minutes ago  48.1 MB

I can inspect both these container images, however issue is when trying to push to registry or save them as tarballs. I am able to push amd64 based image fine, however, arm64 image is having issues:

└─ $ ▶ podman push registry/account/project/image:0.0.2.arm64
Error: registry/account/project/image:0.0.2.arm64: image not known

Just to show that amd64 image went fine:

└─ $ ▶ podman push registry/account/project/image:0.0.2.amd64
Getting image source signatures
Copying blob d5eecb78ef17 skipped: already exists  
Copying blob 417cb9b79ade [--------------------------------------] 0.0b / 0.0b
Copying config 157af62b85 [--------------------------------------] 0.0b / 1.2KiB
Writing manifest to image destination
Storing signatures

Interestingly, this used to work for me and I am not fully sure if something change on the podman in latest version or this is an issue with registry or distroless base images. I am happy to provide more details and apologize if this is not an issue with podman.

Steps to reproduce the issue:

Create a hello world go code and cross-compile for arm64 as a static binary
Copy that binary into distroless base image and build a container image using --arch=arm64 via podman build
Inspect the image and then try to push to a registry or save to a tarball

Describe the results you received:
As described above.

Describe the results you expected:
Expecting to be able to use --arch flag to build arm64 based container images from amd64 machine. As I mentioned, I have been successfully using this workflow with podman but not sure if something change very recently either with podman or with registries.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

└─ $ ▶ podman version
Version:      3.2.0-rc1
API Version:  3.2.0-rc1
Go Version:   go1.16.3
Built:        Wed May  5 14:03:46 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

└─ $ ▶ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.20.2-dev
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 12
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: dusy
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.11.17-300.fc34.x86_64
  linkmode: dynamic
  memFree: 42956730368
  memTotal: 67079872512
  ociRuntime:
    name: crun
    package: crun-0.19.1-2.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 45h 23m 6.44s (Approximately 1.88 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/sdeoras/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.2
        using FUSE kernel interface version 7.31
  graphRoot: /home/sdeoras/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 40
  runRoot: /run/user/1000/containers
  volumePath: /home/sdeoras/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.0-rc1
  Built: 1620248626
  BuiltTime: Wed May  5 14:03:46 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/amd64
  Version: 3.2.0-rc1

Package info (e.g. output of rpm -q podman or apt list podman):

└─ $ ▶ rpm -q podman
podman-3.2.0-0.1.rc1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes on latest version
No on troubleshooting guide

Additional environment details (AWS, VirtualBox, physical, etc.):
This is on my laptop running Fedora 34

The text was updated successfully, but these errors were encountered:

sdeoras · 2021-05-15T01:53:19Z

additional info with debug logs

└─ $ ▶ podman --log-level=debug push registry/account/project/image:0.0.2.arm64
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman --log-level=debug push registry/account/project/image:0.0.2.arm64) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/user/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 37             
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman --log-level=debug push registry/account/project/image:0.0.2.arm64) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/user/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 37             
DEBU[0000] Looking up image "registry/account/project/image:0.0.2.arm64" in local containers storage 
DEBU[0000] Trying "registry/account/project/image:0.0.2.arm64" ... 
DEBU[0000] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@cb0e8f217cebde00b2f4ab793e0b5387afc3100a574eb991c1c7bd42a87f0c3c" 
DEBU[0000] exporting opaque data as blob "sha256:cb0e8f217cebde00b2f4ab793e0b5387afc3100a574eb991c1c7bd42a87f0c3c" 
DEBU[0000] Trying "registry/account/project/image:0.0.2.arm64" ... 
DEBU[0000] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@cb0e8f217cebde00b2f4ab793e0b5387afc3100a574eb991c1c7bd42a87f0c3c" 
DEBU[0000] exporting opaque data as blob "sha256:cb0e8f217cebde00b2f4ab793e0b5387afc3100a574eb991c1c7bd42a87f0c3c" 
DEBU[0000] Trying "registry/account/project/image:0.0.2.arm64" ... 
DEBU[0000] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@cb0e8f217cebde00b2f4ab793e0b5387afc3100a574eb991c1c7bd42a87f0c3c" 
DEBU[0000] exporting opaque data as blob "sha256:cb0e8f217cebde00b2f4ab793e0b5387afc3100a574eb991c1c7bd42a87f0c3c" 
DEBU[0000] Looking up image "registry/account/project/image:0.0.2.arm64" in local containers storage 
DEBU[0000] Trying "registry/account/project/image:0.0.2.arm64" ... 
DEBU[0000] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@cb0e8f217cebde00b2f4ab793e0b5387afc3100a574eb991c1c7bd42a87f0c3c" 
DEBU[0000] Found image "registry/account/project/image:0.0.2.arm64" as "registry/account/project/image:0.0.2.arm64" in local containers storage 
Error: registry/account/project/image:0.0.2.arm64: image not known

sdeoras · 2021-05-15T02:05:32Z

per recommendation at #3982 (comment) i wiped ~/.local/share/containers/storage and tried again but continue to see same pattern of error, i.e., things worked for amd64 container image but not for arm64.

rhatdan · 2021-05-15T09:50:14Z

@vrothberg Do we need to add --arch support to podman push? Is this something that should be handled in libimage?

sdeoras · 2021-05-15T13:30:53Z

confirmed that things are working with this version of podman:

└─ $ ▶ ./podman version
Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.16.3
Git Commit:   51b8ddbc22cf5b10dd76dd9243924aa66ad7db39
Built:        Sat May 15 06:26:41 2021
OS/Arch:      linux/amd64

sdeoras · 2021-05-15T14:13:48Z

hi @rhatdan i am good for now with older version of podman, but wondering if this will get fixed in a future version?

mheon · 2021-05-15T19:11:17Z

I'm reopening, sounds like this got broken during the image library migration. @vrothberg PTAL

rhatdan · 2021-05-16T11:13:15Z

@sdeoras This is definitely a bug. @vrothberg PTAL

When pushing an image, make sure to ignore the platform of the image to push exactly what the user wishes to. Add a test to make sure we're not regressing in the future. To preserve previous behaviour with respect to attempting to push a manifest list, move the platform check below resolving to a manifest list. Fixes: #containers/podman/issues/10344 Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg · 2021-05-17T08:29:37Z

Opened containers/common#557. Thanks for opening the issue!

vrothberg · 2021-05-25T11:22:44Z

Closing as the bug has been fixed. Thanks again for the report!

sdeoras · 2021-05-25T23:17:58Z

thanks @vorburger , this is awesome news! 👍

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label May 15, 2021

rhatdan closed this as completed May 15, 2021

mheon reopened this May 15, 2021

vrothberg self-assigned this May 17, 2021

vrothberg added the In Progress This issue is actively being worked by the assignee, please do not work on this at this time. label May 17, 2021

vrothberg mentioned this issue May 17, 2021

libimage: push: ignore image platform containers/common#557

Merged

vrothberg closed this as completed May 25, 2021

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

image not known error despite image being listed for arm64 container on amd64 machine #10344

image not known error despite image being listed for arm64 container on amd64 machine #10344

sdeoras commented May 15, 2021

sdeoras commented May 15, 2021

sdeoras commented May 15, 2021

rhatdan commented May 15, 2021

sdeoras commented May 15, 2021

sdeoras commented May 15, 2021

mheon commented May 15, 2021

rhatdan commented May 16, 2021

vrothberg commented May 17, 2021

vrothberg commented May 25, 2021

sdeoras commented May 25, 2021

image not known error despite image being listed for arm64 container on amd64 machine #10344

image not known error despite image being listed for arm64 container on amd64 machine #10344

Comments

sdeoras commented May 15, 2021

sdeoras commented May 15, 2021

sdeoras commented May 15, 2021

rhatdan commented May 15, 2021

sdeoras commented May 15, 2021

sdeoras commented May 15, 2021

mheon commented May 15, 2021

rhatdan commented May 16, 2021

vrothberg commented May 17, 2021

vrothberg commented May 25, 2021

sdeoras commented May 25, 2021