ImageContentSourcePolicy Not Redirecting Traffic to Configured Mirror #1999

bmanoj27 · 2024-08-13T13:53:19Z

bmanoj27
Aug 13, 2024

Hello,

I'm currently facing an issue with an air-gapped OpenShift 4.12 cluster where the ImageContentSourcePolicy (ICSP) is not directing traffic to the configured mirror for the registry.k8s.io/pause image. (registry.k8s.io/pause:3.1 --> utility.bastion.local:5000/pause:3.1)

Environment:
OpenShift Version: 4.12
Node OS: Red Hat Enterprise Linux CoreOS (RHCOS) version 414.92
Podman Version: 4.2.0
Air-gapped environment with local registry mirror. (same registry mentioned above utility.bastion.local:5000, added to insecure registry as well)

ICSP Configuration:


apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: operator-0
spec:
  repositoryDigestMirrors:
    - mirrors:
        - 'utility.bastion.local:5000/pause'
      source: registry.k8s.io/pause

Podman Logs from the Node:

[root@worker10 core]# podman pull --log-level=debug "registry.k8s.io/pause:3.1"
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug registry.k8s.io/pause:3.1)
DEBU[0000] Failed to decode the keys ["machine.network_backend"] from "/usr/share/containers/containers.conf".
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] NewControl(/var/lib/containers/storage/overlay): nextProjectID = 1714760573
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=xfs, projectQuotaSupported=true, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Setting parallel job count to 97
DEBU[0000] Pulling image registry.k8s.io/pause:3.1 (policy: always)
DEBU[0000] Looking up image "registry.k8s.io/pause:3.1" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "registry.k8s.io/pause:3.1" ...
DEBU[0000] Trying "registry.k8s.io/pause:3.1" ...
DEBU[0000] Trying "registry.k8s.io/pause:3.1" ...
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Attempting to pull candidate registry.k8s.io/pause:3.1 for registry.k8s.io/pause:3.1
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage]registry.k8s.io/pause:3.1"
Trying to pull registry.k8s.io/pause:3.1...
DEBU[0000] Copying source image //registry.k8s.io/pause:3.1 to destination image [overlay@/var/lib/containers/storage+/run/containers/storage]registry.k8s.io/pause:3.1
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "registry.k8s.io/pause:3.1"
DEBU[0000] No credentials matching registry.k8s.io/pause found in /run/containers/0/auth.json
DEBU[0000] No credentials matching registry.k8s.io/pause found in /root/.config/containers/auth.json
DEBU[0000] No credentials matching registry.k8s.io/pause found in /root/.docker/config.json
DEBU[0000] No credentials matching registry.k8s.io/pause found in /root/.dockercfg
DEBU[0000] No credentials for registry.k8s.io/pause found
DEBU[0000]  No signature storage configuration found for registry.k8s.io/pause:3.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.k8s.io
DEBU[0000] GET https://registry.k8s.io/v2/
DEBU[0010] Ping https://registry.k8s.io/v2/ err Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:43256->192.168.18.230:53: i/o timeout (&url.Error{Op:"Get", URL:"https://registry.k8s.io/v2/", Err:(*net.OpError)(0xc000617ea0)})
DEBU[0010] GET https://registry.k8s.io/v1/_ping
DEBU[0010] Ping https://registry.k8s.io/v1/_ping err Get "https://registry.k8s.io/v1/_ping": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: server misbehaving (&url.Error{Op:"Get", URL:"https://registry.k8s.io/v1/_ping", Err:(*net.OpError)(0xc00087c050)})
DEBU[0010] Accessing "registry.k8s.io/pause:3.1" failed: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:43256->192.168.18.230:53: i/o timeout
WARN[0010] Failed, retrying in 1s ... (1/3). Error: initializing source docker://registry.k8s.io/pause:3.1: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:43256->192.168.18.230:53: i/o timeout
DEBU[0011] Using registries.d directory /etc/containers/registries.d
DEBU[0011] Trying to access "registry.k8s.io/pause:3.1"
DEBU[0011] No credentials matching registry.k8s.io/pause found in /run/containers/0/auth.json
DEBU[0011] No credentials matching registry.k8s.io/pause found in /root/.config/containers/auth.json
DEBU[0011] No credentials matching registry.k8s.io/pause found in /root/.docker/config.json
DEBU[0011] No credentials matching registry.k8s.io/pause found in /root/.dockercfg
DEBU[0011] No credentials for registry.k8s.io/pause found
DEBU[0011]  No signature storage configuration found for registry.k8s.io/pause:3.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0011] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.k8s.io
DEBU[0011] GET https://registry.k8s.io/v2/
DEBU[0021] Ping https://registry.k8s.io/v2/ err Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:45731->192.168.18.230:53: i/o timeout (&url.Error{Op:"Get", URL:"https://registry.k8s.io/v2/", Err:(*net.OpError)(0xc000160190)})
DEBU[0021] GET https://registry.k8s.io/v1/_ping
DEBU[0021] Ping https://registry.k8s.io/v1/_ping err Get "https://registry.k8s.io/v1/_ping": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: server misbehaving (&url.Error{Op:"Get", URL:"https://registry.k8s.io/v1/_ping", Err:(*net.OpError)(0xc00087c5a0)})
DEBU[0021] Accessing "registry.k8s.io/pause:3.1" failed: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:45731->192.168.18.230:53: i/o timeout
WARN[0021] Failed, retrying in 1s ... (2/3). Error: initializing source docker://registry.k8s.io/pause:3.1: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:45731->192.168.18.230:53: i/o timeout
DEBU[0022] Using registries.d directory /etc/containers/registries.d
DEBU[0022] Trying to access "registry.k8s.io/pause:3.1"
DEBU[0022] No credentials matching registry.k8s.io/pause found in /run/containers/0/auth.json
DEBU[0022] No credentials matching registry.k8s.io/pause found in /root/.config/containers/auth.json
DEBU[0022] No credentials matching registry.k8s.io/pause found in /root/.docker/config.json
DEBU[0022] No credentials matching registry.k8s.io/pause found in /root/.dockercfg
DEBU[0022] No credentials for registry.k8s.io/pause found
DEBU[0022]  No signature storage configuration found for registry.k8s.io/pause:3.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0022] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.k8s.io
DEBU[0022] GET https://registry.k8s.io/v2/
DEBU[0032] Ping https://registry.k8s.io/v2/ err Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:46183->192.168.18.230:53: i/o timeout (&url.Error{Op:"Get", URL:"https://registry.k8s.io/v2/", Err:(*net.OpError)(0xc00065a280)})
DEBU[0032] GET https://registry.k8s.io/v1/_ping
DEBU[0032] Ping https://registry.k8s.io/v1/_ping err Get "https://registry.k8s.io/v1/_ping": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: server misbehaving (&url.Error{Op:"Get", URL:"https://registry.k8s.io/v1/_ping", Err:(*net.OpError)(0xc0001606e0)})
DEBU[0032] Accessing "registry.k8s.io/pause:3.1" failed: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:46183->192.168.18.230:53: i/o timeout
WARN[0032] Failed, retrying in 1s ... (3/3). Error: initializing source docker://registry.k8s.io/pause:3.1: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:46183->192.168.18.230:53: i/o timeout
DEBU[0033] Using registries.d directory /etc/containers/registries.d
DEBU[0033] Trying to access "registry.k8s.io/pause:3.1"
DEBU[0033] No credentials matching registry.k8s.io/pause found in /run/containers/0/auth.json
DEBU[0033] No credentials matching registry.k8s.io/pause found in /root/.config/containers/auth.json
DEBU[0033] No credentials matching registry.k8s.io/pause found in /root/.docker/config.json
DEBU[0033] No credentials matching registry.k8s.io/pause found in /root/.dockercfg
DEBU[0033] No credentials for registry.k8s.io/pause found
DEBU[0033]  No signature storage configuration found for registry.k8s.io/pause:3.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0033] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.k8s.io
DEBU[0033] GET https://registry.k8s.io/v2/
DEBU[0043] Ping https://registry.k8s.io/v2/ err Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:41558->192.168.18.230:53: i/o timeout (&url.Error{Op:"Get", URL:"https://registry.k8s.io/v2/", Err:(*net.OpError)(0xc000160b40)})
DEBU[0043] GET https://registry.k8s.io/v1/_ping
DEBU[0043] Ping https://registry.k8s.io/v1/_ping err Get "https://registry.k8s.io/v1/_ping": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: server misbehaving (&url.Error{Op:"Get", URL:"https://registry.k8s.io/v1/_ping", Err:(*net.OpError)(0xc00065a690)})
DEBU[0043] Accessing "registry.k8s.io/pause:3.1" failed: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:41558->192.168.18.230:53: i/o timeout
DEBU[0043] Error pulling candidate registry.k8s.io/pause:3.1: initializing source docker://registry.k8s.io/pause:3.1: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:41558->192.168.18.230:53: i/o timeout
Error: initializing source docker://registry.k8s.io/pause:3.1: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on 192.168.18.230:53: read udp 192.168.18.244:41558->192.168.18.230:53: i/o timeout

/etc/containers/registries.conf file in the node:

unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
short-name-mode = ""

[[registry]]
  prefix = ""
  location = "cp.icr.io/cp"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "utility.bastion.local:5000/cp"
    insecure = true

[[registry]]
  prefix = ""
  location = "registry.k8s.io/pause"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "utility.bastion.local:5000/pause"
    insecure = true

[[registry]]
  prefix = ""
  location = "utility.bastion.local:5000"
  insecure = true

This node and all other nodes are able to pull other images from the registry except for this one.
Please let me know if I need to make any changes in my ICSP?

Thanks & Regards,
Manoj.

Answered by aleskandro

Aug 13, 2024

if the following is set, you should pull by digest.

  mirror-by-digest-only = true

have you tried

podman pull registry.k8s.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea

View full answer

aleskandro · 2024-08-13T14:24:38Z

aleskandro
Aug 13, 2024
Maintainer

if the following is set, you should pull by digest.

  mirror-by-digest-only = true

have you tried

podman pull registry.k8s.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea

4 replies

bmanoj27 Aug 13, 2024
Author

Thanks for the reply, now I understand the issue.

Yes, podman pull registry.k8s.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea works now, pulled the image from the mirror registry.

What changes should I do in my ICSP now?

Thanks!

aleskandro Aug 13, 2024
Maintainer

I'm not sure if the ICSP can support mirroring by tag. We now have ImageTagMirrorSet and ImageDigestMirrorSet.... Iirc, 4.13+ though.

https://docs.okd.io/latest/rest_api/config_apis/imagetagmirrorset-config-openshift-io-v1.html

bmanoj27 Aug 13, 2024
Author

Thanks again, maybe I need to figure out how to do this in Openshift v4.12

aleskandro Aug 13, 2024
Maintainer

Can we mark the above one as 'answer' and close this issue? This is an OKD board and OKD 4.12 is not supported anymore.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ImageContentSourcePolicy Not Redirecting Traffic to Configured Mirror #1999

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

ImageContentSourcePolicy Not Redirecting Traffic to Configured Mirror #1999

bmanoj27 Aug 13, 2024

Replies: 1 comment · 4 replies

aleskandro Aug 13, 2024 Maintainer

bmanoj27 Aug 13, 2024 Author

aleskandro Aug 13, 2024 Maintainer

bmanoj27 Aug 13, 2024 Author

aleskandro Aug 13, 2024 Maintainer

bmanoj27
Aug 13, 2024

Replies: 1 comment 4 replies

aleskandro
Aug 13, 2024
Maintainer

bmanoj27 Aug 13, 2024
Author

aleskandro Aug 13, 2024
Maintainer

bmanoj27 Aug 13, 2024
Author

aleskandro Aug 13, 2024
Maintainer