🌱 Remove CI image pull and tagging for kubeadm injection script #6590

jsturtevant · 2022-06-03T00:11:14Z

What this PR does / why we need it:
After the switch from k8s.gcr.io to registry.k8s.io (see kubernetes/kubernetes#109938 for details), the CI jobs are outputting the following when trying to extract the ci images:

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/capz-conformance-master/1532465763934277632

[2022-06-02 21:09:45] 
[2022-06-02 21:09:49] unpacking registry.k8s.io/kube-apiserver-amd64:v1.25.0-alpha.0.734_ba502ee555924a (sha256:4600d5b413457521bd7c01d24c0b21288c63f05137aa7547b6e0cb5e9a654bef)...done
[2022-06-02 21:09:49] ctr: image "k8s.gcr.io/kube-apiserver-amd64:v1.25.0-alpha.0.734_ba502ee555924a": not found`

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Partially Fixes kubernetes-sigs/cluster-api-provider-azure#2351

other info

I found our jobs are not failing since kubeadm is smart enough because we give it the kubernetesversion in the job config, even without this to pull the correct images so our jobs are still passing.

See https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/9e793b42f79b65cbc830e711dce63a2c1e7c82c0/templates/test/ci/cluster-template-prow-ci-version-windows-containerd-2022.yaml#L90

Is there a reason to pull these images?

stmcginnis · 2022-06-03T16:02:30Z

/lgtm

jackfrancis · 2022-06-03T19:20:40Z

lgtm

sbueringer · 2022-06-06T06:47:02Z

Is there a reason to pull these images?

@jsturtevant We needed the wget + import + tag of those images as they were not available in a registry (at least for CAPO and CAPA too afaik).

It's interesting that for CAPZ it works with image pull, e.g. here is a successfull image pull:

https://storage.googleapis.com/kubernetes-jenkins/logs/capz-conformance-master/1532465763934277632/artifacts/clusters/capz-conf-x78nve/machines/capz-conf-x78nve-control-plane-gsp7w/containerd.log.

Jun 02 21:09:50.735341 capz-conf-x78nve-control-plane-cxrb9 containerd[1319]: time="2022-06-02T21:09:50.735271164Z" level=info msg="PullImage "gcr.io/k8s-staging-ci-images/kube-apiserver:v1.25.0-alpha.0.734_ba502ee555924a""
Jun 02 21:09:54.684557 capz-conf-x78nve-control-plane-cxrb9 containerd[1319]: time="2022-06-02T21:09:54.684487533Z" level=info msg="ImageCreate event &ImageCreate{Name:gcr.io/k8s-staging-ci-images/kube-apiserver:v1.25.0-alpha.0.734_ba502ee555924a,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"

Note: The download script fails in CAPZ but this doesn't fail the entire bootstrap process.

Looks like the CAPO tests are currently failing since the change in image names:

ctr: image "k8s.gcr.io/kube-apiserver-amd64:v1.25.0-alpha.0.729_62d9f8ba80f4cc": not found
...
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.25.0-alpha.0.729_62d9f8ba80f4cc

In general the change looks fine, but I think we have to use the old/new registry depending on the Kubernetes version, otherwise jobs using <v1.25 will fail (if there are any).

@dims Do you know why those images might be available for CAPZ but not for CAPO? (I read about the cloud-local redirects, but I don't know if that explains it)

Note: The logs above show different registries, but I"m not sure if that is the actual issue (maybe just +/- redirect), given that we have kubeadm image pulls in both cases and it works in one case but not the other:

sbueringer · 2022-06-06T06:50:58Z

cc @mdbooth (for CAPO)

dims · 2022-06-06T12:17:03Z

@ameukam @BenTheElder can you please peek?

vincepri

/approve
/lgtm

vincepri · 2022-06-06T18:52:20Z

/hold

for the above

jsturtevant · 2022-06-07T23:36:16Z

@jsturtevant We needed the wget + import + tag of those images as they were not available in a registry (at least for CAPO and CAPA too afaik).

something might be wrong in the configuration on openstack. Kubeadm in Azure is pulling: gcr.io/k8s-staging-ci-images/kube-proxy:v1.25.0-alpha.0.859_198dd7668a19c8\ where as the kubeadm in Openstack is pulling k8s.gcr.io/kube-controller-manager:v1.25.0-alpha.0.729_62d9f8ba80f4cc

notice the difference between gcr.io/k8s-staging-ci-images and k8s.gcr.io/kube-controller-manager. I don't think the CI images are pushed to k8s.gcr.io but are pushed to the gcr.io/k8s-staging-ci-images staging repo? I am not sure why kubeadm is configured to pull from there in Openstack and not Azure.

BenTheElder · 2022-06-08T01:28:42Z

Yeah,

ctr: image "k8s.gcr.io/kube-apiserver-amd64:v1.25.0-alpha.0.729_62d9f8ba80f4cc": not found

An image with a tag like this won't be in k8s.gcr.io, tags there have to be promoted in the k8s.io repo, only release tags are present.

sbueringer · 2022-06-08T05:01:45Z

Thx! Sorry should have seen that.

Okay looks like the following:

we have to download the binaries but not the images
kubeadm automatically uses the CI registry if kubeadm is a CI binary: https://github.com/kubernetes/kubernetes/blob/703f2a7b86a2340fce92b3f9dae12a30254620a3/cmd/kubeadm/app/util/config/common.go#L94-L96
In CAPZ the KCP image repo is not set so kubeadm uses the CI registry: https://storage.googleapis.com/kubernetes-jenkins/logs/capz-conformance-master/1534363744480530432/artifacts/clusters/bootstrap/resources/capz-conf-bl9jla/KubeadmControlPlane/capz-conf-bl9jla-control-plane.yaml
In CAPO the KCP image repo is pinned to k8s.gcr.io: https://storage.googleapis.com/kubernetes-jenkins/logs/periodic-cluster-api-provider-openstack-conformance-test-main-with-k8s-ci-artifacts/1534363241298268160/artifacts/clusters/bootstrap/resources/conformance-8es4na/KubeadmControlPlane/cluster-conformance-8es4na-control-plane.yaml

I would suggest the following:

let's drop the entire image download + tagging from the script because:
- we don't need it
- we can avoid adding a Kubernetes version dependent if/else to tag the old/new image name based on Kubernetes version
CAPO should not pin to the production registry

CecileRobertMichon · 2022-06-08T17:44:02Z

let's drop the entire image download + tagging from the script because:
we don't need it
we can avoid adding a Kubernetes version dependent if/else to tag the old/new image name based on Kubernetes version

+1

Kubeadm has the ability to automatically pull the CI based images if it is a CI binary itself so we don't need to pull and tag these images but we do need to get and replace the correct binaries.

sbueringer · 2022-06-08T17:52:25Z

Looping in CAPA in case they are (still/also) using this script cc @pydctw @richardcase

sbueringer · 2022-06-08T18:00:03Z

Thank you!

/lgtm

It would be good to wait until next week with merge if possible. Just to get feedback from CAPO/CAPA (one of the CAPO maintainers at least is out this week). Otherwise I'm fine with just iterating, CAPO tests are currently broken anyway so this won't make it worse :).

jsturtevant · 2022-06-09T16:15:20Z

/retitle Remove CI image pull and tagging for kubeadm injection script

fabriziopandini

As per comments in the thread above
/lgtm
/approve

k8s-ci-robot · 2022-06-13T11:37:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fabriziopandini, vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [fabriziopandini,vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fabriziopandini · 2022-06-13T11:37:57Z

/hold

fabriziopandini · 2022-06-13T11:38:51Z

It would be good to wait until next week with merge if possible. Just to get feedback from CAPO/CAPA
@mdbooth @pydctw @richardcase

sedefsavas · 2022-06-13T19:34:23Z

CAPA k8s-main conformance test is broken since kubernetes/kubernetes#109938 is merged, not sure how it works for CAPZ but we were using the pulled images here.

I will need to further test if just changing the registry here works instead of removing it.

sedefsavas · 2022-06-13T19:36:35Z

For unblocking CAPZ, how about just changing k8s.gcr.io to registry.k8s.io in this PR so that CAPZ script does not fail? @jsturtevant @CecileRobertMichon

jsturtevant · 2022-06-13T19:47:51Z

capz isn't blocked on this, we haven't taken a dependency on this e2e helper script yet but were considering it after this merged.

For unblocking CAPZ, how about just changing k8s.gcr.io to registry.k8s.io in this PR so that CAPZ script does not fail?

This is what I did in the initial version of this PR but it was discover that the logic for pull/tagging images is not required so switch to removing it #6590 (comment)

sedefsavas · 2022-06-13T19:56:03Z

This is what I did in the initial version of this PR but it was discover that the logic for pull/tagging images is not required so switch to removing it #6590 (comment)

CAPA also doesn't pin the registry to k8s.gcr.io but still it stopped working for us, need to collect the logs.

sedefsavas · 2022-06-13T22:48:01Z

@sbueringer follow up on #6590 (comment)

let's drop the entire image download + tagging from the script

Without importing and tagging part, ci kubeadm version fails to pull images. Errors I see:

root@ip-10-0-149-177:/var/snap/amazon-ssm-agent/5656# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.0-alpha.0.956+0fe0dbf3fb8cf5", GitCommit:"0fe0dbf3fb8cf501c24c87f4113a3819cb86a550", GitTreeState:"clean", BuildDate:"2022-06-13T17:30:11Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

root@ip-10-0-149-177:/var/snap/amazon-ssm-agent/5656# kubeadm init --config /run/kubeadm/kubeadm.yaml 
[init] Using Kubernetes version: v1.25.0-alpha.0.956+0fe0dbf3fb8cf5
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-apiserver:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5: output: time="2022-06-13T21:58:02Z" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5\": failed to resolve reference \"registry.k8s.io/kube-apiserver:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5\": registry.k8s.io/kube-apiserver:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5: not found"
, error: exit status 1
        [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-controller-manager:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5: output: time="2022-06-13T21:58:04Z" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \"registry.k8s.io/kube-controller-manager:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5\": failed to resolve reference \"registry.k8s.io/kube-controller-manager:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5\": registry.k8s.io/kube-controller-manager:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5: not found"
, error: exit status 1
        [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-scheduler:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5: output: time="2022-06-13T21:58:05Z" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \"registry.k8s.io/kube-scheduler:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5\": failed to resolve reference \"registry.k8s.io/kube-scheduler:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5\": registry.k8s.io/kube-scheduler:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5: not found"
, error: exit status 1
        [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-proxy:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5: output: time="2022-06-13T21:58:06Z" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \"registry.k8s.io/kube-proxy:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5\": failed to resolve reference \"registry.k8s.io/kube-proxy:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5\": registry.k8s.io/kube-proxy:v1.25.0-alpha.0.956_0fe0dbf3fb8cf5: not found"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

After changing the registry to the new one in this PR, it works. So looks like we still need to import and tag images.

I don't know how CAPZ test does not fail with this kubeadm error.

sbueringer · 2022-06-14T04:32:32Z

Looks like there is something wrong with kubeadm somehow.

It works in CAPZ because kubeadm detects the CI version and it pulls images from the CI registry automatically, e.g. gcr.io/k8s-staging-ci-images/kube-apiserver:v1.25.0-alpha.0.965_c0cc9116677732 (because of https://github.com/kubernetes/kubernetes/blob/703f2a7b86a2340fce92b3f9dae12a30254620a3/cmd/kubeadm/app/util/config/common.go#L94-L96)

Looks like in CAPA kubeadm somehow tries to pull images from the (new) production registry (registry.k8s.io).

For unblocking CAPZ, how about just changing k8s.gcr.io to registry.k8s.io in this PR so that CAPZ script does not fail?

The problem is that the script then only works with >= v1.25. The older versions are using the old registry.

sbueringer · 2022-06-14T04:42:15Z

I think I found it. CAPZ sets a ci/ version in clusterConfiguration.kubernetesVersion

kubeadm is looking for the ci prefix there, then it automatically uses the test registry/imageRepository (gcr.io/k8s-staging-ci-images)

CecileRobertMichon · 2022-06-14T14:17:59Z

Good find @sbueringer. @sedefsavas can CAPA add the ci/ prefix too? Then we don't need additional script lines to pull the images and it works for all versions.

CAPO should also do the same.

sedefsavas · 2022-06-14T16:17:47Z

Good find!! We will change it in CAPA.

/lgtm

sbueringer · 2022-06-14T18:29:51Z

Okay I think we can go ahead and merge it.

Thx everyone!

/hold cancel

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 3, 2022

k8s-ci-robot requested review from sbueringer and stmcginnis June 3, 2022 00:11

k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jun 3, 2022

jsturtevant changed the title ~~Use registry.k8s.io in kubeadm injection script~~ 🌱 Use registry.k8s.io in kubeadm injection script Jun 3, 2022

k8s-ci-robot assigned stmcginnis Jun 3, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 3, 2022

sbueringer mentioned this pull request Jun 6, 2022

CAPI capi-e2e-main-1-24-latest failing consistently since June 1st #6596

Closed

vincepri approved these changes Jun 6, 2022

View reviewed changes

k8s-ci-robot assigned vincepri Jun 6, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 6, 2022

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 6, 2022

Remove image pulling and tagging

30ba31a

Kubeadm has the ability to automatically pull the CI based images if it is a CI binary itself so we don't need to pull and tag these images but we do need to get and replace the correct binaries.

jsturtevant force-pushed the update-kubeadm-script-registry.k8s.io branch from 75aaea2 to 30ba31a Compare June 8, 2022 17:57

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 8, 2022

k8s-ci-robot assigned sbueringer Jun 8, 2022

k8s-ci-robot changed the title ~~🌱 Use registry.k8s.io in kubeadm injection script~~ Remove CI image pull and tagging for kubeadm injection script Jun 9, 2022

jsturtevant changed the title ~~Remove CI image pull and tagging for kubeadm injection script~~ 🌱 Remove CI image pull and tagging for kubeadm injection script Jun 9, 2022

jsturtevant mentioned this pull request Jun 9, 2022

Use "registry.k8s.io" for Kubernetes artifacts kubernetes-sigs/cluster-api-provider-azure#2356

Merged

3 tasks

fabriziopandini reviewed Jun 13, 2022

View reviewed changes

k8s-ci-robot assigned fabriziopandini Jun 13, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 13, 2022

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 13, 2022

richardcase mentioned this pull request Jun 13, 2022

chore: use "registry.k8s.io" for k8s images kubernetes-sigs/cluster-api-provider-aws#3521

Merged

1 task

sedefsavas mentioned this pull request Jun 13, 2022

K8s main conformance test failing kubernetes-sigs/cluster-api-provider-aws#3525

Closed

k8s-ci-robot assigned sedefsavas Jun 14, 2022

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 14, 2022

k8s-ci-robot merged commit f87f1f3 into kubernetes-sigs:main Jun 14, 2022

k8s-ci-robot added this to the v1.2 milestone Jun 14, 2022

sedefsavas mentioned this pull request Jun 14, 2022

Fix for conformance tests kubernetes-sigs/cluster-api-provider-aws#3529

Merged

4 tasks

CecileRobertMichon mentioned this pull request Jun 17, 2022

fix: tag images when using latest kubernetes-sigs/cluster-api-provider-azure#2398

Merged

3 tasks

jackfrancis mentioned this pull request Jun 23, 2022

🐛 restore image pull/tag foo for kubeadm injection script #6711

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 Remove CI image pull and tagging for kubeadm injection script #6590

🌱 Remove CI image pull and tagging for kubeadm injection script #6590

jsturtevant commented Jun 3, 2022

stmcginnis commented Jun 3, 2022

jackfrancis commented Jun 3, 2022

sbueringer commented Jun 6, 2022 •

edited

Loading

sbueringer commented Jun 6, 2022

dims commented Jun 6, 2022

vincepri left a comment

vincepri commented Jun 6, 2022

jsturtevant commented Jun 7, 2022

BenTheElder commented Jun 8, 2022

sbueringer commented Jun 8, 2022 •

edited

Loading

CecileRobertMichon commented Jun 8, 2022

sbueringer commented Jun 8, 2022 •

edited

Loading

sbueringer commented Jun 8, 2022 •

edited

Loading

jsturtevant commented Jun 9, 2022

fabriziopandini left a comment

k8s-ci-robot commented Jun 13, 2022

fabriziopandini commented Jun 13, 2022

fabriziopandini commented Jun 13, 2022

sedefsavas commented Jun 13, 2022

sedefsavas commented Jun 13, 2022

jsturtevant commented Jun 13, 2022

sedefsavas commented Jun 13, 2022

sedefsavas commented Jun 13, 2022

sbueringer commented Jun 14, 2022 •

edited

Loading

sbueringer commented Jun 14, 2022 •

edited

Loading

CecileRobertMichon commented Jun 14, 2022

sedefsavas commented Jun 14, 2022

sbueringer commented Jun 14, 2022

🌱 Remove CI image pull and tagging for kubeadm injection script #6590

🌱 Remove CI image pull and tagging for kubeadm injection script #6590

Conversation

jsturtevant commented Jun 3, 2022

other info

stmcginnis commented Jun 3, 2022

jackfrancis commented Jun 3, 2022

sbueringer commented Jun 6, 2022 • edited Loading

sbueringer commented Jun 6, 2022

dims commented Jun 6, 2022

vincepri left a comment

Choose a reason for hiding this comment

vincepri commented Jun 6, 2022

jsturtevant commented Jun 7, 2022

BenTheElder commented Jun 8, 2022

sbueringer commented Jun 8, 2022 • edited Loading

CecileRobertMichon commented Jun 8, 2022

sbueringer commented Jun 8, 2022 • edited Loading

sbueringer commented Jun 8, 2022 • edited Loading

jsturtevant commented Jun 9, 2022

fabriziopandini left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jun 13, 2022

fabriziopandini commented Jun 13, 2022

fabriziopandini commented Jun 13, 2022

sedefsavas commented Jun 13, 2022

sedefsavas commented Jun 13, 2022

jsturtevant commented Jun 13, 2022

sedefsavas commented Jun 13, 2022

sedefsavas commented Jun 13, 2022

sbueringer commented Jun 14, 2022 • edited Loading

sbueringer commented Jun 14, 2022 • edited Loading

CecileRobertMichon commented Jun 14, 2022

sedefsavas commented Jun 14, 2022

sbueringer commented Jun 14, 2022

sbueringer commented Jun 6, 2022 •

edited

Loading

sbueringer commented Jun 8, 2022 •

edited

Loading

sbueringer commented Jun 8, 2022 •

edited

Loading

sbueringer commented Jun 8, 2022 •

edited

Loading

sbueringer commented Jun 14, 2022 •

edited

Loading

sbueringer commented Jun 14, 2022 •

edited

Loading