kubeadm image download fails when upgrading only the worker node version by kubespray

[loki-shin]

What happened?

When I ran ansible playbook to upgrade the worker node, it seems that the json part of kubeadm image download or skip was not parsed properly. Even the upgrade-cluster.yaml on the control-plane node worked fine.
I wonder if it is an issue with the latest release or the latest version of Kubernetes.
What more information should I add?

kubectl get nodes (I did cordon myself on worker node.)
NAME STATUS ROLES AGE VERSION
master Ready control-plane 95d v1.29.2
master2 Ready control-plane 9h v1.29.2
master3 Ready control-plane 9h v1.29.2
worker1 Ready,SchedulingDisabled 95d v1.28.2

TASK [download : Download | Download files / images] *************************************************************************************************************************************************************************************************************************************************************************************************************************************
fatal: [worker1]: FAILED! => {"msg": "{{ skip_kubeadm_images | ternary({}, kubeadm_images) }}: {{ dict(names | map('regex_replace', '^(.*)', 'kubeadm\1') | zip( repos | zip(_tags, _groups) | map('zip', keys) | map('map', 'reverse') | map('community.general.dict') | map('combine', defaults))) | dict2items | rejectattr('key', 'in', excluded) | items2ddict }}: {{ repos | map('split', '/') | map(attribute=-1) }}: {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined.....

What did you expect to happen?

I expected the Kubernetes version of the worker node to be the same as the version of the control plane.

How can we reproduce it (as minimally and precisely as possible)?

ansible-playbook upgrade-cluster.yml --become --become-user=root -i inventory/{project}/hosts.yaml -K -e kube_version=v1.29.2 --limit "worker1"

OS

Linux master 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov 8 17:36:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Rocky Linux release 9.3 (Blue Onyx)

Version of Ansible

ansible [core 2.15.9]
python version = 3.9.18 (main, Sep 7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
jinja version = 3.1.2
libyaml = True

Version of Python

Python 3.9.18

Version of Kubespray (commit)

90b0151

Network plugin used

calico

Full inventory with variables

all:
hosts:
master1:
ansible_host: 192.168.0.181
ip: 192.168.0.181
access_ip: 192.168.0.181
master2:
ansible_host: 192.168.0.182
ip: 192.168.0.182
access_ip: 192.168.0.182
master3:
ansible_host: 192.168.0.183
ip: 192.168.0.183
access_ip: 192.168.0.183
worker1:
ansible_host: 192.168.0.184
ip: 192.168.0.184
access_ip: 192.168.0.184
children:
kube_control_plane:
hosts:
master1:
master2:
master3:
kube_node:
hosts:
worker1:
etcd:
hosts:
master1:
master2:
master3:
k8s_cluster:
children:
kube_control_plane:
kube_node:
calico_rr:
hosts: {}

Command used to invoke ansible

ansible-playbook upgrade-cluster.yml --become --become-user=root -i inventory/{project}/hosts.yaml -K -e kube_version=v1.29.2 --limit "worker1"

Output of ansible run

Wednesday 06 March 2024 21:38:31 +0900 (0:00:00.087) 0:01:05.080 *******

TASK [network_plugin/calico : Get Calico default-pool configuration] ********************************************************************************************************************************************************************************************************************************************************************************************
ok: [worker1 -> master1(192.168.0.181)]
Wednesday 06 March 2024 21:38:32 +0900 (0:00:00.674) 0:01:05.755 *******

TASK [network_plugin/calico : Set calico_pool_conf] *************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [worker1 -> master1(192.168.0.181)]
Wednesday 06 March 2024 21:38:32 +0900 (0:00:00.128) 0:01:05.884 *******

TASK [network_plugin/calico : Check if inventory match current cluster configuration] ***************************************************************************************************************************************************************************************************************************************************************************
ok: [worker1 -> master1(192.168.0.181)] => {
"changed": false,
"msg": "All assertions passed"
}
Wednesday 06 March 2024 21:38:32 +0900 (0:00:00.143) 0:01:06.027 *******
Wednesday 06 March 2024 21:38:32 +0900 (0:00:00.081) 0:01:06.109 *******
Wednesday 06 March 2024 21:38:32 +0900 (0:00:00.080) 0:01:06.189 *******
Wednesday 06 March 2024 21:38:33 +0900 (0:00:00.269) 0:01:06.459 *******

TASK [download : Prep_download | Set a few facts] ***************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [worker1]
Wednesday 06 March 2024 21:38:33 +0900 (0:00:00.106) 0:01:06.565 *******
Wednesday 06 March 2024 21:38:33 +0900 (0:00:00.067) 0:01:06.633 *******
Wednesday 06 March 2024 21:38:33 +0900 (0:00:00.067) 0:01:06.701 *******
Wednesday 06 March 2024 21:38:33 +0900 (0:00:00.061) 0:01:06.763 *******
Wednesday 06 March 2024 21:38:33 +0900 (0:00:00.059) 0:01:06.822 *******

TASK [download : Prep_download | Register docker images info] ***************************************************************************************************************************************************************************************************************************************************************************************************
ok: [worker1]
Wednesday 06 March 2024 21:38:35 +0900 (0:00:02.272) 0:01:09.095 *******

TASK [download : Prep_download | Create staging directory on remote node] ***************************************************************************************************************************************************************************************************************************************************************************************
ok: [worker1]
Wednesday 06 March 2024 21:38:36 +0900 (0:00:00.411) 0:01:09.507 *******
Wednesday 06 March 2024 21:38:36 +0900 (0:00:00.069) 0:01:09.576 *******
Wednesday 06 March 2024 21:38:36 +0900 (0:00:00.061) 0:01:09.638 *******

TASK [download : Download | Download files / images] ************************************************************************************************************************************************************************************************************************************************************************************************************
fatal: [worker1]: FAILED! => {"msg": "{{ skip_kubeadm_images | ternary({}, kubeadm_images) }}: {{ dict(names | map('regex_replace', '^(.*)', 'kubeadm\1') | zip( repos | zip(_tags, groups) | map('zip', keys) | map('map', 'reverse') | map('community.general.dict') | map('combine', defaults))) | dict2items | rejectattr('key', 'in', excluded) | items2dict }}: {{ repos | map('split', '/') | map(attribute=-1) }}: {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ repos | map('split', '/') | map(attribute=-1) }}: {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ dict(names | map('regex_replace', '^(.*)', 'kubeadm\1') | zip( repos | zip(_tags, _groups) | map('zip', keys) | map('map', 'reverse') | map('community.general.dict') | map('combine', defaults))) | dict2items | rejectattr('key', 'in', excluded) | items2dict }}: {{ repos | map('split', '/') | map(attribute=-1) }}: {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ repos | map('split', '/') | map(attribute=-1) }}: {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ skip_kubeadm_images | ternary({}, kubeadm_images) }}: {{ dict(names | map('regex_replace', '^(.*)', 'kubeadm\1') | zip( repos | zip(_tags, groups) | map('zip', keys) | map('map', 'reverse') | map('community.general.dict') | map('combine', defaults))) | dict2items | rejectattr('key', 'in', excluded) | items2dict }}: {{ repos | map('split', '/') | map(attribute=-1) }}: {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ repos | map('split', '/') | map(attribute=-1) }}: {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ dict(names | map('regex_replace', '^(.*)', 'kubeadm\1') | zip( repos | zip(_tags, _groups) | map('zip', keys) | map('map', 'reverse') | map('community.general.dict') | map('combine', defaults))) | dict2items | rejectattr('key', 'in', excluded) | items2dict }}: {{ repos | map('split', '/') | map(attribute=-1) }}: {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ repos | map('split', '/') | map(attribute=-1) }}: {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ images | map(attribute=0) }}: {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined. {{ kubeadm_images_raw.stdout_lines | map('split', ':') }}: 'kubeadm_images_raw' is undefined. 'kubeadm_images_raw' is undefined"}

NO MORE HOSTS LEFT **********************************************************************************************************************************************************************************************************************************************************************************************************************************************

PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************
worker1 : ok=120 changed=2 unreachable=0 failed=1 skipped=113 rescued=0 ignored=0

Wednesday 06 March 2024 21:38:36 +0900 (0:00:00.701) 0:01:10.339 *******

download : Prep_download | Register docker images info --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.27s
kubernetes/preinstall : Ensure kube-bench parameters are set --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.98s
bootstrap-os : Install libselinux python package --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.98s
bootstrap-os : Assign inventory name to unconfigured hostnames (non-CoreOS, non-Flatcar, Suse and ClearLinux, non-Fedora) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.70s
kubernetes/preinstall : Create kubernetes directories ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.39s
kubernetes/preinstall : Install packages requirements ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.34s
kubernetes/preinstall : Ensure ping package -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.34s
bootstrap-os : Gather host facts to get ansible_distribution_version ansible_distribution_major_version -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.30s
kubernetes/preinstall : Mask swap.target (persist swapoff) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.23s
Gather necessary facts (hardware) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.18s
kubernetes/preinstall : Set selinux policy --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.14s
bootstrap-os : Ensure bash_completion.d folder exists ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.14s
kubernetes/preinstall : Remove swapfile from /etc/fstab -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.08s
bootstrap-os : Create remote_tmp for it is used by another module ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.02s
kubernetes/preinstall : Disable fapolicyd service -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.02s
bootstrap-os : Check presence of fastestmirror.conf ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.99s
bootstrap-os : Add proxy to yum.conf or dnf.conf if http_proxy is defined -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.98s
adduser : User | Create User ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.97s
bootstrap-os : Gather host facts to get ansible_os_family ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.97s
network_plugin/calico : Check if calicoctl.sh exists ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.96s

Anything else we need to know

No response

[VannTen]

I think this is the same as #10912 (which does have not a super informative title, yes...)
/triage duplicate

[hyunwoo1123]

I got the same error... and my os is ubuntu 22.04.

I was tring to join one more worker node, so I edited hosts.yml and inventory.ini, and following command.

ansible-playbook -i inventory/test-cluster/inventory.ini --become --become-user=root scale.yml

[mk-raven]

Also got the same error with scale.yml role on ubuntu 22.04

[hyunwoo1123]

Also got the same error with scale.yml role on ubuntu 22.04

what version u use? about kubespray

[LifeMoroz]

same error from latest master 4baa2c8

[LifeMoroz]

Quick fix for version 4baa2c8

- name: Download | Get kubeadm binary and list of required images
  include_tasks: prep_kubeadm_images.yml
  when:
    - not skip_downloads
# Comment line below
#    - inventory_hostname in groups['kube_control_plane']
  tags:
    - download
    - upload

I'm not sure that this is valid, but worker node has joined successfully