You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using FCOS 40.20241019.3.0 to create nodes in k8s cluster in AWS (instance type c7g.16xlarge). the kubelet is unable to start up.
Please see attached logs for failure on reading the CPU information.
Reproduction steps
Launch ec2 instance in aws with instance type c7g.16xlarge
Run these following commands.
ls -lah /sys/devices/system/node/node0/cpu* | wc -l
Expected behavior
There should be 64 entries for 64 cores on the machine
Actual behavior
There is currently 16 entries only, and sometime even less.
System details
AWS (instance type c7g.16xlarge)
rpm-ostree status -b just hangs and does not return anything
Fedora CoreOS 40.20241006.3.0
Butane or Ignition config
No response
Additional information
Error logs from kubelet binary
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601584 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu0, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu0/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601611 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu1, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu1/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601628 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu10, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu10/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601641 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu11, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu11/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601656 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu12, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu12/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601668 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu13, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu13/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601681 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu14, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu14/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601693 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu15, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu15/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601705 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu2, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu2/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601718 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu3, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu3/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601730 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu4, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu4/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601742 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu5, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu5/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601754 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu6, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu6/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601767 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu7, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu7/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601779 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu8, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu8/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.601791 9258 sysinfo.go:434] Cannot read core id for /sys/devices/system/node/node0/cpu9, core_id file does not exist, err: open /sys/devices/system/node/node0/cpu9/topology/core_id: no such file or directory
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602006 9258 machine.go:65] Cannot read vendor id correctly, set empty.
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602447 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu0/topology/core_id, assuming 0 for core_id of CPU 0
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602461 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu0/topology/physical_package_id, assuming 0 physical_package_id of CPU 0
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602477 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu1/topology/core_id, assuming 0 for core_id of CPU 1
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602493 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu1/topology/physical_package_id, assuming 0 physical_package_id of CPU 1
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602512 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu10/topology/core_id, assuming 0 for core_id of CPU 10
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602524 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu10/topology/physical_package_id, assuming 0 physical_package_id of CPU 10
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602539 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu11/topology/core_id, assuming 0 for core_id of CPU 11
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602549 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu11/topology/physical_package_id, assuming 0 physical_package_id of CPU 11
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602562 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu12/topology/core_id, assuming 0 for core_id of CPU 12
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602572 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu12/topology/physical_package_id, assuming 0 physical_package_id of CPU 12
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602585 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu13/topology/core_id, assuming 0 for core_id of CPU 13
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602595 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu13/topology/physical_package_id, assuming 0 physical_package_id of CPU 13
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602608 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu14/topology/core_id, assuming 0 for core_id of CPU 14
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602618 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu14/topology/physical_package_id, assuming 0 physical_package_id of CPU 14
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602631 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu15/topology/core_id, assuming 0 for core_id of CPU 15
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602641 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu15/topology/physical_package_id, assuming 0 physical_package_id of CPU 15
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602655 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu2/topology/core_id, assuming 0 for core_id of CPU 2
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602665 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu2/topology/physical_package_id, assuming 0 physical_package_id of CPU 2
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602679 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu3/topology/core_id, assuming 0 for core_id of CPU 3
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602689 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu3/topology/physical_package_id, assuming 0 physical_package_id of CPU 3
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602703 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu4/topology/core_id, assuming 0 for core_id of CPU 4
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602714 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu4/topology/physical_package_id, assuming 0 physical_package_id of CPU 4
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602732 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu5/topology/core_id, assuming 0 for core_id of CPU 5
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602743 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu5/topology/physical_package_id, assuming 0 physical_package_id of CPU 5
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602760 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu6/topology/core_id, assuming 0 for core_id of CPU 6
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602770 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu6/topology/physical_package_id, assuming 0 physical_package_id of CPU 6
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602784 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu7/topology/core_id, assuming 0 for core_id of CPU 7
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602793 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu7/topology/physical_package_id, assuming 0 physical_package_id of CPU 7
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602807 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu8/topology/core_id, assuming 0 for core_id of CPU 8
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602817 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu8/topology/physical_package_id, assuming 0 physical_package_id of CPU 8
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602830 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu9/topology/core_id, assuming 0 for core_id of CPU 9
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.602840 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu9/topology/physical_package_id, assuming 0 physical_package_id of CPU 9
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603276 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu0/topology/physical_package_id, assuming 0 for physical_package_id of CPU 0
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603288 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu0/topology/physical_package_id, assuming 0 physical_package_id of CPU 0
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603301 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu1/topology/physical_package_id, assuming 0 for physical_package_id of CPU 1
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603312 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu1/topology/physical_package_id, assuming 0 physical_package_id of CPU 1
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603325 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu10/topology/physical_package_id, assuming 0 for physical_package_id of CPU 10
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603335 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu10/topology/physical_package_id, assuming 0 physical_package_id of CPU 10
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603348 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu11/topology/physical_package_id, assuming 0 for physical_package_id of CPU 11
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603360 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu11/topology/physical_package_id, assuming 0 physical_package_id of CPU 11
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603376 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu12/topology/physical_package_id, assuming 0 for physical_package_id of CPU 12
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603394 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu12/topology/physical_package_id, assuming 0 physical_package_id of CPU 12
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603410 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu13/topology/physical_package_id, assuming 0 for physical_package_id of CPU 13
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603420 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu13/topology/physical_package_id, assuming 0 physical_package_id of CPU 13
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603433 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu14/topology/physical_package_id, assuming 0 for physical_package_id of CPU 14
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603443 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu14/topology/physical_package_id, assuming 0 physical_package_id of CPU 14
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603458 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu15/topology/physical_package_id, assuming 0 for physical_package_id of CPU 15
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603468 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu15/topology/physical_package_id, assuming 0 physical_package_id of CPU 15
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603481 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu2/topology/physical_package_id, assuming 0 for physical_package_id of CPU 2
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603491 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu2/topology/physical_package_id, assuming 0 physical_package_id of CPU 2
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603503 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu3/topology/physical_package_id, assuming 0 for physical_package_id of CPU 3
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603513 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu3/topology/physical_package_id, assuming 0 physical_package_id of CPU 3
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603527 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu4/topology/physical_package_id, assuming 0 for physical_package_id of CPU 4
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603538 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu4/topology/physical_package_id, assuming 0 physical_package_id of CPU 4
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603552 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu5/topology/physical_package_id, assuming 0 for physical_package_id of CPU 5
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603562 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu5/topology/physical_package_id, assuming 0 physical_package_id of CPU 5
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603575 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu6/topology/physical_package_id, assuming 0 for physical_package_id of CPU 6
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603586 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu6/topology/physical_package_id, assuming 0 physical_package_id of CPU 6
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603604 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu7/topology/physical_package_id, assuming 0 for physical_package_id of CPU 7
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603615 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu7/topology/physical_package_id, assuming 0 physical_package_id of CPU 7
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603631 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu8/topology/physical_package_id, assuming 0 for physical_package_id of CPU 8
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603642 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu8/topology/physical_package_id, assuming 0 physical_package_id of CPU 8
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603654 9258 sysfs.go:512] Cannot open /sys/bus/cpu/devices/cpu9/topology/physical_package_id, assuming 0 for physical_package_id of CPU 9
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: W1107 18:03:59.603664 9258 sysfs.go:518] Cannot open /sys/bus/cpu/devices/cpu9/topology/physical_package_id, assuming 0 physical_package_id of CPU 9
....
Nov 07 18:03:59 ip-10-191-136-65 kubelet[9246]: E1107 18:03:59.640184 9258 kubelet.go:1511] "Failed to start ContainerManager" err="invalid Node Allocatable configuration. Resource \"cpu\" has an allocatable of {{1000 -3} {<nil>} DecimalSI}, capacity of {{-1000 -3} {<nil>} DecimalSI}"
Describe the bug
When using FCOS 40.20241019.3.0 to create nodes in k8s cluster in AWS (instance type c7g.16xlarge). the kubelet is unable to start up.
Please see attached logs for failure on reading the CPU information.
Reproduction steps
Expected behavior
There should be 64 entries for 64 cores on the machine
Actual behavior
There is currently 16 entries only, and sometime even less.
System details
rpm-ostree status -b
just hangs and does not return anythingButane or Ignition config
No response
Additional information
Error logs from kubelet binary
New AMIs –
Old AMIs which is working fine–
The text was updated successfully, but these errors were encountered: