Talos 1.6.2 boot kernel panic on RPi 4

[zargony]

Bug Report

Talos Linux 1.6.2 doesn't boot up on Raspberry Pi 4B

Description

I upgraded one of my home lab Raspberry Pi 4B nodes from 1.6.1 to 1.6.2 using talosctl -n xxx upgrade --image ghcr.io/siderolabs/installer:v1.6.2. After a while, it rebooted but didn't come back online (not even a ping reply). After attaching a monitor, I could see the kernel starting to boot but suddenly reboot in the middle of it. I wasn't able to get logs of it, but therefore recorded a video. It's scrolling too fast to recognize much, but in one frame, you can see Oops: 0000000096000046 [#1] SMP.

I was able to reproduce this by repeatedly writing metal-rpi_generic-arm64.raw.xz of either 1.6.2 or 1.6.1 to the SD card and trying to boot. 1.6.1 boots up fine while 1.6.2 got stuck in a reboot loop.

Logs

n/a because kernel panics before network comes up. Here's a video of it.

Environment

• Talos version: [talosctl version --nodes <problematic nodes>]

Client:
	Tag:         v1.6.2
	SHA:         26eee755
	Built:
	Go version:  go1.21.6 X:loopvar
	OS/Arch:     darwin/arm64
Server:
	NODE:        rigani
	Tag:         v1.6.1
	SHA:         0af17af3
	Built:
	Go version:  go1.21.5 X:loopvar
	OS/Arch:     linux/arm64
	Enabled:     RBAC

• Kubernetes version: [kubectl version --short]

Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1

• Platform:

Raspberry Pi 4B 8GB

[smira]

Thanks for reporting it, there will be Talos 1.6.3 with a newer Linux upstream version, let's check there.

[bhuism]

Compute module 4 also does not boot (the cm4 hardware is equivalent to the rpi4)

[nberlee]

it is the sensor module added, I paused at the right moment

[smira]

thanks, we'll remove it for arm64!

[zargony]

Not sure if this can be concluded from the screen above. It's in the middle of scrolling as you can see by various artefacts, so I think the cause might also be something else. I'm trying to get a better capture of the boot process (it should be possible to add panic=60 to the kernel cmdline, but somehow by keyboard doesn't work with the booting Pi right now).

[nberlee]

I've got 1.6.2 working on the pi with this workaround:
Add before upgrading the pi to the machineconfig:

machine:
   install:
     extraKernelArgs:
       - initcall_blacklist=sensors_nct6683_init

@zargony
Its not about whats not readable in the screenshot, but what is. And there is an init call to a kernel module in the stacktrace which was added in 1.6.2

[zargony]

• initcall_blacklist=sensors_nct6683_init

Works on my Pi as well. Cool, thanks a lot!