RPI Boot problems on RPI3b and RPI3b+ and boot firmware 20220120 (latest stable)

[nmbath]

Deploying stable firmeware (20220120 ) onto the RPI, it fails to boot.

The RPI gives one long flash of the green LED followed by 2 very short. Nothing further happens.

If I add "uart_2ndstage=1" to config.txt the RPIs then boot.

If I then regress the firmeware to a much earlier version the machine boots. I have tested with 2019-09-24 and also 2021-02-25 and they boot fine.

With a working bootable machine (5.10.95 kernel), copy bootcode.bin, fixup* and start* from 20220120 and machine fails to boot.

Happens on both RPI3b (rev 1.2) and RPI3b+ (rev 1.3)

[pelwell]

It's working for me:

pi@raspberrypi:~ $ tail -1 /proc/cpuinfo 
Model		: Raspberry Pi 3 Model B Rev 1.2
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.92-v7+ #1514 SMP Mon Jan 17 17:36:39 GMT 2022 armv7l GNU/Linux
pi@raspberrypi:~ $ vcgencmd version
Jan 20 2022 13:58:22 
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

and:

pi@raspberrypi:~ $ tail -1 /proc/cpuinfo 
Model		: Raspberry Pi 3 Model B Plus Rev 1.3
pi@raspberrypi:~ $ vcgencmd version
Jan 20 2022 13:58:22 
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.92-v7+ #1514 SMP Mon Jan 17 17:36:39 GMT 2022 armv7l GNU/Linux

1. What is in your config.txt?
2. Confirm the vcgencmd version and uname -a output when you do get it to boot.
3. The output from the UART when it boots with uart_2ndstage=1 might also be helpful.

[nmbath]

Many Thanks,

fairly vanilla config..

kernel=u-boot.bin

[pi2]
device_tree=bcm2709-rpi-2-b.dtb

[pi3]
device_tree=bcm2710-rpi-3-b.dtb
dtoverlay=miniuart-bt
core_freq=400
core_freq_min=400

[pi3+]
device_tree=bcm2710-rpi-3-b-plus.dtb
dtoverlay=miniuart-bt
core_freq=400
core_freq_min=400

[all]
dtparam=spi=on

Linux raspberrypi2 5.10.95-v7 #1 SMP Sun Feb 13 12:15:29 UTC 2022 armv7l GNU/Linux

tail -1 /proc/cpuinfo
Model : Raspberry Pi 3 Model B Rev 1.2

root@raspberrypi2:~# ./vcgencmd version
Jan 20 2022 13:58:22
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

dmesg
[ 0.080093] raspberrypi-firmware soc:firmware: Attached to firmware from 2022-01-20T13:58:22, variant start
[ 0.090106] raspberrypi-firmware soc:firmware: Firmware hash is bd88f66f8952d34e4e0613a85c7a6d3da49e13e2

and here is the boot loader debug with uart_2ndstage=1 enabled
Raspberry Pi Bootcode
Read File: config.txt, 287
Read File: start.elf, 2964864 (bytes)
Read File: fixup.dat, 7223 (bytes)
MESS:00:00:01.167935:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.172426:0: brfs: File read: 287 bytes
MESS:00:00:01.229515:0: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:01.235681:0: HDMI0:EDID error reading EDID block 0 attempt 1
MESS:00:00:01.242018:0: HDMI0:EDID error reading EDID block 0 attempt 2
MESS:00:00:01.248355:0: HDMI0:EDID error reading EDID block 0 attempt 3
MESS:00:00:01.254692:0: HDMI0:EDID error reading EDID block 0 attempt 4
MESS:00:00:01.261028:0: HDMI0:EDID error reading EDID block 0 attempt 5
MESS:00:00:01.267365:0: HDMI0:EDID error reading EDID block 0 attempt 6
MESS:00:00:01.273702:0: HDMI0:EDID error reading EDID block 0 attempt 7
MESS:00:00:01.280039:0: HDMI0:EDID error reading EDID block 0 attempt 8
MESS:00:00:01.286375:0: HDMI0:EDID error reading EDID block 0 attempt 9
MESS:00:00:01.292470:0: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:01.300394:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.304766:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:02.129147:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:02.136705:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:02.142614:0: *** Restart logging
MESS:00:00:02.146489:0: brfs: File read: 287 bytes
MESS:00:00:02.151818:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:02.159106:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 1
MESS:00:00:02.165964:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 2
MESS:00:00:02.172821:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 3
MESS:00:00:02.179679:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 4
MESS:00:00:02.186537:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 5
MESS:00:00:02.193395:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 6
MESS:00:00:02.200252:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 7
MESS:00:00:02.207109:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 8
MESS:00:00:02.213968:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 9
MESS:00:00:02.220583:0: hdmi: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:02.226487:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:02.234279:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 1
MESS:00:00:02.241138:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 2
MESS:00:00:02.247994:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 3
MESS:00:00:02.254853:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 4
MESS:00:00:02.261711:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 5
MESS:00:00:02.268568:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 6
MESS:00:00:02.275426:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 7
MESS:00:00:02.282284:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 8
MESS:00:00:02.289141:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 9
MESS:00:00:02.295757:0: hdmi: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:02.301625:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:00:02.310117:0: HDMI0: hdmi_pixel_encoding: 162000000
MESS:00:00:02.315814:0: vec: vec_middleware_power_on: vec_base: 0x7e806000 rev-id 0x00002708 @ vec: 0x7e806100 @ 0x00000420 enc: 0x7e806060 @ 0x00000220 cgmsae: 0x7e80605c @ 0x00000000
MESS:00:00:02.343183:0: brfs: File read: /mfs/sd/bcm2710-rpi-3-b.dtb
MESS:00:00:02.347846:0: Loading 'bcm2710-rpi-3-b.dtb' to 0x100 size 0x77b7
MESS:00:00:02.368749:0: brfs: File read: 30647 bytes
MESS:00:00:02.449866:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:02.453852:0: brfs: File read: 287 bytes
MESS:00:00:02.458540:0: Failed to load overlay 'miniuart-bt'
MESS:00:00:02.463736:0: dtparam: spi=on
MESS:00:00:02.475137:0: brfs: File read: /mfs/sd/overlays/miniuart-bt.dtbo
MESS:00:00:02.492250:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:02.496222:0: Read command line from file 'cmdline.txt':
MESS:00:00:02.502108:0: 'dwc_otg.lpm_enable=0 console=serial0,115200'
MESS:00:00:03.075111:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:00:03.143914:0: brfs: File read: 44 bytes
MESS:00:00:03.178198:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:03.182073:0: Loading 'u-boot.bin' to 0x8000 size 0x704fc
MESS:00:00:03.188070:0: Device tree loaded to 0x2eff8300 (size 0x7c60)
MESS:00:00:03.196034:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:03.202282:0: uart: Baud rate change done...
MESS:00:00:03.205694:0: uart: Baud rateMMC: mmc@7e202000: 0
Loading Environment from FAT... ** No device specified **
Loading Environment from MMC... OK
In: serial
Out: serial
Err: serial
Net: No ethernet found.
Hit any key to stop autoboot: 0

and this is the SD card layout

fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 15 GB, 15931539456 bytes, 31116288 sectors
486192 cylinders, 4 heads, 16 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type
/dev/mmcblk0p1 * 64,0,1 764,2,20 8192 97875 89684 43.7M c Win95 FAT32 (LBA)
/dev/mmcblk0p2 768,0,1 1023,63,32 98304 10268375 10170072 4965M 83 Linux
/dev/mmcblk0p3 1023,63,32 1023,63,32 10268672 20537343 10268672 5014M 83 Linux
/dev/mmcblk0p4 1023,63,32 1023,63,32 20537344 31115263 10577920 5165M 83 Linux

It is a headless machine, so no HDMI connected, just a terminal session on the serial GPIO Pins. Without uart_2ndstage=1 I get one long green LED and then two very brief and faint, then nothing. Not even anything on the serial console.

I have not tracked back through all the firmeware releases, to find out which versions first stops the boot.

[pelwell]

We're using the same firmware - I get:

[    0.080129] raspberrypi-firmware soc:firmware: Attached to firmware from 2022-01-20T13:58:22, variant start
[    0.090141] raspberrypi-firmware soc:firmware: Firmware hash is bd88f66f8952d34e4e0613a85c7a6d3da49e13e2

There's nothing obviously suspicious in there. Using your config.txt, and with kernel7.img copied to u-boot.bin, it boots.

Can you dd up your MBR and boot partition and upload it somewhere, so I can flash it to a spare card? Something like:

$ sudo dd if=/dev/mmcblk0 of=boot.img bs=512 count=97876

[nmbath]

Phil,

Many thanks. Can be found here https://www.dropbox.com/s/hps3x9m0yfncdqc/image?dl=0

It is very strange as the image used for this boot has been used by 3 people, everyone has failed. All of us as soon as we either set the boot debug or downgrade the boot firmware have our device boot.

[pelwell]

I'm not saying there is no problem, but the kernel is definitely entered. If I substitute kernel7.img for u-boot.bin I see the 4 raspberries, and with uart_enable=1 you can see it gets as far as it can without a root filing system. And looking at bus activity with a debugger shows that u-boot is polling the UART flags register.

[nmbath]

Phil, It is defiantly stumping me, and very strange when uart_2ndstage=1 is enabled boot happens. Any further ideas on where to look?

[pelwell]

I'm going to look at the state of the clocks next.

[pelwell]

The clocks match, and so do the GPIO settings, but what differs is the configuration of UART0.

The firmware can drive either UART - UART0/ttyAMA0 or UART1/ttyS0 - but it only initiliases one, and only if uart_enable=1. The 2nd-stage loader (bootcode.bin on an Pi 3) only uses UART0, and if uart_2ndstage=1.

In the failing case, both uart_enable and uart_2ndstage are zero, so UART0 is uninitialised when the "kernel" is started. However, if I manually configure it using the debugger, copying the values from a uart_2ndstage=1 run, U-boot springs into life. First the tail-end of a uart-2ndstage=1 run:

MESS:00:00:03.289922:0: uart: Baud rate change done...
MESS:00:00:03.293333:0: uart: Baud rateMMC:   mmc@7e202000: 0
Loading Environment from FAT... ** No device specified **
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  0
** Unrecognized filesystem type **

Then after removing uart_2ndstage=1, booting, attaching the debugger and poking the UART0 registers:

U-Boot> MC:   mmc@7e202000: 0
Loading Environment from FAT... ** No device specified **
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  0
** Unrecognized filesystem type **
U-Boot>
U-Boot> help
?         - alias for 'help'
base      - print or set address offset
bdinfo    - print Board Info structure
...

It seems that U-boot is relying on the UART being initialised. The UART flags register shows that before the manual configuration the TX FIFO is full, which would cause U-boot to stall. I don't understand why this used to work with older firmware, but this is how I would expect the firmware (including the bootloader) to behave.

[nmbath]

Once user replaced miniuart-bt-overlay.dtb with a version from elsewhere named miniuart-bt.dtbo and booting started. I have not yet tested myself.

It would seem from your investigation and the input from one of the users, that maybe the miniuart-bt-overlay.dtb is not initialising things correctly.

[nmbath]

It seems to be a file naming issue... I have just

cp miniuart-bt-overlay.dtb miniuart-bt.dtbo

and the image boots.... maybe something has changed in a boot firmeware at some stage around mapping between entry in config.txt and file name.

At least I can look at our openembedded build system and get this working.

Many Thanks

[pelwell]

Wow - I didn't even spot the Failed to load overlay 'miniuart-bt' in the firmware logging. That would account for the behaviour, since the firmware would not see the serial0 alias being moved, and would initialise the wrong UART.

Support for the original "-overlay.dtb" naming (deprecated since 2015) was finally withdrawn in April 2020 when the overlay_map.dtb file was introduced - it simplified the code to do so.

[nmbath]

Great to know route cause and why going back to pre April 2020 worked.. I guess with all my regression of boot firmeware I lost track of which did and didn't work. I have just retested and the 2019 version worked, anything I tried in late 2020 and 2021 failed.

Although the "Failed to load overlay 'miniuart-bt'" was in there, two lines later is what looks like a success with the minuart-bt.dtbo being read!!

MESS:00:00:02.458540:0: Failed to load overlay 'miniuart-bt'
MESS:00:00:02.463736:0: dtparam: spi=on
MESS:00:00:02.475137:0: brfs: File read: /mfs/sd/overlays/miniuart-bt.dtbo

Many thanks for your help and consider this closed.