Using OSTree with Raspberry Pi (Raspberry Pi OS)

[goldyfruit]

Hi,

I'm trying to use OSTree with my Raspberry Pi to deploy atomic system updates but so far I'm facing an issue during the ostree admin deploy.

Every time I'm trying to deploy an update I'm getting this error:

error: Failed to find kernel in /usr/lib/modules, /usr/lib/ostree-boot or /boot

I'm using debootstrap to build my image and then this image is commit into my repo.

I would like to know if it's possible to use OSTree with Raspberry Pi OS? I guess I'm missing something...
Does anyone ever tried to use OSTree with Raspberry Pi OS?

[ricardosalveti]

AFAIK the raspberry pi OS maintains the kernel at the first partition (vfat), which gets loaded right after the boot firmware, so wonder if this error is simply because the kernel is not installed at the rootfs as well.

Check for the kernel package, see if it is deploying it somewhere in the rootfs.

The other issue I see with rpi os is the lack of a standard bootloader (e.g. u-boot), so the standard ostree boot path might not work right.

If you want to use ostree but without the rpi os, recommend you to check https://github.com/advancedtelematic/meta-updater or https://github.com/foundriesio/meta-lmp (https://docs.foundries.io/latest/reference-manual/linux/linux.html).

[jlebon]

Fedora IoT also targets the Raspberry Pi: https://docs.fedoraproject.org/en-US/iot/reference-platforms/.

[goldyfruit]

@ricardosalveti The rapsberry-kernel package deploys modules into /lib/modules (as for all Debian like distributions) and kernels into /boot.

/boot/kernel.img
/boot/kernel7.img
/boot/kernel7l.img
/boot/kernel8.img

@jlebon I would to stay on Debian like (so far).

Thanks guys 👍

[ricardosalveti]

@ricardosalveti The rapsberry-kernel package deploys modules into /lib/modules (as for all Debian like distributions) and kernels into /boot.

/boot/kernel.img
/boot/kernel7.img
/boot/kernel7l.img
/boot/kernel8.img

So this is probably the issue, as I believe ostree might be looking for a more standard naming for the kernel.

From src/libostree/ostree-sysroot-deploy.c:

/* Locate kernel/initramfs in the tree; the current standard is to look in
 * /usr/lib/modules/$kver/vmlinuz first.
 *
 * Originally OSTree defined kernels to be found underneath /boot
 * in the tree.  But that means when mounting /boot at runtime
 * we end up masking the content underneath, triggering a warning.
 *
 * For that reason, and also consistency with the "/usr defines the OS" model we
 * later switched to defining the in-tree kernels to be found under
 * /usr/lib/ostree-boot. But since then, Fedora at least switched to storing the
 * kernel in /usr/lib/modules, which makes sense and isn't ostree-specific, so
 * we prefer that now. However, the default Fedora layout doesn't put the
 * initramfs there, so we need to look in /usr/lib/ostree-boot first.
 */
static gboolean
get_kernel_from_tree (int                  deployment_dfd,
                      OstreeKernelLayout **out_layout,
                      GCancellable        *cancellable,
                      GError             **error)
{

[goldyfruit]

I guess the solution will be to create few symlinks.

[wmanley]

See also: #1801

[goldyfruit]

See also: #1801

Thanks @wmanley, I found this issue but the last comment was from Jan 24, 2019.

[wmanley]

@goldyfruit: If you're up for it I can guide you through implementing #1801? I think it's a good starter issue. Specifically I mean adding support for the RPi bootloader to ostree - rather than adding a plugin system.

[goldyfruit]

@wmanley I would love the guidance for sure! 👍
My goal is to build an OSTree repository to upgrade my Pis when a commit happened on the tree, I'm using the Raspberry Pi OS 64-bit.

[wmanley]

Great.

I'm not sure what your level of experience is with these various technologies here, so do let me know if I'm stating the obvious at any time or if I should go into more detail.

I'd recommend reading https://ostreedev.github.io/ostree/atomic-upgrades/ , in particular "The bootversion" and "The /ostree/boot directory" parts are relevant. Also the ostree contributing tutorial should be useful: https://ostreedev.github.io/ostree/contributing-tutorial/

The general idea is: ostree supports multiple different bootloaders by having an interface: OstreeBootloader that several different classes implement - one for each bootloader type. There's OstreeBootloaderUboot, OstreeBootloaderGrub2, OstreeBootloaderSyslinux and OstreeBootloaderZipl. Each OstreeBootloader implementation knows what files to write into /boot in the right format such that the system bootloader will boot with the right kernel, command line arguments, device trees and initrd.

The task will be to create a new bootloader implementation that can do the same thing but for the Raspberry Pi.

Roughly speaking the process will be:

1. Get ostree building for you such that you can run the existing tests.
2. Create a new bootloader implementation by copying an existing one and renaming everything
3. Modify that implementation such that it will create and modify the right files under boot. This will be useful for that: https://www.raspberrypi.org/documentation/configuration/config-txt/boot.md . In particular I think the os_prefix setting will be the important one in config.txt.
4. Write some tests that it's behaving correctly
5. Get the changes merged into ostree upstream

We'll probably hit a few speed-bumps on the way. /boot on the RPi is a FAT partition so doesn't support symlinks which may cause a few issues. But we can solve these issues as we come across them.

[goldyfruit]

@wmanley Thanks, I've to familiarize myself will all of these.
I saw the PR (great work!).

[goldyfruit]

@wmanley Wow your change already got merged!

[starnight]

I can share how is Endless OS deployed with OSTree on RPi 4. Maybe this can give you some idea.

According to the boot folder document,

• boot files are stored on the first partition of the SD card, which is formatted with the FAT file system.
• When the Raspberry Pi is powered on, VideoCore in the SoC loads various files (start bala bala.elf and Device Tree files especially) from the boot partition/folder in order to start up the various processors, then it boots the Linux kernel.
• Linux kernel's file name must be kernel8.img on RPi 4 with 64 bits OS.

All of the mentioned above is hardware related. Then, it goes with general Linux case.

However, if it goes with full system with OSTree, then it will be a little bit complicated:

• OSTree places kernel, ramfs/initramfs and Device Tree into /boot/ostree/<hash code>/, not the boot partition/folder directly.
• FAT file system does not support symbolic, nor hard links.

So, the boot firmware file (start bala bala.elf) cannot find Linux kernel, ramfs/initramfs, neither Device Tree correctly.

What if consider the file name kernel8.img as a simple entry, and place a middle program named kernel8.img. Then, the boot firmware file (start bala bala.elf) can find kernel8.img and execute it.

Endless OS places U-Boot as the middle program, named it as kernel8.img and places it in the boot partition/folder.

• U-Boot can find other storage and partitions.
• U-Boot can load files in the partitions, including extended file systems. Operating system with OSTree can be placed in the partition with extended file system.
• U-Boot has Scripting Capabilities.

Endless OS also places a boot script boot.scr for U-Boot in the /boot/. The boot script loads a environment file /boot/uEnv.txt provided by OStree first. It includes the location of Linux kernel, ramfs/initramfs, and Device Tree provided by OStree. Then, the boot script can load all of them and boot kernel up.

With this configuration on RPi 4, Endless OS puts:

• hardware related booting procedures in the boot partition/folder.
• operating system with OSTree in another partition with ext4 file system.

This solves the booting system with OSTree issue. But, the files in the boot partition/folder cannot be updated by OSTree. Have to do it manually, if the updating is needed. Because, they are not in the scope of OSTree.

I have wrapped up all of the mentioned, including example in a slide A Journey to Boot Linux on Raspberry Pi and gave a talk in COSCUP 2020

You can download and try Endless OS image for RPi 4 https://endlessos.com/download/
Although, it is still in beta version :)

[starnight]

I'm trying to use OSTree with my Raspberry Pi to deploy atomic system updates but so far I'm facing an issue during the ostree admin deploy.

Every time I'm trying to deploy an update I'm getting this error:

error: Failed to find kernel in /usr/lib/modules, /usr/lib/ostree-boot or /boot

Please check the kernel, ramfs/initramfs and Device Tree are installed into the corresponding directory before ostree commit and ostree admin deploy.

[goldyfruit]

Thanks @starnight this is great stuff!

How do you manage the files into /boot then, like new kernel, firmwares, etc...?
What happens if someone is running the rpi-update command?

For those who would like to try to install U-Boot on RPi: https://andrei.gherzan.ro/linux/uboot-on-rpi/

[starnight]

Thanks @starnight this is great stuff!

How do you manage the files into /boot then, like new kernel, firmwares, etc...?

I guess you mean the boot partition/folder, not the /boot/ for OSTree. I prepared a storage layout in the slide A Journey to Boot Linux on Raspberry Pi's page 25.

What happens if someone is running the rpi-update command?

For those who would like to try to install U-Boot on RPi: https://andrei.gherzan.ro/linux/uboot-on-rpi/

Do you mean rpi-update before ostree commit?
If it is yes, then users have to move the files, like kernel, ramfs/initramfs, and Device Tree to directory before ostree commit to corresponding directory manually. You can prepare anther script for it, of course.
PS. I have not tried the overlay folder before. Because, Endless OS uses Linux mainline kernel, not the kernel from Raspberry Pi OS.

For the start bala.elf stuffs, also need to move them to the boot partition/folder manually. You can also prepare anther script for it, of course. These are not in the scope of OSTree as mentioned in #2223 (comment)

[wmanley]

Hi @goldyfruit: How are you getting on?

[goldyfruit]

Hi @goldyfruit: How are you getting on?

@wmanley I still looking into it, it's more complicated than I thought xD