crouton on x86_64 *and* arm, kernel version issue with libc updating in chroot

[clrmntn42]

Hello!

I'm having an issue updating my crouton installs.

They fail at updating libc with this error:

ERROR: Your kernel version indicates a revision number
of 255 or greater. Glibc has a number of built in
assumptions that this revision number is less than 255.
If you've built your own kernel, please make sure that any
custom version numbers are appended to the upstream

I realize this is not a crouton script issue, but an issue with the chrome os version number.

Does anyone know of a way to work around this?

Been stuck with this issue for a while, so I'd thought I'd ask here.

Any help appreciated!

uname -a output:

Linux localhost 4.4.257-19930-ga3a8dfd6a994 #1 SMP PREEMPT Wed Mar 3 21:02:41 PST 2021 aarch64 GNU/Linux

Thanks!

kernel number with a dash or some other delimiter.

Please paste the output of the following command here: sudo edit-chroot -all

Please describe your issue:

If known, describe the steps to reproduce the issue:

[DennisLfromGA]

This is also happening in xenial & bionic it appears -

• Help please. Help please. #4441
• crouton on arm, kernel version issue with libc updating in chroot · Issue crouton on x86_64 *and* arm, kernel version issue with libc updating in chroot #4429 · dnschneid/crouton
• sid installation issues on Chrome OS 90.0.4430.11 sid installation issues on Chrome OS 90.0.4430.11 #4434

I believe it's outdated distros and there may not be anything that can be done. 🤔

[brendangregg]

I hit the same issue, same kernel version. A 255 revision limit rings a bell...it's sound like the LINUX_VERSION_CODE encoding, which packs the kernel version in a uint32. It is generated by /usr/include/linux/version.h

define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))

So 257 overflows and makes this kernel 4.5.1. glibc needs a different packing scheme.

Turns out that breakage like this was known at the time of releasing 4.4.256: https://lore.kernel.org/lkml/[email protected]/ -- see that and other recent threads on lkml.

Seems that the actual check is coming from a post-install script: /var/lib/dpkg/info/libc6:i386.preinst . I don't know enough about crouton yet to know how to manually step through the process so that I can edit this file and comment out the test.

[brendangregg]

@clrmntn42 can you please change the ticket description and delete "arm"? This also affects x86_64. Maybe the summary should be "crouton fails on newer kernels (e.g., 4.4.257)".

[brendangregg]

I hacked it up. This approach is not supported or recommended -- I don't know what else it breaks, so use at your own risk. Also, be warned that this hacking will take you around an hour, and you need an extra Linux system. And, in hindsight, this is a hacky even for a hack: I think there's a better way to do this (I'll post that last). Anyway, here's what I did:

1. Begin install:

crouton -r xenial -txfce

2. Ctrl-Z after it retrieves libc, to leave the installer paused.

3. Open a new terminal tab, and:

cd /tmp/crouton.*/xenial-*/var/cache/apt/archives

There you'll find the libc6 package. E.g., libc6_2.23-0ubuntu3_amd64.deb

4. This is pretty hacky. So I copied the libc6 deb package to a USB drive and then copied it to a Linux system (see https://unix.stackexchange.com/questions/138188/easily-unpack-deb-edit-postinst-and-repack-deb#138190), and then edited the DEBIAN/preinst file and changed the test:

    if [ "$system" = "Linux" ]
    then
        # Test to make sure z < 255, in x.y.z-n form of kernel version
        # Also make sure we don't trip on x.y.zFOO-n form
        kernel_rev=$(uname -r | sed 's/\([0-9]*\.\)\{1,2\}\([0-9]*\)\(.*\)/\2/')
        if [ "$kernel_rev" -ge 255 ]
        then
            echo "ERROR: Your kernel version indicates a revision number"
            echo "of 255 or greater.  Glibc has a number of built in"
            echo "assumptions that this revision number is less than 255."
            echo "If you\'ve built your own kernel, please make sure that any"
            echo "custom version numbers are appended to the upstream"
            echo "kernel number with a dash or some other delimiter."
            echo
            exit 1
        fi

It's the "-ge 255" line. I changed it from 255 to something much bigger (999). Then repackaged the deb, and copied it back to the location on the chrome book. This is switching out the deb package file while the installer is paused.

5. fg to continue the crouton installer, and it picks up the modified libc6.

6. Then it crashes with the same error! But this is later in the install sequence, and this time it's a slightly different libc6 version (libc6_2.23-0ubuntu11.2_amd64.deb). You'll find it in this directory, or something similar:

cd /usr/local/chroots/xenial/var/cache/apt/archives

7. Go through the whole copying to a USB drive and dpkg-deb process to edit the DEBIAN/preinst again, and making a fixed version of the deb file and putting it back there.

8. Now run the update, and do the Ctrl-Z process again at a point I forget what:

crouton -r xenial -u -t xfce

A problem here is that the update overwrites the .deb file, so I had to try again a few times copying the patched version over. I'm battling a race condition.

9. For me it then crashed again since I needed a patched libc6_2.23-0ubuntu11.2_i386.deb as well. All the same process again, although this time the package system became confused and wouldn't make forward progress. I finally had to manually force the libc6 i386 install using chroot:

enter-chroot -u 0 dpkg -i /var/cache/apt/archives/libc6_2.23-0ubuntu11.2_i386.deb

Then do the update again. And it finally finished!

startxfce4

And now I'm in Xfce. On a 4.4.257 kernel.

It was only by step 9 that I figured out I could enter-chroot -u 0 to run commands: so I think this could be simplified by doing pauses and forcing dpkg -i installs at the right times.

Also, I wonder if I could have installed the Linux (Beta) feature so that I could have the dpkg-deb commands locally, and save the trip over a USB stick to another system.

I'm sure someone will whittle this down using these techniques and make the above steps look way too verbose in comparison. :-)

(BTW, I first tried something even hackier by untar-ing and editing the deb files on the chromebook with vi and tar alone, but I wasn't able to get them back in the ar format. This is hacking with just the default software on the chromebook. I suspect it's doable, I just didn't get there.)

(Also, if I had a newer BPF up and running, I'd be able to copy-out over the uname(2) return values and change the kernel versions that way. This is an ancient trick that predates eBPF.)

UPDATE: My chromebook just displayed a kernel version error and then blew up spraying fire and sparks everywhere, and ejected keys into the ceiling. Just kidding. :) As I said, use at your own risk.

[clrmntn42]

I hacked it up. This approach is not supported or recommended -- I don't know what else it breaks, so use at your own risk. Also, be warned that this hacking will take you around an hour, and you need an extra Linux system. And, in hindsight, this is a hacky even for a hack: I think there's a better way to do this (I'll post that last). Anyway, here's what I did:

1. Begin install:

crouton -r xenial -txfce

1. Ctrl-Z after it retrieves libc, to leave the installer paused.
2. Open a new terminal tab, and:

cd /tmp/crouton.*/xenial-*/var/cache/apt/archives

There you'll find the libc6 package. E.g., libc6_2.23-0ubuntu3_amd64.deb

1. This is pretty hacky. So I copied the libc6 deb package to a USB drive and then copied it to a Linux system (see https://unix.stackexchange.com/questions/138188/easily-unpack-deb-edit-postinst-and-repack-deb#138190), and then edited the DEBIAN/preinst file and changed the test:

    if [ "$system" = "Linux" ]
    then
        # Test to make sure z < 255, in x.y.z-n form of kernel version
        # Also make sure we don't trip on x.y.zFOO-n form
        kernel_rev=$(uname -r | sed 's/\([0-9]*\.\)\{1,2\}\([0-9]*\)\(.*\)/\2/')
        if [ "$kernel_rev" -ge 255 ]
        then
            echo "ERROR: Your kernel version indicates a revision number"
            echo "of 255 or greater.  Glibc has a number of built in"
            echo "assumptions that this revision number is less than 255."
            echo "If you\'ve built your own kernel, please make sure that any"
            echo "custom version numbers are appended to the upstream"
            echo "kernel number with a dash or some other delimiter."
            echo
            exit 1
        fi

It's the "-ge 255" line. I changed it from 255 to something much bigger (999). Then repackaged the deb, and copied it back to the location on the chrome book. This is switching out the deb package file while the installer is paused.

1. fg to continue the crouton installer, and it picks up the modified libc6.
2. Then it crashes with the same error! But this is later in the install sequence, and this time it's a slightly different libc6 version (libc6_2.23-0ubuntu11.2_amd64.deb). You'll find it in this directory, or something similar:

cd /usr/local/chroots/xenial/var/cache/apt/archives

1. Go through the whole copying to a USB drive and dpkg-deb process to edit the DEBIAN/preinst again, and making a fixed version of the deb file and putting it back there.
2. Now run the update, and do the Ctrl-Z process again at a point I forget what:

crouton -r xenial -u -t xfce

A problem here is that the update overwrites the .deb file, so I had to try again a few times copying the patched version over. I'm battling a race condition.

1. For me it then crashed again since I needed a patched libc6_2.23-0ubuntu11.2_i386.deb as well. All the same process again, although this time the package system became confused and wouldn't make forward progress. I finally had to manually force the libc6 i386 install using chroot:

enter-chroot -u 0 dpkg -i /var/cache/apt/archives/libc6_2.23-0ubuntu11.2_i386.deb

Then do the update again. And it finally finished!

startxfce4

And now I'm in Xfce. On a 4.4.257 kernel.

It was only by step 9 that I figured out I could enter-chroot -u 0 to run commands: so I think this could be simplified by doing pauses and forcing dpkg -i installs at the right times.

Also, I wonder if I could have installed the Linux (Beta) feature so that I could have the dpkg-deb commands locally, and save the trip over a USB stick to another system.

I'm sure someone will whittle this down using these techniques and make the above steps look way too verbose in comparison. :-)

(BTW, I first tried something even hackier by untar-ing and editing the deb files on the chromebook with vi and tar alone, but I wasn't able to get them back in the ar format. This is hacking with just the default software on the chromebook. I suspect it's doable, I just didn't get there.)

(Also, if I had a newer BPF up and running, I'd be able to copy-out over the uname(2) return values and change the kernel versions that way. This is an ancient trick that predates eBPF.)

UPDATE: My chromebook just displayed a kernel version error and then blew up spraying fire and sparks everywhere, and ejected keys into the ceiling. Just kidding. :) As I said, use at your own risk.

Hi brendangregg,

Excellent, I will try your idea out now on this RK3399 chromebook and report back! Cheers!

[clrmntn42]

if I understand the kernel thread discussions correctly, the kernel versions are updated to numbers above 255, but they hacked the kernel version to report back a lesser version number to avoid the issue on linux....for some reason the chromebook devs took the version number hack and did not incorporate the version check fix???

[dnschneid]

Apologies for being MIA (will continue to be so for a while, unfortunately). We'll probably have to do a similar hack in crouton in order to get libc installed for the first time.

Once it's installed, you can use something like this LD_PRELOAD hack to make all calls to uname cap at 255, which should allow stuff like package upgrades to progress without modification. I haven't done much testing on the code so it's possible it's insufficient, but hopefully that helps people who are trying to hack their way around this nuisance.

[pureleaves1]

The way I see it is theirs 3 thing we can do...

1.We can manually change the kernel number to something higher.

2.Wait and hope for the next Chrome upodate to fix this.

3.Revert back to a previous chrome version.

[Kmart777]

Just purchased a Chromebook Plus (520QAB-K04) looking forward to using it as a development platform. It has version 4.4.268 kernel that exceeds 255. Any idea when this issue will be reconciled with a Chrome update, or do I need to return this Chromebook and get an older version from eBay? Thx

[Kmart777]

Disregard... After reading the Wiki Chrome OS article I discovered that the intent of newer Chromebooks is to use Crostini, and not the Crouton solution which appears to have been an interim solution.