Kernelstub fails with more than one kernel line installed

[ayoungethan]

Current implementations of kernelstub on Pop!_OS fail when installing more than one kernel line, such as linux-generic and linux-lowlatency. The alternative, to install only linux-lowlatency and remove linux-generic, also fails because core elements of Pop!_OS (such as the system76-driver) have a hard dependency on linux-generic, and uninstalling the one uninstalls the other.

This is a problem, as the linux-lowlatency or similar kernel line is still important to allow Pop!_OS systems to do low latency processing, such as for digital audio workstations. This prevents Pop!_OS from being a serious consideration for low-latency applications, and thus will alienate or frustrate a core target part of the System76 target demographic: Makers. Specifically, multimedia users, but also in use cases far beyond content creators, such as IoT security: https://www.nabto.com/blog/low-latency

This discussion isn't limited to embedded systems, but also user computing systems (smartphones, tablets, and full-scale laptop and desktop computers, which need to have an "out of the box" low latency interface with the IoT in order for them to work effectively as human interface devices and hubs).

There are three possible and not-mutually-exclusive solutions:

1. Configure kernelstub to handle more than one kernel line gracefully.
2. Make the core system76 components dependent on ANY/ALL kernels that system76 provides with Pop!_OS, including -lowlatency, for example, so that installing -lowlatency and uninstalling -generic does not result in the removal of core OS software
3. Adopt a universal kernel: Explore adopting a philosophy closer to Mac OS X, which has also targeted itself very successfully at the "maker niche." (history available here: https://www.cse.unsw.edu.au/~cs9242/10/lectures/09-OSXAudiox4.pdf, briefly, Apple was dying and only had its multimedia customers while developing OS X, and so multimedia engineers This philosophy is summarized as follows: Make small sacrifices in throughput performance in system and kernel optimization to allow for relatively large gains in latency performance "out of the box."

Make linux-lowlatency or another lowlatency kernel (such as Liqorix) available to install without needing any command line configuration or breaking the system.
Option a. Problem: Currently, kernelstub is only able to handle a single kernel line without becoming a mess. While it is technically possible to install a second kernel, it destroys the functionality of the boot menu and the kernel that gets booted is whatever kernel was updated last. Solution: Make kernelstub able to handle more than one kernel gracefully; the user can choose the latest version of -generic or -lowlatency (or liquorix, etc) via menu at boot
Option b. Problem: Currently, S76 and Pop!_OS have hard dependencies on the -generic kernel line, making it impossible to uninstall the -generic kernel without breaking the system. Solution: Make the -lowlatency alternative a drop-in replacement for the -generic kernel, as an alternate dependency on core Pop!_os and S76 components that currently depend strictly on the -generic kernel.
Option c. Problem: Currently, Pop!_OS does not come with a universal kernel, but linux-generic, which is still insufficient for lowlatency audio and similar latency-dependent work. Solution: Adopt a more universal kernel line that covers most use scenarios at the sacrifice of some throughput performance.

Liquorix.net claims to maintain a kernel that is in-line with this philosophy, and I think it would be worth looking at how their kernel differs from -generic and -lowlatency to determine the extent to which System76 might supply it or a similarly-configured kernel by default in its systems to more effectively cover most usage scenarios, until the -generic kernel line can catch up to the game with the realization that latency performance deserves higher priority, even at the expense of some throughput performance, for most use cases.

This third option makes a lot of sense to me, as System76 is in a similar market niche of providing and controlling and optimizing both hardware and software to go well together. Making this adjustment will be a signficant departure from other distributions that will turn heads and gain attention, which is good marketing potential as well as more respectful of the user psychology.

[ayoungethan]

Note: if consistent <10ms lowlatency performance free of xruns are possible on S76 hardware and Pop!_OS with the -generic kernel, this bug becomes much less relevant.

https://en.wikipedia.org/wiki/Latency_%28audio%29

A latency of 10 milliseconds or better is the target for audio circuits within professional production structures.[15]
In one study listeners found latency greater than 15ms to be noticeable.[18]

My computer is in for repair but I generally test this with every upgrade cycle first, as it is by far the simplest solution!

[ayoungethan]

Here is the current workflow needed to run more than one kernel line on Pop!_OS in the upstream report:
isantop#26

[cloudishBenne]

isantop#26 (comment)

I finally created a fully working manager custom-kernel, which operates on top of, but separate from, kernelstub. Feel free to try it out and see if it fits your case!