rawhide: ext.config.var-mount.scsi-id fails

[c4rt0]

Kola test ext.config.var-mount.scsi-id is failing in Jenkins since Feb-13-2024 with the following error:

[2024-02-14T13:09:05.272Z] --- FAIL: ext.config.var-mount.scsi-id (56.26s)
[2024-02-14T13:09:05.272Z]         harness.go:1820: mach.Start() failed: machine 82d6a3cd-5a42-4c15-9b80-3f2439561b35 entered emergency.target in initramfs

Full console log of the jenkins job can be found here.

When looking at the console.txt of the failed test we see:

[    5.964318] ignition[977]: Stage: disks
...
[    5.974301] ignition[977]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]
[    5.979497] systemd[1]: Expecting device dev-disk-by\x2did-scsi\x2d0NVME_VirtualMultipath_disk1.device - /dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1...

and further:

[51.057963] systemd[1]: dev-disk-by\x2did-scsi\x2d0NVME_VirtualMultipath_disk1.device: Job dev-disk-by\x2did-scsi\x2d0NVME_VirtualMultipath_disk1.device/start timed out.
[51.062464] ignition disks: createPartitions: op(1): [failed]   waiting for devices [/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]: device unit dev-disk-by\x2did-scsi\x2d0NVME_VirtualMultipath_disk1.device timeout
Failed to start ignition-disks.service Ignition (disks).

See 'systemctl status ignition-disks.service' for details.

Dependency failed for ignition-complete.target - Ignition Complete.

Dependency failed for initrd.target - Initrd Default Target.

[   51.072377] systemd[1]: Timed out waiting for device dev-disk-by\x2did-scsi\x2d0NVME_VirtualMultipath_disk1.device - /dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1.
[   51.076315] ignition[977]: disks failed

[   51.237327] ignition[977]: Ignition failed: create partitions failed: failed to wait on disks devs: device unit dev-disk-by\x2did-scsi\x2d0NVME_VirtualMultipath_disk1.device timeout
[   51.239411] systemd[1]: dev-disk-by\x2did-scsi\x2d0NVME_VirtualMultipath_disk1.device: Job dev-disk-by\x2did-scsi\x2d0NVME_VirtualMultipath_disk1.device/start failed with result 'timeout'.

[dustymabe]

Can you share any relevant bits from the test logs? For example, what is in the console.txt for that test?

[dustymabe]

denial/snooze for now while you investigate @c4rt0 coreos/fedora-coreos-config#2854

[c4rt0]

Thank's for the denial @dustymabe.
Here's the console.txt from this test.

[jlebon]

Nice find! I would check for changes in udev rules (/usr/lib/udev/rules.d). Probably 60-persistent-storage.rules or one of the *sg3*.rules files.

To know which package a file comes from, you can use rpm -qf /path/to/file.

[jbtrystram]

Thanks @jlebon for the hint !

the symlink that this test rely on is generated by 63-scsi-sg3_symlink.rules.

Indeed the 63-scsi-sg3_symlink.rules have changed in the 1.48 release of sg3_utls

The new version use different udev attributes to generate the symlink : https://github.com/doug-gilbert/sg3_utils/blob/2355dc4b451989291df695148cd8d8d03b3d987e/scripts/58-scsi-sg3_symlink.rules#L10

Unfortunately the new way of creating the symlink use only TLVS letters (see udevadm info output)

So i think a fix would be to attach a label to the qemu disk and use that . Example
Or maybe there is a better way to have QEMU attach the info to the device in way that comply with that rule

looking at how cosa start the VM the fix is probably there : -device virtio-scsi-pci,id=scsi_mpath10 -device scsi-hd,bus=scsi_mpath10.0,drive=mpath10,vendor=NVME,product=VirtualMultipath,wwn=13645024510734875233,serial=disk1

[jbtrystram]

Looks like we may be able to fix how cosa generates the qemu args

[dustymabe]

I guess the question we have to ask ourselves is if this is behavior we think people are relying on or was it just behavior that was convenient that we were relying on for our tests?

If it's the former then we might need to engage upstream to fix it or at least consider the regression. If it's the latter then we can safely just change our usage and move on.

[jbtrystram]

I intuitively think people may rely on this, but they have "real" disks that would expose the values in the corrects fields and the issue is more on how QEMU "fakes" the disk.
However i did a quick check on my PC and I only see E fields (i,e the same issue would occur) like a pasted above. My system is only SATA and NVME though, SCSI disk may be different I am not sure

[c4rt0]

The next step (after posting the above) was to run rawhide on this server and compare the results. Unfortunately during the cosa build process, my machine became unresponsive to the point that I could no longer produce what was intended (unrelated hardware issue).

[c4rt0]

I guess the question we have to ask ourselves is if this is behavior we think people are relying on or was it just behavior that was convenient that we were relying on for our tests?

If it's the former then we might need to engage upstream to fix it or at least consider the regression. If it's the latter then we can safely just change our usage and move on.

How to find out if it's one or the other? What would be the next step here?

Let's say we start implementing the fix, considering it was convenient ... rather quickly someone would express dissatisfaction - right?
I've been chewing all the above since yesterday and just to be transparent, at this stage I am not quite sure how to proceed.

[jlebon]

Mailing list thread for the upstream changes: https://listman.redhat.com/archives/dm-devel/2023-March/053645.html

The upstream changes seem reasonable to me but I would not at all be surprised if there are users/customers making use of the symlinks that are being removed. They would have to migrate to using the WWN-based ones, but if it's widespread enough, we (Fedora, Red Hat) might have to decide to re-enable some of those symlinks (or maybe only during upgrades e.g. via LEAPP). This will land in c10s soon also if it hasn't already and I think upgrade testing will also happen in that context at some point. I don't think there's a need to differ at the CoreOS level; we should just follow the distro.

I think ext.config.var-mount.scsi-id should be improved regardless. What it's testing is valid (verifying that scsi-* symlinks are created in the initramfs), but how it's testing it is an issue. It doesn't need to be a multipath device, I think that was just a convenient way to get a SCSI block device added to test the rules (probably based on a suggestion from me). But in fact, that hits one of the issues that the upstream change warns about (/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1 in that test is racy and can point to either /dev/sda or /dev/sdb). I think a better fix there is to support something like 5G:channel=scsi,wwn=... and have kola actually attach the disk as a SCSI device instead of virtio-blk. Then the test would verify that an Ignition config can reference the by-id/scsi-3$WWN symlink (which matches what the reporter was trying to do in https://bugzilla.redhat.com/show_bug.cgi?id=1990506).

[jbtrystram]

Thanks @jlebon for the really detailed answer !

Or maybe there is a better way to have QEMU attach the info to the device in way that comply with that rule

So i was not too far off there :)
I'll cook up something based on your suggestion

[dustymabe]

A few PRs here that take care of this issue:

• kola/qemuexec: Add a wwn option for scsis disks coreos-assembler#3772
• ext-tests/ext Fix var-mount/scsi-id failure on rawhide fedora-coreos-config#2943
• Denylist: re-enable ext/var-mount/scsi-id fedora-coreos-config#2964 (comment)