nvmf-connect-nbft.service service dependencies

[tbzatek]

This is a follow-up on #2078
Cc: @mwilck

I've been trying to get nvmf-connect-nbft.service working on RHEL 9 that is based on NetworkManager. Turns out the network-online.target only fires once at boot since in the Boot-from-NBFT scenario at least one network interface (say nbft0 for illustration) is always up, thus satisfying the definition of Network is Online. (network-online.target, see https://systemd.io/NETWORK_ONLINE/).

In reality, once a secondary network interface comes up (say nbft1) no action is taken, except of NetworkManager-dispatcher.service activation that doesn't trigger any other actions/services/units.

Ultimately my questions are:

1. is the nvmf-connect-nbft.service supposed to react on any "link up" or "interface configured (DHCP)" events?
2. is this supposed to be a one-time-off service only as a secondary connect measure after switchroot?
3. I see no interface name matching, reacting to events on any network interface?

Does this work this way with wicked on your side? I'm trying to find a network management software-universal solution first.

My testing scenario looks like this:

1. boot qemu with two network interfaces up, let the UEFI connect both defined boot attempts so that the ACPI NBFT table lists two HFIs
2. while at the grub menu, set the link on a second interface down (e.g. through qemu console set_link net1 off command)
3. boot into linux, let the dracut nvmf module time out on a second connection
4. after boot, bring the link up
5. expected: NetworkManager-dispatcher.service sets the interface up and nvmf-connect-nbft.service is triggered

[mwilck]

1. is the `nvmf-connect-nbft.service` supposed to react on any "link up" or "interface configured (DHCP)" events?

No. This is what we should ultimately implement. "network online" is obviously much too coarse. I am unsure what the requirements would be though. There are no uevents for this type of event. So I guess we'd need some daemon listing for netlink events (?) and taking appropriate action.

iSCSI hasn't been able to come up with a solution to this problem in 20 years.

2. is this supposed to be a one-time-off service only as a secondary connect measure after switchroot?

Yes.

3. I see no interface name matching, reacting to events on any network interface?

No.

Does this work this way with wicked on your side? I'm trying to find a network management software-universal solution first.

Wicked includes a unit called wicked.service which is ordered before network-online.service. It's a oneshot service that runs wicked --systemd ifup all. This will try to bring up all interfaces that are configured as "mandatory". The details (like timeouts waiting for the interfaces) are configured in wicked's own config file. When this command finishes (and thus the unit is considered started), most interfaces will likely be up (although there's no guarantee).

I suppose that this is the difference wrt NM, where network-online.target is reached if just a single interface is up.

[tbzatek]

Great, thanks for your answers, it's a good starting point now that we know where do we stand.

Noting some hints to follow-up on based on the recent Timberland SIG call:

• create a systemd unit for each interface that could be used as a dependency?
• handle DHCP lease renewal with a different IP address (--> i.e. do we want to tear down the kernel connection to prevent excessive recovery attempts?)
• look how iSCSI multipathing is handled across distros

[tbzatek]

All my experiments to find an elegant systemd-only upstream-ready solution led to nowhere. Even if I manage to come up with a systemd unit for a particular network interface and watch for link changes (very nasty solution, don't ask), the interface would not have the necessary addresses set up (at least in RHEL9 w/ NetworkManager) and any calls to nvme-cli would fail anyway.

Thus I resorted for a custom NM hook that I plan to keep downstream for the time being.

Let's close this until somebody comes with a good solution.

[mwilck]

Thanks. I'd be grateful if you could publish your downstream solution, perhaps in the timberland repos somewhere?

[tbzatek]

It's just this simple hook placed in /etc/NetworkManager/dispatcher.d/nvmf-connect-nbft.sh:

#!/bin/bash

if [[ "$1" == nbft* ]] && [[ "$2" == "up" ]]; then
    systemctl start nvmf-connect-nbft.service
fi

This is still undergoing testing. There are some side effects, besides the nvmf-connect-nbft.service spawned once during boot, it's spawned again for each nbft interface on system startup. While this service is intended to be run multiple times, it brings minor delays in the boot process.