systemd-udev-settle.service is overused #73095

lheckemann · 2019-11-09T07:24:17Z

Describe the bug
Related: #53446

Using systemd-udev-settle "is not recommended" (man 8 systemd-udev-settle.service). Nevertheless, we have quite a few services depend on it:

$ rg -l systemd-udev-settle
nixos/tests/misc.nix
nixos/modules/hardware/ksm.nix
nixos/modules/security/lock-kernel-modules.nix
nixos/modules/virtualisation/openvswitch.nix
nixos/modules/virtualisation/libvirtd.nix
nixos/modules/virtualisation/lxd.nix
nixos/modules/virtualisation/anbox.nix
nixos/modules/tasks/kbd.nix
nixos/modules/services/x11/xserver.nix
nixos/modules/services/hardware/tcsd.nix
nixos/modules/services/hardware/brltty.nix
nixos/modules/services/networking/dhcpcd.nix
nixos/modules/services/scheduling/atd.nix
nixos/modules/services/hardware/trezord.nix
nixos/modules/services/hardware/acpid.nix
nixos/modules/system/boot/networkd.nix
nixos/modules/system/boot/systemd.nix
nixos/modules/tasks/filesystems/zfs.nix

On some systems/kernels (e.g. mobile-nixos kernels, cc @samueldr @kirelagin), this can cause boot to wait for systemd-udev-settle to time out, which is no fun.

Expected behavior
Services depend only on the devices that they need, either by waiting for them themselves or by depending on systemd's relevant .device unit.

Metadata

system: "aarch64-linux"
host os: Linux 3.18.71, NixOS, 19.09.941.27a5ddcf747 (Loris)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3
channels(root): "nixos-19.09.941.27a5ddcf747"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

The text was updated successfully, but these errors were encountered:

worldofpeace · 2019-11-09T21:39:43Z

@lheckemann Should we merge #25311 for display-manager?
TBH, I've been running a config to disable it for a while now.

bqv · 2020-04-04T13:56:23Z

As of now...

Removed from:

nixos/modules/services/scheduling/atd.nix    
nixos/modules/services/x11/xserver.nix
nixos/modules/tasks/kbd.nix

Added to:

nixos/modules/config/console.nix

worldofpeace · 2020-04-04T21:33:56Z

@bqv Thanks. Is the one for xserver.nix in 20.03?

bqv · 2020-04-04T21:51:01Z

@worldofpeace It is not, only the removal from tasks/kbd.nix and the addition is

jtojnar · 2020-04-05T23:55:34Z

Had to re-add it to GDM: f74f2f3

lheckemann · 2020-04-06T06:25:47Z

@jtojnar maybe gdm can depend on any DRI device instead of the global systemd-udev-settle.service? This would probably require some udev rules, but be a lot more elegant IMHO.

jtojnar · 2020-04-06T06:31:18Z

But display-manager.service should already run After systemd-logind.service, which contains:

ExecStartPre=-/nix/store/ijka1p5ndsjq60frdhlq5il0s9mgyypj-kmod-26/sbin/modprobe -abq drm

I would expect that to work, unless we are affected by systemd/systemd#14322.

lheckemann · 2020-04-06T06:54:34Z

But the kernel module being loaded doesn't necessarily imply the device being available, does it?

asbachb · 2020-04-23T00:41:53Z

It's possible to remove it from dhcpdc.service? Arch just relies on network.target (https://git.archlinux.org/svntogit/packages.git/tree/trunk/dhcpcd.service?h=packages/dhcpcd)

Looking at the commit does not explain why it was added (4c6171c) @srhb

srhb · 2020-04-24T05:06:06Z

Hi @asbachb . Sorry, I can't remember that far back, but there's some notes in the PR which brought in the commit, maybe it's helpful. It looks like I wasn't strictly happy with it even back then. :)

#45421

stale · 2020-10-21T16:25:44Z

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

lheckemann · 2020-10-22T08:46:19Z

Still valid…

rnhmjoj · 2020-12-22T08:54:20Z

I think udev-settle can be removed from dhcpcd: it was introduced to compesate for #44524, but dhcpcd is not ordered Before network.target anymore.
I tried removing the udev-settle dependency and running

nix-build -A nixosTests.predictable-interface-names --check --keep-failed

in a loop and it's always succeeding.

EDIT:

It seems #79532 is the reason for the success. So yes, I think the (terrible) udev-settle hack can be safely removed from networkd and dhcpcd.

xaverdh · 2020-12-22T09:48:40Z

So after #79532, does networkd still need udev-settle (was introduced in #39340)?

rnhmjoj · 2020-12-22T10:16:26Z

Btw, I think it's safe to say the issue now is "systemd-udev-settle" is used.
This is what systemd will print on every boot starting from the next NixOS release:

Usage of the systemd service unit systemd-udev-settle.service is deprecated. It
inserts artificial delays into the boot process without providing the
guarantees other subsystems traditionally assumed it provides. Relying on this
service is racy, and it is generally a bug to make use of it and depend on it.

Traditionally, this service's job was to wait until all devices a system
possesses have been fully probed and initialized, delaying boot until this
phase is completed. However, today's systems and hardware generally don't work
this way anymore, hardware today may show up any time and take any time to be
probed and initialized. Thus, in the general case, it's no longer possible to
correctly delay boot until "all devices" have been processed, as it is not
clear what "all devices" means and when they have been found. This is in
particular the case if USB hardware or network-attached hardware is used.

Modern software that requires some specific hardware (such as a network device
or block device) to operate should only wait for the specific devices it needs
to show up, and otherwise operate asynchronously initializing devices as they
appear during boot and during runtime without delaying the boot process.

It is a defect of the software in question if it doesn't work this way, and
still pulls systemd-udev-settle.service into the boot process.

Please file a bug report against the following units, with a request for it to
be updated to operate in a hotplug fashion without depending on
systemd-udev-settle.service:

    @OFFENDING_UNITS@

systemd-udev-settle is a terrible hack[1] and should never[2] ever[3] used, seriously it's very bad. It was used as a stop-gap solution for issue #39069, but thanks to PR #79532 it can be removed now. [1]: systemd/systemd#7293 (comment) [2]: #73095 [3]: #107341

rnhmjoj · 2021-02-24T08:18:41Z

rnhmjoj · 2021-02-26T23:48:04Z

@prusnak the trezord service defined in NixOS pulls in udev-settle as dependency, but the upstream unit never did. Can trezord handle hardware discovery asynchronously or are there issues?

rnhmjoj · 2021-02-27T01:27:10Z

There may be a good news: Gnome has just merged a patch that looks like it could fix the issue with gdm.
I tried it by overriding the gdm package and removing udev-settle, but the gnome3 test still fails, the error looks different, though.

Here's how I tested:

--- a/nixos/modules/services/x11/display-managers/gdm.nix
+++ b/nixos/modules/services/x11/display-managers/gdm.nix
@@ -5,7 +5,14 @@ with lib;
 let
 
   cfg = config.services.xserver.displayManager;
-  gdm = pkgs.gnome3.gdm;
+  gdm = pkgs.gnome3.gdm.overrideAttrs (old: {
+    patches = old.patches ++ [
+      (pkgs.fetchpatch {
+        url = "https://gitlab.gnome.org/GNOME/gdm/-/merge_requests/128.patch";
+        sha256 = "1h0rf7d1h4b5clpx38yy4j64ailmwjayklz52qnv6lbsx42ix1vj";
+      })
+    ];
+  });
 
   xSessionWrapper = if (cfg.setupCommands == "") then null else
     pkgs.writeScript "gdm-x-session-wrapper" ''
@@ -168,9 +175,6 @@ in
       "systemd-machined.service"
       # setSessionScript wants AccountsService
       "accounts-daemon.service"
-      # Failed to open gpu '/dev/dri/card0': GDBus.Error:org.freedesktop.DBus.Error.AccessDenied: Operation not permitted
-      # https://github.com/NixOS/nixpkgs/pull/25311#issuecomment-609417621
-      "systemd-udev-settle.service"
     ];
 
     systemd.services.display-manager.after = [
@@ -180,7 +184,6 @@ in
       "getty@tty${gdm.initialVT}.service"
       "plymouth-quit.service"
       "plymouth-start.service"
-      "systemd-udev-settle.service"
     ];
     systemd.services.display-manager.conflicts = [
       "getty@tty${gdm.initialVT}.service"

@worldofpeace @jtojnar Any change you might take a look at this? I hope I'm doing something wrong.

domenkozar · 2021-03-18T10:20:01Z

I can confirm a similar issue with missing /dev/dri/card0 on 20.09 with i3+xfce. It does not reproduce on master.

Workaround for 20.09:

  systemd.services.display-manager.after = [ "systemd-udev-settle.service" ];

peterhoeg · 2021-03-23T03:18:11Z

We have a separate issue it seems. The canonical way of solving this is by using the [email protected] to ensure a given module is loaded and then have a dependency on [email protected] from sddm/gdm. However, we are not installing that unit and it needs to be patched as the path to modprobe is in /sbin. Do you mind trying with a dummy module that does that and then introduce a dependency on that instead?

peterhoeg · 2021-03-23T03:36:51Z

Another thing that would be fun to try as I don't think it's the udev settle thing that does it - we are dealing with a race condition here (see sddm/sddm#1316) and I think that udev-settle just takes long enough for the devices to be in place.

Instead of your workaround, can you try adding systemd.services.display-manager.serviceConfig.ExecStartPre = "/bin/sh -c 'sleep 3'"; to see if any delay will work. My guess is that it will.

See NixOS#73095 It looks like it can finally be removed: the `Failed to open gpu '/dev/dri/card0'` error is gone and the nixosTests.gnome test is passing (checked 4 times in a row).

rnhmjoj · 2021-08-16T09:56:22Z

Pinging openvswitch maintainers/users: @ts468 @netixx @risicle @nspin @kmcopper
Does anyone known or can test if openvswitch (ovsdb.service) works without udev-settle? Thanks.

rnhmjoj · 2021-09-15T08:02:34Z

Finally GDM appears to be working without udev-settle.
We're now left with only three modules

lock-kernel-modules.nix: probably the only service where udev-settle might be somewhat appropriate.
zfs.nix: this is up to upstream to fix (openzfs/zfs#100891)
openvswitch.nix: @edude03, can you take a look, please?

peterhoeg · 2021-09-15T09:25:00Z

Regarding lock-kernel-modules.nix, I see a few options:

simply run udevadm settle as part of the script run by the disable-kernel-module-loading unit
turn it into a timer that runs a configurable time after boot. An admin of a given system would say "90 seconds after boot" is when it should look down.
introduce a new target that we're booting to instead of the regular multi-user.target that pulls in systemd-udev-settle after the "normal" target

Option 1 seems like the most straight-forward option to me.

rnhmjoj · 2021-09-15T09:40:28Z

A timer is also pretty simple to set up. Something like this should work:

  systemd.timers.disable-kernel-module-loading = {
    wantedBy = [ "timers.target" ];
    timerConfig.OnBootSec = cfg.timeout;
  };

The problem is that, unless the user picks up an appropriate value, it could lock the system too early. So, yes option 1 is the safest, I think.

peterhoeg · 2021-09-15T09:59:09Z

The problem is that, unless the user picks up an appropriate value, it could lock the system too early. So, yes option 1 is the safest, I think.

Option 1 gives us the behaviour closest to the current situation, that's for sure. I can also see a case for the timer option but if the purpose is to get rid of the warning, then option 1 is a no-brainer.

stale · 2022-04-25T12:15:29Z

I marked this as stale due to inactivity. → More info

rnhmjoj · 2022-04-25T13:08:37Z

This is still relevant, but not a big deal: the only remaining items are openvswitch and zfs.
The former seems unmaintained in Nixpkgs and the latter still needs to be fixed by upstream.

peterhoeg · 2022-04-28T08:38:38Z

We could handle this by having a NixOS option that disables systemd-udev-settle.service by default unless zfs or openvswitch are enabled.

rnhmjoj · 2022-04-28T11:02:12Z

That or not installing the unit at all and adding a preStart="udevadm settle"; where needed, instead.

pmarreck · 2022-08-24T20:03:51Z

Is there any way to reduce the amount of time it waits? I'm root on ZFS (which is very nice otherwise, but otherwise makes this check futile, since the pools are obviously already mounted at that point or it wouldn't be booting) and it adds 7 seconds to my boot time, and is currently also the most time-consuming thing in my boot:

❯ systemd-analyze blame
7.101s systemd-udev-settle.service

Attempting to mask it in my configuration results in this, probably due to the zfs dependency:

Failed to start zfs-import.target: Unit systemd-udev-settle.service is masked.
warning: error(s) occurred while switching to the new configuration

I will ping the other ticket on openzfs as well

peterhoeg · 2022-08-25T00:57:07Z

As a quick fix, how about just overriding it:

{
  systemd.services.zfs-import-cache = {
    requires = [ "" "whatever-it-does-require.service" ];
  };
}

The trick is the "" which clears out whatever we had before.

pmarreck · 2022-08-26T20:22:01Z

@peterhoeg I mean... that might work, although it looks pretty hacky. I do use an L2ARC on an NVMe drive, would that prevent a rebuild of the L2ARC cache after reboot? (i.e., the persistence feature that was introduced relatively recently in openzfs)

peterhoeg · 2022-08-31T13:51:51Z

While I have zfs on 2 machines, I simply don't know enough to be able to say anything remotely intelligent about your case. You could try having zfs-import-cache do wants = [ "whatever-devices" ]; and see if that does the trick. But obviously testing in production this way, isn't great.

rnhmjoj · 2023-09-02T23:01:10Z

I'm closing as zfs is the only remaining service and it up to upstream to fix it.

lheckemann added the 0.kind: bug Something is broken label Nov 9, 2019

kmcopper mentioned this issue Nov 20, 2019

nixos/ksm: Enhancements and new ksmtuned package. #73823

Closed

10 tasks

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 21, 2020

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 22, 2020

rnhmjoj mentioned this issue Dec 22, 2020

[20.09] 3-minute delay at boot waiting for wpa_supplicant to restart; may also result in failure to configure the wireless interface #107341

Closed

rnhmjoj mentioned this issue Dec 22, 2020

nixos/{networkd,dhcpcd}: remove udev-settle hack #107382

Merged

5 tasks

rnhmjoj mentioned this issue Feb 26, 2021

nixos/ksm: rewrite using systemd-tmpfiles #114464

Merged

3 tasks

domenkozar mentioned this issue Mar 18, 2021

display-manager: systemd-udev-settle serves no purpose, boot 10% faster #25311

Merged

7 tasks

rnhmjoj mentioned this issue May 7, 2021

21.05 Feature Freeze #121972

Closed

rnhmjoj mentioned this issue Jun 6, 2021

wpa_supplicant service fails at boot, but starts fine after boot (with systemctl start wpa_supplicant) #101963

Closed

rnhmjoj mentioned this issue Aug 16, 2021

nixos/gdm: remove udev-settle dependency #134302

Merged

2 tasks

zeratax mentioned this issue Aug 19, 2021

Mangohud not worked #134679

Closed

rnhmjoj mentioned this issue Sep 15, 2021

nixos/lock-kernel-modules: use udevadm settle #138001

Merged

5 tasks

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 25, 2022

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 25, 2022

pmarreck mentioned this issue Aug 24, 2022

systemd-udev-settle.service is deprecated. Please fix zfs-import-cache.service not to pull it in. openzfs/zfs#10891

Open

rnhmjoj closed this as completed Sep 2, 2023

SuperSandro2000 mentioned this issue Dec 16, 2023

zfs: udevadm[2006]: systemd-udev-settle.service is deprecated. Please fix zfs-import-${pool}.service not to pull it in. #244562

Closed

rnhmjoj mentioned this issue Jan 5, 2024

nixos/clevis: init & support for zfs, bcachefs and luks #257525

Merged

4 tasks

systemd-udev-settle.service is overused #73095

systemd-udev-settle.service is overused #73095

Comments

lheckemann commented Nov 9, 2019

worldofpeace commented Nov 9, 2019

bqv commented Apr 4, 2020

worldofpeace commented Apr 4, 2020

bqv commented Apr 4, 2020

jtojnar commented Apr 5, 2020

lheckemann commented Apr 6, 2020 • edited Loading

jtojnar commented Apr 6, 2020

lheckemann commented Apr 6, 2020

asbachb commented Apr 23, 2020

srhb commented Apr 24, 2020 • edited Loading

stale bot commented Oct 21, 2020

lheckemann commented Oct 22, 2020

rnhmjoj commented Dec 22, 2020 • edited Loading

xaverdh commented Dec 22, 2020

rnhmjoj commented Dec 22, 2020

rnhmjoj commented Feb 24, 2021 • edited Loading

rnhmjoj commented Feb 26, 2021

rnhmjoj commented Feb 27, 2021

domenkozar commented Mar 18, 2021

peterhoeg commented Mar 23, 2021

peterhoeg commented Mar 23, 2021

rnhmjoj commented Aug 16, 2021

rnhmjoj commented Sep 15, 2021

peterhoeg commented Sep 15, 2021

rnhmjoj commented Sep 15, 2021

peterhoeg commented Sep 15, 2021 via email

stale bot commented Apr 25, 2022

rnhmjoj commented Apr 25, 2022

peterhoeg commented Apr 28, 2022

rnhmjoj commented Apr 28, 2022 • edited Loading

pmarreck commented Aug 24, 2022 • edited Loading

peterhoeg commented Aug 25, 2022 • edited Loading

pmarreck commented Aug 26, 2022

peterhoeg commented Aug 31, 2022

rnhmjoj commented Sep 2, 2023

lheckemann commented Apr 6, 2020 •

edited

Loading

srhb commented Apr 24, 2020 •

edited

Loading

rnhmjoj commented Dec 22, 2020 •

edited

Loading

rnhmjoj commented Feb 24, 2021 •

edited

Loading

rnhmjoj commented Apr 28, 2022 •

edited

Loading

pmarreck commented Aug 24, 2022 •

edited

Loading

peterhoeg commented Aug 25, 2022 •

edited

Loading