Using NetworkManager in the initramfs

[dustymabe]

We had been planning to move to using Networkmanager in the initrd for a while. We now have a host of issues that have cropped up that we think would be solved by making this move.

This is a separate ticket (related to #24) that specifically can track moving to networkmanager in the initrd.

[dustymabe]

There is an existing WIP PR for this (coreos/fedora-coreos-config#259), but it's blocked on another issue.

[dustymabe]

After some experimentation with using NetworkManager in the initramfs it's clear that we want to do a few things to keep our life sane. Here are a few points that are interesting:

• Networking configuration can be delivered by ignition
• Ignition configuration may need to be fetched over the network, which requires networking

Considering this information there are several scenarios that we want to support:

1. user provides networking configuration via Ignition
2. user provides networking configuration via dracut kernel commandline args

However, we need to think about the different scenarios here and what the use case is for each. Below I'll indicate which ways networking configuration was delivered via Ignition: (yes/no) and dracut: (yes/no).

• Ignition: no dracut: no
  • In this case most platforms will just use the defaults (dhcp).
  • Some platforms may be able to sideload in configuration from metadata services (DigitalOcean, Packet, etc). The configuration from those services will be propagated into the real root and used persistently.
• Ignition: no dracut: yes
  • any config provided by the user to dracut for networking will get passed on to the real root and persist for all subsequent boots
• Ignition: yes dracut: no
  • use ignition provided config
• Ignition: yes dracut: yes
  • In some cases users may have different requirements for initramfs networking than real root networking (one example could be that the requirements for initramfs networking are much simpler and the requirements for real root networking is more complicated).
  • use ignition provided config, do not pass through any initramfs networking configuration.

More simply put:

if ignition:
     use ignition
else
    if dracut
        pass through dracut to real root

[dustymabe]

Another side effect of the networking split configuration delivery is that we want the networking in the real root itself to be exactly what is going to be brought up on every subsequent reboot of the machine so we'd like to teardown any networking brought up in the initramfs and have the real root properly bring it all the way up again.

Since we are using NetworkManager being able to do something like nmcli networking off would be really useful here, but currently NetworkManager is not persistently running in the initramfs so nmcli can't talk to it over dbus. In the short term we can write our own bash script to teardown networking manually.

[bgilbert]

There are platform-specific issues too. DigitalOcean and Packet need to bring up just enough networking (link-local for the former, DHCP for the latter) to fetch the rest of their network config from a metadata service. Historically Afterburn has handled this, though we've discussed moving the functionality into an NM plugin.

[cgwalters]

Might be pretty easy to run the whole initramfs under the equivalent of unshare --net.

[dustymabe]

@bgilbert I updated #394 (comment) to mention DigitalOcean and Packet.

[dustymabe]

Might be pretty easy to run the whole initramfs under the equivalent of unshare --net.

interesting

[dustymabe]

Since we are using NetworkManager being able to do something like nmcli networking off would be really useful here, but currently NetworkManager is not persistently running in the initramfs so nmcli can't talk to it over dbus. In the short term we can write our own bash script to teardown networking manually.

In a discussion with Thomas Haller it was encouraged that we investigate running NM via systemd in the initrd and he asked me to open an RFE so it can be fleshed out: https://bugzilla.redhat.com/show_bug.cgi?id=1814038

In the short term I'll explore the teardown via script approach.

[dustymabe]

Existing PRs that should bring NM into the initrd:

• add coreos-teardown-initramfs-network.service ignition-dracut#159
• fedora-coreos-base: use NetworkManager for networking in the initramfs fedora-coreos-config#310

Placing a reminder here to review systemd-resolved after we get those merged: coreos/fedora-coreos-config#288

[dustymabe]

Another side effect of the networking split configuration delivery is that we want the networking in the real root itself to be exactly what is going to be brought up on every subsequent reboot of the machine so we'd like to teardown any networking brought up in the initramfs and have the real root properly bring it all the way up again.

What I describe(d) above is exactly the problem detailed in #233

[dustymabe]

NetworkManager networking in the initramfs is now available in the latest testing release (31.20200323.2.0), which you can find on the download page.

This was implemented by the following two pull requests:

• add coreos-teardown-initramfs-network.service ignition-dracut#159
• fedora-coreos-base: use NetworkManager for networking in the initramfs fedora-coreos-config#310

We believe this change will fix the following outstanding issues:

• OKD-4.4 install, FCOS does not persist NIC settings provided as kernel args #425
• Exoscale build: Resolv.conf not created at first boot #415
• No resolv.conf being created in initramfs after pxe boot #393
• static IP configured via NM keyfile is not respected on first boot #358
• DNS issues on Azure #356
• initramfs networking config overrides /etc/NetworkManager/system-connections on first boot #233

And also possibly:

• PXE: ip=dhcp network configuration seems to stop boot progress #412

Please let us know if you encounter any issues because of this change or if this change did in fact resolve any other issues you were seeing.