OKD-4.4 install, FCOS with multiple NIC settings provided as kernel args

[cgruver]

FCOS version: 31.20200323.2.0 from testing Stream

Platform: Libvirt

Boot: iPXE

With a single NIC, the ip= kernel args are persisted. This was fixed by #394 and #425

Now, I am trying a build with 2 NICs per master and worker node so that I can use the second NIC on my storage network.

During the FCOS install, it is only persisting the configuration of the second NIC specified in the kernel args.

iPXE file:

#!ipxe

kernel http://10.11.11.10/install/fcos/vmlinuz ip=10.11.11.101::10.11.11.1:255.255.255.0:okd4-prd-master-0.domain.org:eth0:none ip=10.11.12.101:::255.255.255.0::eth1:none nameserver=10.11.11.10 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://10.11.11.10/install/fcos/install.xz coreos.inst.ignition_url=http://10.11.11.10/install/fcos/ignition/master.ign coreos.inst.platform_id=qemu console=ttyS0
initrd http://10.11.11.10/install/fcos/initrd

Relevant log snippets from the console during install:

[  OK  ] Started dracut pre-udev hook.
[    3.242147] systemd[1]: Starting udev Kernel Device Manager...
         Starting udev Kernel Device Manager...
[    3.285511] systemd-vconsole-setup[327]: KD_FONT_OP_GET failed while trying to get the font metadata: Invalid argument
[    3.285513] systemd[1]: Started Journal Service.
[    3.287954] systemd-vconsole-setup[327]: Fonts will not be copied to remaining consoles
[  OK      3.291852] systemd-modules-load[326]: Inserted module 'fuse'
0m] Started Journal Service.
[    3.293944] dracut-cmdline[344]: dracut-31.20200323.2.0 (CoreOS) dracut-050-26.git20200316.fc31
[    3.298404] dracut-cmdline[344]: Using kernel command line parameters: rd.driver.pre=btrfs ip=10.11.11.101::10.11.11.1:255.255.255.0:okd4-prd-master-0.domain.org:eth0:none ip=10.11.12.101:::255.255.255.0::eth1:none nameserver=10.11.11.10 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://10.11.11.10/install/fcos/install.xz coreos.inst.ignition_url=http://10.11.11.10/install/fcos/ignition/master.ign coreos.inst.platform_id=qemu console=ttyS0
[    3.298491] audit: type=1130 audit(1585426624.191:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
         Startin[    3.311722] systemd[1]: Starting Create Volatile Files and Directories...
g Create Volatile Files and Directories...

...... Initial install & reboot

[  OK  ] Started Network Manager Script Dispatcher Service.
[  OK  ] Started Network Manager Wait Online.
[  OK  ] Reached target Network is Online.
         Starting CoreOS Installer...
######################################################################## 100.0%
[    7.853413] coreos-installer-service[815]: coreos-installer install /dev/sda --ignition /tmp/coreos-installer-JiQkJo --firstboot-args rd.neednet=1 ip=10.11.12.101:::255.255.255.0::eth1:none nameserver=10.11.11.10  --image-url http://10.11.11.10/install/fcos/install.xz --platform qemu
[    7.876767] coreos-installer-service[815]: Downloading image from http://10.11.11.10/install/fcos/install.xz
[    7.879440] coreos-installer-service[815]: Downloading signature from http://10.11.11.10/install/fcos/install.xz.sig
[   18.171488] kauditd_printk_skb: 26 callbacks suppressed

As you can see, it dropped the configuration for eth0 so when it tries to pull the ignition config, it cannot access the network:

         Starting Ignition (fetch)...
[    4.386805] ignition[589]: Ignition 2.2.1
[    4.388803] ignition[589]: Stage: fetch
[    4.390766] ignition[589]: reading system config file "/usr/lib/ignition/base.ign"
[    4.393955] ignition[589]: no config URL provided
[    4.395824] ignition[589]: reading system config file "/usr/lib/ignition/user.ign"
[    4.417920] ignition[589]: Adding "root-ca" to list of CAs
[    4.419636] ignition[589]: GET https://api-int.okd4-prd.redacted.domain.org:22623/config/master: attempt #1
[    4.422583] ignition[589]: GET error: Get https://api-int.okd4-prd.redacted.domain.org:22623/config/master: dial tcp: lookup api-int.okd4-prd.redacted.domain.org on 10.11.11.10:53: dial udp 10.11.11.10:53: connect: network is unreachable

For reference. The installation with a single NIC works just fine. It just appears to be dropping the config of the first NIC and only using the second, when I specify 2 NICs.

[dustymabe]

This looks like it is a bug (or maybe feature 😄) in the installer where it doesn't consider that there are some dracut networking args where it is valid to specify them more than once.

@bgilbert should we modify the code to handle multiple args for the dracut networking args that support it?

[dustymabe]

A workaround for now would be to specify the networking configuration via files delivered by Ignition (though you'd still have to specify it as a karg for at least one interface so you could download the Ignition config from the remote). A head start on creating the config files is doing something like:

$ /usr/libexec/nm-initrd-generator -s -- ip=10.11.11.101::10.11.11.1:255.255.255.0:okd4-prd-master-0.domain.org:eth0:none ip=10.11.12.101:::255.255.255.0::eth1:none nameserver=10.11.11.10 rd.neednet=1

*** Connection 'eth0' ***

[connection]
id=eth0
uuid=2ec5a874-bc6d-44a6-8da9-ea9fed8e8495
type=ethernet
interface-name=eth0
multi-connect=1
permissions=

[ethernet]
mac-address-blacklist=

[ipv4]
address1=10.11.11.101/24,10.11.11.1
dhcp-hostname=okd4-prd-master-0.domain.org
dns=10.11.11.10;
dns-search=
may-fail=false
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dhcp-hostname=okd4-prd-master-0.domain.org
dns-search=
method=disabled

[proxy]

*** Connection 'eth1' ***

[connection]
id=eth1
uuid=7a74d93a-8d2d-4c17-b3d1-45288af915a1
type=ethernet
interface-name=eth1
multi-connect=1
permissions=

[ethernet]
mac-address-blacklist=

[ipv4]
address1=10.11.12.101/24
dns-search=
may-fail=false
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=disabled

[proxy] 

so you'd create two files via Ignition or fcct. Something like:

• /etc/NetworkManager/system-connections/eth0.nmconnection
• /etc/NetworkManager/system-connections/eth1.nmconnection

[cgruver]

Thanks @dustymabe I'll try the ignition files while you guys work on this. I need to learn more of that ecosystem anyway since I'm new to FCOS/RHCOS.

We'll call it a feature for now. ;-) That's so much more optimistic.

[bgilbert]

Yup, it's a bug: coreos/coreos-installer#156. PR in coreos/coreos-installer#195.

[dustymabe]

This was fixed upstream in coreos/coreos-installer#228.

The fix for this went into testing stream release 31.20200505.2.0. Please try out the new release and report issues.

[cgruver]

Excellent!!!

I will build a new cluster today and see if I can use the second NIC for my. storage network.

[dustymabe]

The fix for this went into stable stream release 31.20200505.3.0.

[cgruver]

I have verified this fix with an OKD 4.4 UPI build with 2 NICs

Thanks!