sriov: accept setting the eswitch mode without VFs (LP: #2020409) #454

daniloegea · 2024-04-09T11:55:26Z

The embedded switch mode can be set even if the interface doesn't have any virtual functions. This is a requisite to use the PF with only scalable functions for example.

With this change, the presence of the 'embedded-switch-mode' will be enough to identify the interface as a PF and emit the systemd services required to apply the configuration.

The changes required by the CLI will be done is the next commit.

A PPA for Noble is available here https://launchpad.net/~danilogondolfo/+archive/ubuntu/netplan-sriov

Description

I tested some scenarios on real hardware to make sure the changes work as intended and the existing features are not broken.

Test 1: basic configuration - PFs without VFs (the use case for scalable functions)

It tests if setting the embedded switch mode will now work when the PFs don’t have VFs (when SR-IOV is disabled basically).

Initial state:

root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.0
pci/0000:03:00.0: mode legacy inline-mode none encap-mode basic
root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.1
pci/0000:03:00.1: mode legacy inline-mode none encap-mode basic

The eswitch mode legacy is set to both interfaces

Applying the configuration below will change the eswitch mode to switchdev. Note that these PFs don’t have VFs.

network:
  version: 2
  ethernets:
    enp3s0f0np0:
      match:
        macaddress: 98:03:9b:c3:ef:ba
      mtu: 9000
      set-name: enp3s0f0np0
      embedded-switch-mode: switchdev
    enp3s0f1np1:
      match:
        macaddress: 98:03:9b:c3:ef:bb
      mtu: 9000
      set-name: enp3s0f1np1
      embedded-switch-mode: switchdev

After calling netplan applythe eswitch mode will be changed

root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.0
pci/0000:03:00.0: mode switchdev inline-mode none encap-mode basic
root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.1
pci/0000:03:00.1: mode switchdev inline-mode none encap-mode basic

Test 2: change the eswitch mode but with VFs

Initial state

root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.0
pci/0000:03:00.0: mode legacy inline-mode none encap-mode basic
root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.1
pci/0000:03:00.1: mode legacy inline-mode none encap-mode basic
root@node-laveran:~# cat /sys/class/net/enp3s0f0np0/device/sriov_numvfs
0
root@node-laveran:~# cat /sys/class/net/enp3s0f1np1/device/sriov_numvfs
0

Both PFs are on legacy mode with zero VFs.

The configuration below is applied, setting them to switchdev and at the same time creating two VFs:

network:
  version: 2
  ethernets:
    enp3s0f0np0:
      match:
        macaddress: 98:03:9b:c3:ef:ba
      mtu: 9000
      set-name: enp3s0f0np0
      embedded-switch-mode: switchdev
    enp3s0f1np1:
      match:
        macaddress: 98:03:9b:c3:ef:bb
      mtu: 9000
      set-name: enp3s0f1np1
      embedded-switch-mode: switchdev

    enp3s0f0v0:
      link: enp3s0f0np0
    enp3s0f0v1:
      link: enp3s0f0np0
    enp3s0f1v0:
      link: enp3s0f1np1
    enp3s0f1v1:
      link: enp3s0f1np1

Result:

root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.0
pci/0000:03:00.0: mode switchdev inline-mode none encap-mode basic
root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.1
pci/0000:03:00.1: mode switchdev inline-mode none encap-mode basic
root@node-laveran:~# cat /sys/class/net/enp3s0f0np0/device/sriov_numvfs
2
root@node-laveran:~# cat /sys/class/net/enp3s0f1np1/device/sriov_numvfs
2

Test 3: VF LAG activation still works after these changes

VF LAG requires that the setup happens in a very specific order.

Before creating the LAG interface containing both PFs, the eswitch mode must be set and all the VFs must be unbound. Then after waiting for networkd to setup the bond, netplan rebind needs to be called to bind the VFs to the driver.

To test if it’s working I use the configuration below and reboot the system:

network:
  version: 2
  ethernets:
    enp3s0f0np0:
      match:
        macaddress: 98:03:9b:c3:ef:ba
      mtu: 9000
      set-name: enp3s0f0np0
      embedded-switch-mode: switchdev
      virtual-function-count: 32
      delay-virtual-functions-rebind: true
    enp3s0f1np1:
      match:
        macaddress: 98:03:9b:c3:ef:bb
      mtu: 9000
      set-name: enp3s0f1np1
      embedded-switch-mode: switchdev
      virtual-function-count: 32
      delay-virtual-functions-rebind: true

  bonds:
    bond1:
      interfaces:
        - enp3s0f0np0
        - enp3s0f1np1
      macaddress: 98:03:9b:c3:ef:ba
      mtu: 9000
      parameters:
        down-delay: 0
        lacp-rate: fast
        mii-monitor-interval: 100
        mode: 802.3ad
        transmit-hash-policy: layer3+4
        up-delay: 0

All the settings are applied and the LAG state is active

root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.0
pci/0000:03:00.0: mode switchdev inline-mode none encap-mode basic
root@node-laveran:~# devlink dev eswitch show pci/0000:03:00.1
pci/0000:03:00.1: mode switchdev inline-mode none encap-mode basic
root@node-laveran:~# cat /sys/class/net/enp3s0f0np0/device/sriov_numvfs
32
root@node-laveran:~# cat /sys/class/net/enp3s0f1np1/device/sriov_numvfs
32
root@node-laveran:~# cat /sys/kernel/debug/mlx5/0000\:03\:00.0/lag/state
active
root@node-laveran:~# cat /sys/kernel/debug/mlx5/0000\:03\:00.1/lag/state
active

Checklist

Runs make check successfully.
Retains 100% code coverage (make check-coverage).
New/changed keys in YAML format are documented.
(Optional) Adds example YAML for new feature.
(Optional) Closes an open bug in Launchpad.

The embedded switch mode can be set even if the interface doesn't have any virtual functions. This is a requisite to use the PF with only scalable functions for example. With this change, the presence of the 'embedded-switch-mode' will be enough to identify the interface as a PF and emit the systemd services required to apply the configuration. The changes required by the CLI will be done is the next commit.

Refactor get_vf_count_and_functions() in 3 different ones. It was doing too many things and was hard to read. Three of its 5 parameters were being used as output and one of them was being changed by _get_target_interface() in the same call chain. The new _get_physical_functions() will also return interfaces that only have the property embedded_switch_mode. This is a requirement to enable netplan to change the eswitch mode even if it doesn't have VFs. See LP: #2020409 Adapt the unit tests to the new functions and implement new tests.

The call to pcidev.devlink_set was still inside a check for the existence of VFs. Move it to outside of it. Add a test to cover the case where the unbind_vfs operation fails.

slyon

Thank you for the excellent testing story!
Also, I like the refactoring & breakdown into smaller methods a lot!

LGTM. I left some minor inline comments, it's up to you if you want to address those.

slyon · 2024-04-17T14:02:01Z

netplan_cli/cli/sriov.py

+            for interface in interfaces:
+                if not _interface_matches(netdef, interface):
+                    continue
+                # we have a matching PF
+                # error out if we matched more than one
+                if len(matches) > 1:
+                    raise ConfigurationError('matched more than one interface for a PF device: %s' % netdef.id)
+                matches.add(interface)
+            if matches:
+                return list(matches)[0]


suggestion (non-blocking): I think some of this could be replaced by the utils.find_matching_iface() method. Error handling cannot be clearly translated, though.

this method returns None if it either found more than one match or no matches at all. Based on the current logic I need to know if there's multiple matches. But yeah I understand there some code duplication going on here... maybe it's a task for a future refactoring :|

slyon · 2024-04-17T14:03:06Z

netplan_cli/cli/sriov.py

+            # we walk through all the system interfaces to determine if there is
+            # more than one matched interface
+            for interface in interfaces:
+                if not _interface_matches(netdef, interface):


question (non-blocking): Why not just call netdef._match_interface() directly here?

I don't have a strong opinion here, but I don't know it looks cleaner with netdef._match_interface wrapped in this method...

daniloegea force-pushed the mlx_switchdev_lp2020409 branch 2 times, most recently from a784da7 to ef39285 Compare April 15, 2024 10:44

daniloegea changed the title ~~[WIP] sriov: accept setting the eswitch mode without VFs~~ [WIP] sriov: accept setting the eswitch mode without VFs (LP: #2020409) Apr 15, 2024

daniloegea force-pushed the mlx_switchdev_lp2020409 branch from ef39285 to 936af5c Compare April 15, 2024 12:29

daniloegea changed the title ~~[WIP] sriov: accept setting the eswitch mode without VFs (LP: #2020409)~~ sriov: accept setting the eswitch mode without VFs (LP: #2020409) Apr 15, 2024

daniloegea marked this pull request as ready for review April 15, 2024 12:30

daniloegea added 2 commits April 15, 2024 16:15

daniloegea force-pushed the mlx_switchdev_lp2020409 branch from 936af5c to bdce7ff Compare April 15, 2024 15:21

daniloegea marked this pull request as draft April 16, 2024 09:29

daniloegea force-pushed the mlx_switchdev_lp2020409 branch 2 times, most recently from d8dcab7 to cf31ec0 Compare April 16, 2024 11:33

daniloegea marked this pull request as ready for review April 16, 2024 13:28

daniloegea requested a review from slyon April 16, 2024 13:29

cli/sriov: set eswitch regardless of pcidev.vfs

391b7e5

The call to pcidev.devlink_set was still inside a check for the existence of VFs. Move it to outside of it. Add a test to cover the case where the unbind_vfs operation fails.

daniloegea force-pushed the mlx_switchdev_lp2020409 branch from cf31ec0 to 391b7e5 Compare April 17, 2024 09:38

slyon approved these changes Apr 17, 2024

View reviewed changes

daniloegea merged commit 07be775 into canonical:main Apr 18, 2024
12 of 16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sriov: accept setting the eswitch mode without VFs (LP: #2020409) #454

sriov: accept setting the eswitch mode without VFs (LP: #2020409) #454

daniloegea commented Apr 9, 2024 •

edited by slyon

Loading

slyon left a comment •

edited

Loading

slyon Apr 17, 2024

daniloegea Apr 18, 2024

slyon Apr 17, 2024

daniloegea Apr 18, 2024

sriov: accept setting the eswitch mode without VFs (LP: #2020409) #454

sriov: accept setting the eswitch mode without VFs (LP: #2020409) #454

Conversation

daniloegea commented Apr 9, 2024 • edited by slyon Loading

Description

Checklist

slyon left a comment • edited Loading

Choose a reason for hiding this comment

slyon Apr 17, 2024

Choose a reason for hiding this comment

daniloegea Apr 18, 2024

Choose a reason for hiding this comment

slyon Apr 17, 2024

Choose a reason for hiding this comment

daniloegea Apr 18, 2024

Choose a reason for hiding this comment

daniloegea commented Apr 9, 2024 •

edited by slyon

Loading

slyon left a comment •

edited

Loading