Handle duplication during parser second pass (LP: #2007682)

[daniloegea]

Description

When the parser needs a second pass, some configuration is being added again to their respective lists. We handle these cases already in other places either by checking if the values are present in their data structures or checking if the number of nodes in the YAML subtree matches the number of values in the lists/hashes. The second approach doesn't work well though when the same interface is defined in another file with extra configuration (example LP: #2003061)

This PR addresses 5 cases of duplication:

1. DNS domain search strings (LP: #2007682)
2. DNS servers IPs (LP: #2007682)
3. IP route rules
4. OpenvSwitch protocols
5. OpenvSwitch controllers

The example below will generate duplicated data for the routing-policy, nameservers addresses and search domains:

network:
  bonds:
    aggi:
      routing-policy:
        - table: 1
          from: 1.2.3.4
      nameservers:
        addresses:
        - 8.8.8.8
        search:
        - example.com
      interfaces:
      - ens3f0
  ethernets:
    ens3f0:
      dhcp4: true

Result:

[Match]
Name=aggi

[Network]
LinkLocalAddressing=ipv6
DNS=8.8.8.8
DNS=8.8.8.8
Domains=example.com example.com
ConfigureWithoutCarrier=yes

[RoutingPolicyRule]
From=1.2.3.4
Table=1

[RoutingPolicyRule]
From=1.2.3.4
Table=1

The example below will generate ovs commands with duplicated protocols and controller addresses:

network:
  bridges:
    br123:
      openvswitch:
        protocols:
          - OpenFlow10
          - OpenFlow11
          - OpenFlow12
        controller:
          addresses:
            - tcp:127.0.0.1:6653
            - tcp:127.0.0.2:6653
      interfaces:
        - nic1
  ethernets:
    nic1: {}

Result:

...
ExecStart=/usr/bin/ovs-vsctl set Bridge br123 protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow10,OpenFlow11,OpenFlow12
ExecStart=/usr/bin/ovs-vsctl set Bridge br123 external-ids:netplan/protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow10,OpenFlow11,OpenFlow12
ExecStart=/usr/bin/ovs-vsctl set-controller br123 tcp:127.0.0.1:6653 tcp:127.0.0.2:6653 tcp:127.0.0.1:6653 tcp:127.0.0.2:6653
ExecStart=/usr/bin/ovs-vsctl set Bridge br123 external-ids:netplan/global/set-controller=tcp:127.0.0.1:6653,tcp:127.0.0.2:6653,tcp:127.0.0.1:6653,tcp:127.0.0.2:6653

This behavior is also visible with netplan get:

$ netplan get --root-dir /workspace/netplan-issues/fakeroot_duplicates
network:
  version: 2
  ethernets:
    ens3f0:
      dhcp4: true
    nic1: {}
  bridges:
    br123:
      interfaces:
      - nic1
      openvswitch:
        protocols:
        - OpenFlow10
        - OpenFlow11
        - OpenFlow12
        - OpenFlow10
        - OpenFlow11
        - OpenFlow12
        controller:
          addresses:
          - "tcp:127.0.0.1:6653"
          - "tcp:127.0.0.2:6653"
          - "tcp:127.0.0.1:6653"
          - "tcp:127.0.0.2:6653"
  bonds:
    aggi:
      nameservers:
        addresses:
        - 8.8.8.8
        - 8.8.8.8
        search:
        - example.com
        - example.com
      interfaces:
      - ens3f0
      routing-policy:
      - table: 1
        from: "1.2.3.4"
      - table: 1
        from: "1.2.3.4"

This is the new behavior:

$ LD_LIBRARY_PATH=build/src/  PYTHONPATH=. ./src/netplan.script get --root-dir /workspace/netplan-issues/fakeroot_duplicates
network:
  version: 2
  ethernets:
    ens3f0:
      dhcp4: true
    nic1: {}
  bridges:
    br123:
      interfaces:
      - nic1
      openvswitch:
        protocols:
        - OpenFlow10
        - OpenFlow11
        - OpenFlow12
        controller:
          addresses:
          - "tcp:127.0.0.1:6653"
          - "tcp:127.0.0.2:6653"
  bonds:
    aggi:
      nameservers:
        addresses:
        - 8.8.8.8
        search:
        - example.com
      interfaces:
      - ens3f0
      routing-policy:
      - table: 1
        from: "1.2.3.4"

Checklist

• Runs make check successfully.
• Retains 100% code coverage (make check-coverage).
• New/changed keys in YAML format are documented.
• (Optional) Adds example YAML for new feature.
• (Optional) Closes an open bug in Launchpad. LP#2007682

[daniloegea]

Well, it looks like the memory leak checker works! 😅

[slyon]

Nice work! This is mostly looking good to me. One remark and some nitpicks inline.

If you could split & squash your final "memoryleak fixes" commit into the corresponding previous commits, and do the same for the fixes for my inline remarks, we could make this PR a rebase & merge, instead of a squash merge, as I think the commits have a really nice structure with one logical change per commit.