90_custom.yaml duplicate entries

[jimmykarily]

Originally reported by Dave on Slack and now reproduced in the bundles test.

With the bundles tests config passed as a cdrom datasource (like the tests do), the /oem directory has various files:

kairos-aiup:~ # ls /oem/
90_custom.yaml	grubenv  lost+found  userdata  userdata.yaml

kairos-aiup:~ # cat /oem/userdata
#cloud-config

stages:
  initramfs:
    - name: "Set user and password"
      users:
        kairos:
          passwd: "kairos"
      hostname: kairos-{{ trunc 4 .Random }}
fail_on_bundles_errors: true

debug: true

install:
  auto: true
  reboot: true
  device: auto
  grub_options:
    extra_cmdline: foobarzz
  bundles:
  - rootfs_path: /var/lib/extensions/kubo
    targets:
    - container://ttl.sh/31fb00c6-a4d6-4c0f-9646-32a8a5d8f7c4:24h

kairos-aiup:~ # cat /oem/userdata.yaml 
#cloud-config

stages:
  initramfs:
    - name: "Set user and password"
      users:
        kairos:
          passwd: "kairos"
      hostname: kairos-{{ trunc 4 .Random }}
fail_on_bundles_errors: true

debug: true

install:
  auto: true
  reboot: true
  device: auto
  grub_options:
    extra_cmdline: foobarzz
  bundles:
  - rootfs_path: /var/lib/extensions/kubo
    targets:
    - container://ttl.sh/31fb00c6-a4d6-4c0f-9646-32a8a5d8f7c4:24h

kairos-aiup:~ # cat /oem/90_custom.yaml 
#cloud-config

debug: true
fail_on_bundles_errors: true
install:
    auto: true
    bundles:
        - rootfs_path: /var/lib/extensions/kubo
          targets:
            - container://ttl.sh/31fb00c6-a4d6-4c0f-9646-32a8a5d8f7c4:24h
        - rootfs_path: /var/lib/extensions/kubo
          targets:
            - container://ttl.sh/31fb00c6-a4d6-4c0f-9646-32a8a5d8f7c4:24h
    device: auto
    grub_options:
        extra_cmdline: foobarzz
    reboot: true
stages:
    initramfs:
        - hostname: kairos-{{ trunc 4 .Random }}
          name: Set user and password
          users:
            kairos:
                passwd: kairos
        - hostname: kairos-{{ trunc 4 .Random }}
          name: Set user and password
          users:
            kairos:
                passwd: kairos

Notice the duplicate entries in the 90_custom.yaml file above

[jimmykarily]

Sanity check, the datasource cdrom doesn't have duplicate entries:

kairos-ft00:/home/kairos # mkdir /tmp/cdrom
kairos-ft00:/home/kairos # mount /dev/sr1 -t iso9660 -o ro /tmp/cdrom
kairos-ft00:/home/kairos # ls /tmp/cdrom/
meta-data  user-data
kairos-ft00:/home/kairos # cat /tmp/cdrom/user-data 
#cloud-config

stages:
  initramfs:
    - name: "Set user and password"
      users:
        kairos:
          passwd: "kairos"
      hostname: kairos-{{ trunc 4 .Random }}
fail_on_bundles_errors: true

debug: true

install:
  auto: true
  reboot: true
  device: auto
  grub_options:
    extra_cmdline: foobarzz
  bundles:
  - rootfs_path: /var/lib/extensions/kubo
    targets:
    - container://ttl.sh/31fb00c6-a4d6-4c0f-9646-32a8a5d8f7c4:24h

[jimmykarily]

Reproducer outside tests:

• Create a cloud config with: earthly -P +prepare-bundles-tests . This will create a bundles-config.yaml in the root directory of the kairos repo
• Create a datasource.iso from the above config: earthly +datasource-iso --CLOUD_CONFIG=bundles-config.yaml . This will create the file build/datasource.iso
• Boot a kairos core image, into "manual install" with the above iso mounted too (2nd cdrom)
• Run kairos install
• Let the installation finish and reboot (it automatically reboots due to the config we provided)
• Check the files in /oem

The above allows us to copy a customized kairos-agent and use that to start the installation with more logs etc.

[jimmykarily]

userdata and userdata.yaml are generated here: https://github.com/mudler/yip/blob/48147fae9dbcc91559cd590976816ca3f65a3bff/pkg/plugins/datasource.go#L246-L252

90_custom.yaml is generated here: https://github.com/kairos-io/kairos-agent/blob/b2c2220f3e2b83b65089d6a60f90eb72541827e7/pkg/elemental/elemental.go#L398

[jimmykarily]

I found the guilty code: https://github.com/kairos-io/kairos-sdk/blob/f37fc75b5dfadcb8b17d91ca8b47066ec68df881/collector/collector.go#L258

We read every files that either contains userdata in the name or has a yaml or yml extension. Given we have both userdata and userdata.yaml in /oem , both pass the check and are read.

There is no difference between userdata.yaml and userdata according to the yip code (see link above). We only write userdata.yaml when it passes validation which means, yip assumes only userdata.yaml will actually be used. I think the correct fix is to remove the match for userdata from the kairos-sdk, since after all, userdata.yaml will be used due to the extension.

I removed the userdata matching condition and the duplication is gone.

[jimmykarily]

New kairos-agent packages are building: https://github.com/kairos-io/packages/actions/runs/6971523152
When they are ready, we will just bump the luet repositories on kairos-io/kairos and this issue can be QAed and closed.

[jimmykarily]

Actually, since this was discovered through the bundles test, maybe it's a good place to test for regressions. E.g. test that the produced 90_custom.yaml configuration doesn't have duplicate entries. Let's do that too.