[BUG] - nebari upgrade incorrect image matching on profiles.dask_worker

[sblair-metrostar]

Describe the bug

nebari upgrade is incorrectly expecting the profiles.dask_worker section to have a kubespawner_override subsection for image tag updates. Per the schema, the image value should be at the top level of the named profile.

Expected behavior

Command should successfully upgrade any matching image tags included in the profiles.dask_worker section to the current release.

OS and architecture in which you are running Nebari

Ubuntu Linux, x64

How to Reproduce the problem?

Run nebari upgrade -c nebari-config.yaml on a file with something like the following present:

Image will be found in the following (incorrect) yaml:

profiles:
  dask_worker:
    test:
      kubespawner_override:
        image: quay.io/nebari/nebari-jupyterlab:2023.4.1

Image will be missed in the following (correct) yaml:

profiles:
  dask_worker:
    test:
      image: quay.io/nebari/nebari-jupyterlab:2023.4.1

Command output

No response

Versions and dependencies used.

Nebari: 2023.7.2.dev

Compute environment

None

Integrations

No response

Anything else?

No response

[sblair-metrostar]

After digging into this further, I think this was just confusion over the yaml schema coupled with some odd behavior from legacy unit tests.

The relevant portion of the upgrade.py check that was initially failing for me was due to the jupyterlab and dask_worker subsections having different types, a list and dictionary respectively. I'd assumed they would be the same, and it's honestly a bit confusing why they don't, but making that change resolved the underlying issue without a code change.

However, that same code seems to be expecting a kubespawner_override section to contain the image value that needs upgraded, same as the jupyterlab section. Existing unit tests pertaining to qhub upgrades, though, have the image value without the kubespawner_override header. I've updated my test to pass with the code as written, but not sure which is correct given the legacy behavior and how the other dask_worker values are used. I have to assume the legacy tests are correct and the kubespawner_override portion of the dask_worker image upgrade should be removed?

[sblair-metrostar]

Actually yeah I think that is the bug. Just occurred to me you guys added schema validation for all this stuff. The kubespawner_override section does appear to be invalid for the dask_worker profiles.