ApplicationSets cannot template generator values within remote repo values.yaml override when using multiple sources

[jessebot]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Before we get started, I should mention my main goal is to have proper syntax highlighting on the values override values.

I can't use a valuesObject due to #14912, and in #9583 (comment) it was kindly suggested to try out the multiple sources route.

I cannot template use a generator to template anything inside a values.yaml file override done via the multiple sources method.

To Reproduce
Here's my ApplicationSet for mastodon:

---
# third sync wave because it has to be up after postgres
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: mastodon-app-set
  namespace: argocd
spec:
  # enable go templating
  goTemplate: true
  # this generator allows us to values from an external k8s secret
  generators:
    - plugin:
        configMapRef:
          name: secret-var-plugin-generator
        input:
          parameters:
            secret_vars:
              - mastodon_hostname
              - mastodon_s3_endpoint
              - global_cluster_issuer
  template:
    metadata:
      name: mastodon-web-app
      annotations:
        argocd.argoproj.io/sync-wave: "3"
        argocd.argoproj.io/sync-options: ApplyOnly=true
    spec:
      project: mastodon
      destination:
        server: https://kubernetes.default.svc
        namespace: mastodon
      syncPolicy:
        syncOptions:
          - ApplyOutOfSyncOnly=true
      sources:
        # forked from: https://github.com/mastodon/chart/tree/main
        - repoURL: 'https://small-hack.github.io/mastodon-helm-chart'
          targetRevision: 5.0.0
          chart: mastodon
          helm:
            valueFiles:
            - $values/mastodon/small-hack/values/values-override.yaml
        - repoURL: 'https://github.com/small-hack/argocd-apps.git'
          targetRevision: eso-helm-chart-test
          ref: values

Above I'm using a generator that just pulls from a secret, but I'm happy to put together an example with a list generator to be more clear too. The reason the generator is important is because I have values I want to override based on an existing Kubernetes secret. The values override file contains this a few parts where mastodon_hostname is supposed to be rendered:

ingress:
  enabled: true
  annotations:
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: {{ .global_cluster_issuer }}
    # ensure that NGINX's upload size matches Mastodon's
    nginx.ingress.kubernetes.io/proxy-body-size: 40m
  ingressClassName: nginx
  hosts:
    - host: {{ .mastodon_hostname }}
      paths:
        - path: '/'
  tls:
    - secretName: mastodon-tls
      hosts:
        - {{ .mastodon_hostname }}

click me for the full values-override.yaml example

image:
  repository: ghcr.io/mastodon/mastodon
  pullPolicy: IfNotPresent

mastodon:
  createAdmin:
    enabled: false
    existingSecret: mastodon-admin-credentials
    secretKeys:
      usernameKey: username
      passwordKey: password
      emailKey: email

  cron:
    # -- run `tootctl media remove` every week
    removeMedia:
      enabled: true
      schedule: "0 0 * * 0"

  # -- available locales: https://github.com/mastodon/mastodon/blob/main/config/application.rb#L71
  locale: en
  local_domain: {{ .mastodon_hostname }}

  # -- Use of WEB_DOMAIN requires careful consideration: https://docs.joinmastodon.org/admin/config/#federation
  # You must redirect the path LOCAL_DOMAIN/.well-known/ to WEB_DOMAIN/.well-known/ as described
  # Example: mastodon.example.com
  web_domain: null

  # -- If set to true, the frontpage of your Mastodon server will always redirect to the first profile in the database and registrations will be disabled.
  singleUserMode: false

  # -- Enables "Secure Mode" for more details see: https://docs.joinmastodon.org/admin/config/#authorized_fetch
  authorizedFetch: false

  # -- Enables "Limited Federation Mode" for more detauls see: https://docs.joinmastodon.org/admin/config/#limited_federation_mode
  limitedFederationMode: false

  extraVolumes:
    - name: postgres-ca
      secret:
        secretName: mastodon-postgres-server-ca-key-pair
        defaultMode: 0440

    - name: postgres-client-certs
      secret:
        secretName: mastodon-postgres-mastodon-cert
        defaultMode: 0440

  extraVolumeMounts:
    - name: postgres-ca
      mountPath: /etc/secrets/ca

    - name: postgres-client-certs
      mountPath: /etc/secrets/mastodon

  s3:
    enabled: true
    existingSecret: "mastodon-s3-credentials"
    hostname: {{ .mastodon_s3_endpoint }}
    secretKeys:
      s3AccessKeyID: S3_USER
      s3AccessKey: S3_PASSWORD
      s3Bucket: BUCKET
      s3Endpoint: ENDPOINT
      s3Hostname: HOSTNAME

  secrets:
    # these must be set manually; autogenerated keys are rotated on each upgrade
    existingSecret: "mastodon-server-secrets"

  sidekiq:
    workers:
    - name: all-queues
      # -- Number of threads / parallel sidekiq jobs that are executed per Pod
      concurrency: 25
      # -- Sidekiq queues for Mastodon that are handled by this worker. See https://docs.joinmastodon.org/admin/scaling/#concurrency
      # See https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues for how to weight queues as argument
      queues:
        - default,8
        - push,6
        - ingress,4
        - mailers,2
        - pull
        # Make sure the scheduler queue only exists once and with a worker that has 1 replica.
        - scheduler

  smtp:
    auth_method: login
    ca_file: /etc/ssl/certs/ca-certificates.crt
    delivery_method: smtp
    domain: {{ .mastodon_hostname }}
    enable_starttls: 'auto'
    from_address: toots@{{ .mastodon_hostname }}
    openssl_verify_mode: peer
    port: 587
    reply_to: no-reply@{{ .mastodon_hostname }}
    tls: true
    # keys must be named `server`, `login`, `password`
    existingSecret: mastodon-smtp-credentials

  streaming:
    port: 4000
    # -- this should be set manually since os.cpus() returns the number of CPUs on
    # the node running the pod, which is unrelated to the resources allocated to
    # the pod by k8s
    workers: 1
    # -- The base url for streaming can be set if the streaming API is deployed to
    # a different domain/subdomain.
    base_url: null
    # -- Number of Streaming Pods running
    replicas: 1

  web:
    port: 3000
    # -- Number of Web Pods running
    replicas: 1
    minThreads: "5"
    maxThreads: "5"
    workers: "2"
    persistentTimeout: "20"

  metrics:
    statsd:
      # -- Enable statsd publishing via STATSD_ADDR environment variable
      address: ""

  # Sets the PREPARED_STATEMENTS environment variable: https://docs.joinmastodon.org/admin/config/#prepared_statements
  preparedStatements: true

ingress:
  enabled: true
  annotations:
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: {{ .global_cluster_issuer }}
    # ensure that NGINX's upload size matches Mastodon's
    nginx.ingress.kubernetes.io/proxy-body-size: 40m
  ingressClassName: nginx
  hosts:
    - host: {{ .mastodon_hostname }}
      paths:
        - path: '/'
  tls:
    - secretName: mastodon-tls
      hosts:
        - {{ .mastodon_hostname }}


# https://github.com/bitnami/charts/tree/main/bitnami/elasticsearch#parameters
elasticsearch:
  # `false` will disable full-text search
  # if you enable ES after the initial install, you will need to manually run
  # RAILS_ENV=production bundle exec rake chewy:sync
  # (https://docs.joinmastodon.org/admin/optional/elasticsearch/)
  enabled: true
  master:
    replicaCount: 1
    autoscaling:
      minReplicas: 1
  data:
    replicaCount: 1
  coordinating:
    replicaCount: 1
  ingest:
    replicaCount: 1

externalDatabase:
  enabled: true
  hostname: mastodon-postgres-rw.mastodon.svc
  port: "5432"
  database: mastodon
  user: mastodon
  existingSecret: "mastodon-pgsql-credentials"
  sslmode: "verify-full"
  sslcert: "/etc/secrets/mastodon/tls.crt"
  sslkey: "/etc/secrets/mastodon/tls.key"
  sslrootcert: "/etc/secrets/ca/ca.crt"

# https://github.com/bitnami/charts/tree/main/bitnami/postgresql#parameters
postgresql:
  enabled: false

# https://github.com/bitnami/charts/tree/main/bitnami/redis#parameters
redis:
  enabled: false
  hostname: "mastodon-redis-master"
  port: 6379
  auth:
    # with a key of redis-password set to the password you want
    existingSecret: "mastodon-redis-credentials"

service:
  type: ClusterIP
  port: 80

externalAuth:
  oidc:
    enabled: false
  oauth_global:
    # -- Automatically redirect to OIDC, CAS or SAML, and don't use local account authentication when clicking on Sign-In
    omniauth_only: false

# -- https://github.com/mastodon/mastodon/blob/main/Dockerfile#L75
# if you manually change the UID/GID environment variables, ensure these values match:
podSecurityContext:
  runAsUser: 991
  runAsGroup: 991
  fsGroup: 991

securityContext: {}

serviceAccount:
  # -- Specifies whether a service account should be created
  create: true
  # -- Annotations to add to the service account
  annotations: {}

This is the error I get when using the above combination of an ApplicationSet referencing a remote published helm chart and then a values.yaml living in the same directory as the ApplicationSet:

Failed to load target state: failed to generate manifest for source 1 of 2: rpc error: code = Unknown desc = Manifest generation error (cached): helm template . --name-template mastodon-web-app --namespace mastodon --kube-version 1.27 --values <path to cached source>/mastodon/small-hack/values/values-override.yaml <api versions removed> --include-crds failed exit status 1: Error: failed to parse /mastodon/small-hack/values/values-override.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".mastodon_hostname":interface {}(nil)}

To be clear, ApplicationSet generators of any kind work with inline yaml via helm.values. Here's an example working ApplicationSet:

Click me for ApplicationSet using inline yaml in helm.values

---
# third sync wave because it has to be up after postgres
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: mastodon-app-set
  namespace: argocd
spec:
  # enable go templating
  goTemplate: true
  # this generator allows us to values from an external k8s secret
  generators:
    - plugin:
        configMapRef:
          name: secret-var-plugin-generator
        input:
          parameters:
            secret_vars:
              - mastodon_hostname
              - mastodon_s3_endpoint
              - global_cluster_issuer
  template:
    metadata:
      name: mastodon-web-app
      annotations:
        argocd.argoproj.io/sync-wave: "3"
        argocd.argoproj.io/sync-options: ApplyOnly=true
    spec:
      project: mastodon
      destination:
        server: https://kubernetes.default.svc
        namespace: mastodon
      syncPolicy:
        syncOptions:
          - ApplyOutOfSyncOnly=true
      source:
        # forked from: https://github.com/mastodon/chart/tree/main
        repoURL: 'https://small-hack.github.io/mastodon-helm-chart'
        targetRevision: 5.0.0
        chart: mastodon
        helm:
          values: |
            image:
              repository: ghcr.io/mastodon/mastodon
              pullPolicy: IfNotPresent

            mastodon:
              createAdmin:
                enabled: false
                existingSecret: mastodon-admin-credentials
                secretKeys:
                  usernameKey: username
                  passwordKey: password
                  emailKey: email

              cron:
                # -- run `tootctl media remove` every week
                removeMedia:
                  enabled: true
                  schedule: "0 0 * * 0"

              # -- available locales: https://github.com/mastodon/mastodon/blob/main/config/application.rb#L71
              locale: en
              local_domain: {{ .mastodon_hostname }}

              # -- Use of WEB_DOMAIN requires careful consideration: https://docs.joinmastodon.org/admin/config/#federation
              # You must redirect the path LOCAL_DOMAIN/.well-known/ to WEB_DOMAIN/.well-known/ as described
              # Example: mastodon.example.com
              web_domain: null

              # -- If set to true, the frontpage of your Mastodon server will always redirect to the first profile in the database and registrations will be disabled.
              singleUserMode: false

              # -- Enables "Secure Mode" for more details see: https://docs.joinmastodon.org/admin/config/#authorized_fetch
              authorizedFetch: false

              # -- Enables "Limited Federation Mode" for more detauls see: https://docs.joinmastodon.org/admin/config/#limited_federation_mode
              limitedFederationMode: false

              extraVolumes:
                - name: postgres-ca
                  secret:
                    secretName: mastodon-postgres-server-ca-key-pair
                    defaultMode: 0440

                - name: postgres-client-certs
                  secret:
                    secretName: mastodon-postgres-mastodon-cert
                    defaultMode: 0440

              extraVolumeMounts:
                - name: postgres-ca
                  mountPath: /etc/secrets/ca

                - name: postgres-client-certs
                  mountPath: /etc/secrets/mastodon

              s3:
                enabled: true
                existingSecret: "mastodon-s3-credentials"
                hostname: {{ .mastodon_s3_endpoint }}
                secretKeys:
                  s3AccessKeyID: S3_USER
                  s3AccessKey: S3_PASSWORD
                  s3Bucket: BUCKET
                  s3Endpoint: ENDPOINT
                  s3Hostname: HOSTNAME

              secrets:
                # these must be set manually; autogenerated keys are rotated on each upgrade
                existingSecret: "mastodon-server-secrets"

              sidekiq:
                workers:
                - name: all-queues
                  # -- Number of threads / parallel sidekiq jobs that are executed per Pod
                  concurrency: 25
                  # -- Sidekiq queues for Mastodon that are handled by this worker. See https://docs.joinmastodon.org/admin/scaling/#concurrency
                  # See https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues for how to weight queues as argument
                  queues:
                    - default,8
                    - push,6
                    - ingress,4
                    - mailers,2
                    - pull
                    # Make sure the scheduler queue only exists once and with a worker that has 1 replica.
                    - scheduler

              smtp:
                auth_method: login
                ca_file: /etc/ssl/certs/ca-certificates.crt
                delivery_method: smtp
                domain: {{ .mastodon_hostname }}
                enable_starttls: 'auto'
                from_address: toots@{{ .mastodon_hostname }}
                openssl_verify_mode: peer
                port: 587
                reply_to: no-reply@{{ .mastodon_hostname }}
                tls: true
                # keys must be named `server`, `login`, `password`
                existingSecret: mastodon-smtp-credentials

              streaming:
                port: 4000
                # -- this should be set manually since os.cpus() returns the number of CPUs on
                # the node running the pod, which is unrelated to the resources allocated to
                # the pod by k8s
                workers: 1
                # -- The base url for streaming can be set if the streaming API is deployed to
                # a different domain/subdomain.
                base_url: null
                # -- Number of Streaming Pods running
                replicas: 1

              web:
                port: 3000
                # -- Number of Web Pods running
                replicas: 1
                minThreads: "5"
                maxThreads: "5"
                workers: "2"
                persistentTimeout: "20"

              metrics:
                statsd:
                  # -- Enable statsd publishing via STATSD_ADDR environment variable
                  address: ""

              # Sets the PREPARED_STATEMENTS environment variable: https://docs.joinmastodon.org/admin/config/#prepared_statements
              preparedStatements: true

            ingress:
              enabled: true
              annotations:
                kubernetes.io/tls-acme: "true"
                cert-manager.io/cluster-issuer: {{ .global_cluster_issuer }}
                # ensure that NGINX's upload size matches Mastodon's
                nginx.ingress.kubernetes.io/proxy-body-size: 40m
              ingressClassName: nginx
              hosts:
                - host: {{ .mastodon_hostname }}
                  paths:
                    - path: '/'
              tls:
                - secretName: mastodon-tls
                  hosts:
                    - {{ .mastodon_hostname }}


            # https://github.com/bitnami/charts/tree/main/bitnami/elasticsearch#parameters
            elasticsearch:
              # `false` will disable full-text search
              # if you enable ES after the initial install, you will need to manually run
              # RAILS_ENV=production bundle exec rake chewy:sync
              # (https://docs.joinmastodon.org/admin/optional/elasticsearch/)
              enabled: true
              master:
                replicaCount: 1
                autoscaling:
                  minReplicas: 1
              data:
                replicaCount: 1
              coordinating:
                replicaCount: 1
              ingest:
                replicaCount: 1

            externalDatabase:
              enabled: true
              hostname: mastodon-postgres-rw.mastodon.svc
              port: "5432"
              database: mastodon
              user: mastodon
              existingSecret: "mastodon-pgsql-credentials"
              sslmode: "verify-full"
              sslcert: "/etc/secrets/mastodon/tls.crt"
              sslkey: "/etc/secrets/mastodon/tls.key"
              sslrootcert: "/etc/secrets/ca/ca.crt"

            # https://github.com/bitnami/charts/tree/main/bitnami/postgresql#parameters
            postgresql:
              enabled: false

            # https://github.com/bitnami/charts/tree/main/bitnami/redis#parameters
            redis:
              enabled: false
              hostname: "mastodon-redis-master"
              port: 6379
              auth:
                # with a key of redis-password set to the password you want
                existingSecret: "mastodon-redis-credentials"

            service:
              type: ClusterIP
              port: 80

            externalAuth:
              oidc:
                enabled: false
              oauth_global:
                # -- Automatically redirect to OIDC, CAS or SAML, and don't use local account authentication when clicking on Sign-In
                omniauth_only: false

            # -- https://github.com/mastodon/mastodon/blob/main/Dockerfile#L75
            # if you manually change the UID/GID environment variables, ensure these values match:
            podSecurityContext:
              runAsUser: 991
              runAsGroup: 991
              fsGroup: 991

            securityContext: {}

            serviceAccount:
              # -- Specifies whether a service account should be created
              create: true
              # -- Annotations to add to the service account
              annotations: {}

The only issue with that is that in most IDEs, it will show up as a string, and not yaml, and so the syntax highlighting is off. See Screenshots for example.

I just can't use helm.valuesObject or helm.valuesFiles (with an override values.yaml in a different repo from the helm chart), and wasn't sure how to move forward.

Expected behavior

I'd expect the templated values from the generator to be rendered in the values-override.yaml and for it to work the same as in-line yaml overriding.

Screenshots

Example of the error when using multiple sources:

Example of the in-line yaml not rendering properly (this also breaks the LSP's ability to follow where you are, so if your cursor is anywhere under values, it just shows at the top as being one giant values string type, so if you need to get really nested, this can make troubleshooting hard):

The reason I keep pressing this is because I'm dyslexic and syntax highlighting really helps me read.

Version
Local CLI version installed via linuxbrew.

argocd: v2.9.2+c5ea5c4.dirty
  BuildDate: 2023-11-20T18:19:09Z
  GitCommit: c5ea5c4df52943a6fff6c0be181fde5358970304
  GitTreeState: dirty
  GoVersion: go1.21.4
  Compiler: gc
  Platform: linux/amd64
  
argocd-server: v2.9.2+c5ea5c4.dirty
  BuildDate: 2023-11-20T18:19:09Z
  GitCommit: c5ea5c4df52943a6fff6c0be181fde5358970304
  GitTreeState: dirty
  GoVersion: go1.21.4
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:11:23Z
  Helm Version: v3.13.2+g2a2fb3b
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.20.0

If it is helpful, Argo CD is installed via k3s on metal via an Argo CD ApplicationSet:
https://github.com/small-hack/argocd-apps/blob/eso-helm-chart-test/argocd/argocd_appset.yaml

Logs

I'm not sure which logs to check, but I am happy to provide them if you let me know. It's not failing in a massive way like the other issue, but again, very happy to assist in any way.