CRDs not upgraded from 5.27.4 to 5.42.3

[mailbox171]

Describe the bug

Hello,
we upgraded our argocd (Helm chart) installation going from version 5.27.4 to 5.42.3 of the package (argocd 2.6 to 2.7).
We use a Terraform Helm provider to deploy/update the chart.

Everything seems ok - all our applications stayed Running all the time - but I noticed argoproj CRDs in our k8s cluster have not been upgraded, and they are still the same. They have been created months ago, and never changed.
Is it an issue? or maybe it's just because they have not changed between the two versions (5.27.4 --->> 5.42.3)?

This sentence in the README make me think that CRDs should be upgraded when the Helm chart is upgraded: "Helm cannot upgrade custom resource definitions in the /crds folder [by design] . . . Starting with 5.2.0, the CRDs have been moved to /templates to address this design decision."

Our next step will be to upgrade argocd 2.7 to 2.8, so we will upgrade to the latest version of the chart. Are the CRDs going to change this time? If so, what will happen to running Applications, Projects, .. are they going to be up all the time during the upgrade? or they will be stopped, deleted.. ?

Thanks a lot

Related helm chart

argo-cd

Helm chart version

5.42.3

To Reproduce

Deploy argocd chart 5.27.4
Check CRDs
Ugrade chart to 5.42.3
CRDs are the same

Expected behavior

Maybe CRDs should have been upgraded?

Screenshots

No response

Additional context

No response

[yu-croco]

@mailbox171
I think CRD is supposed to upgrade by upgrading the Helm Chart, which means it's not a bug of Helm Chart.

Did you handle https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd#520 when you upgrade your Helm Chart to 5.2.0?
Maybe you can investigate in your CRD's annotation/label.

[mailbox171]

@yu-croco thanks a lot for you answer, but I am afraid I didn't make myself clear enough.

• In June 23 we installed the chart version 5.27.4 (way newer than 5.2.0, so CRD should be managed by the chart).

• Now we are upgrading from 5.27.4 to 5.42.3, upgrade seems successful, all applications stayed Running at all times, web console shows new version

Checking the CRDs (see for example "applications" and "applicationsets" annotations below), we see they have not been upgraded.
So, since the expected behavior was "CRD is supposed to upgrade by upgrading the Helm Chart", now the question is:

Why the CRD have not been upgraded by upgrading the Helm Chart?

Please, I really need help on understanding this.
Possible reasons may be:

1. A possible Bug (so please restore the "Bug" label)
2. Our mistake, our configuration has a problem
3. Between 5.27.4 to 5.42.3 CRDs haven't changed, so there was no need to update CRDs
4. ... others?

Thanks a lot!

kubectl describe crd applications.argoproj.io 

Name:         applications.argoproj.io
Namespace:    
Labels:       app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=applications.argoproj.io
              app.kubernetes.io/part-of=argocd
Annotations:  helm.sh/resource-policy: keep
              meta.helm.sh/release-name: argo
              meta.helm.sh/release-namespace: argocd
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2023-06-30T09:38:49Z
  Generation:          2
  Resource Version:    103751166
  UID:                 ce9a2520-6a68-4b4a-bad0-647c48ba28db




kubectl describe crd applicationsets.argoproj.io

Name:         applicationsets.argoproj.io
Namespace:    
Labels:       app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=applicationsets.argoproj.io
              app.kubernetes.io/part-of=argocd
Annotations:  helm.sh/resource-policy: keep
              meta.helm.sh/release-name: argo
              meta.helm.sh/release-namespace: argocd
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2023-06-30T09:38:49Z
  Generation:          2
  Resource Version:    103751184
  UID:                 7091719c-5e80-4b4d-b1c9-019eebcb8a77

[yu-croco]

@mailbox171

In June 23 we installed the chart version 5.27.4 (way newer than 5.2.0, so CRD should be managed by the chart).

Now we are upgrading from 5.27.4 to 5.42.3, upgrade seems successful, all applications stayed Running at all times, web console shows new version

Thank you for the info.
Ah, then crd is originally moved to templates/ . 🤔

We use a Terraform Helm provider to deploy/update the chart.

In my project, I upgraded Argo CD to the latest version every time via Helm Chart and it worked well all the time, but I am not so sure the behavior of Terraform. 🤔

I'm not so sure at all if it's a bug, but I put the label back in case.

[yu-croco]

@mailbox171

Between 5.27.4 to 5.42.3 CRDs haven't changed, so there was no need to update CRDs

You can do investigate yourself this by searching PR.... 🙄

but I noticed argoproj CRDs in our k8s cluster have not been upgraded, and they are still the same. They have been created months ago, and never changed.

[mailbox171]

Digging deeper, I think I was looking just at part of the CRD upgrades.

• The crd objects are not changed in k8s (most likely the API stay the same between the two versions).

• The implementation is actually upgraded, as I can see - for example - from the application-controller POD annotations

Am I right?

Name:             argo-argocd-application-controller-0
Namespace:        argocd
Priority:         0
Service Account:  argocd-application-controller
Node:             . . .
Start Time:       Tue, 24 Oct 2023 20:15:48 +0200
Labels:           app.kubernetes.io/component=application-controller
                  app.kubernetes.io/instance=argo
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=argocd-application-controller
                  app.kubernetes.io/part-of=argocd
                  app.kubernetes.io/version=v2.7.11
                  controller-revision-hash=argo-argocd-application-controller-64896c7dc4
                  helm.sh/chart=argo-cd-5.42.3
                  statefulset.kubernetes.io/pod-name=argo-argocd-application-controller-0
Annotations:      . . .
                  kubectl.kubernetes.io/restartedAt: 2023-10-13T17:44:43+03:00
Status:           Running
. . .
Controlled By:  StatefulSet/argo-argocd-application-controller

[mailbox171]

We will now upgrade from 5.42.3 to 5.46.8

I guess that should result in also applying the CRD definitions changes here:
feat(argo-cd): Upgrade Argo CD to 2.8.0 (#2170)
3e35b0c

[mkilchhofer]

We at @swisspost also use the Terraform helm provider to manage Argo CD's installation. And the CRDs are upgraded all the time when needed.

Starting from helm chart version 5.2.0, the CRDs are living in the <chart_root>/templates directory of the chart (prior they lived inside <chart_root>/crds). Every release >5.2.0 should also uprgade the CRDs when using Tools that are built around the helm SDK (terraform for flux-v2 for example). Tools that are using helm only for templating and installs the rendered manifests directly via a Go client or kubectl (kustomize for example) are acting different.

Is it an issue? or maybe it's just because they have not changed between the two versions (5.27.4 --->> 5.42.3)?

Normally CRDs change from minor (app) version to minor version:

• they changed from 2.5 to 2.6 -> feat(argo-cd): Upgrade Argo CD to 2.6.0 #1758
• they changed from 2.6 to 2.7 -> feat(argo-cd): Upgrade Argo CD to v2.7.0 #1989
• they changes from 2.7 to 2.9 -> feat(argo-cd): Upgrade Argo CD to 2.8.0 #2170

[mailbox171]

@mkilchhofer thanks a lot. I really appreciate your help.
With your hints I made some significant progress. I think I miss a last clarification now.

1. We went from 2.6 to 2.7

2. From the PR (feat(argo-cd): Upgrade Argo CD to v2.7.0 #1989) seems that "commonAnnotationsEnvsubst" was introduced in 2.7, and it wasn't there in 2.6 applications-crd.yaml

3. We see the new field in our applications crd, se it seems it has been updated. Right?

kubectl describe  crd applications  | grep CommonAnnotationsEnvsubst
                            Description:  CommonAnnotationsEnvsubst specifies whether to apply env variables substitution for annotation values
                              Description:  CommonAnnotationsEnvsubst specifies whether to apply env variables substitution for annotation values
                        Description:  CommonAnnotationsEnvsubst specifies whether to apply env variables substitution for annotation values

What is misleading for me is the "Creation Timestamp: 2023-06-30T09:38:49Z". Is it updated when the crd is upgraded?
I assumed it was.. maybe I was wrong.

And then, what is the meaning of "Resource Version: 103751166". Is there a reference somewhere where the version number can be checked?

kubectl describe crd applications.argoproj.io 

Name:         applications.argoproj.io
Namespace:    
Labels:       app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=applications.argoproj.io
              app.kubernetes.io/part-of=argocd
Annotations:  helm.sh/resource-policy: keep
              meta.helm.sh/release-name: argo
              meta.helm.sh/release-namespace: argocd
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2023-06-30T09:38:49Z
  Generation:          2
  Resource Version:    103751166
  UID:                 ce9a2520-6a68-4b4a-bad0-647c48ba28db

[mkilchhofer]

Aaah now I see, you got confused by the Creation Timestamp. Unfortunately, this is how kubernetes works: when a resource first occurs in a cluster, this field gets set. You can only monitor the update if the field Resource Version and/or Generation increases.
(Keep in mind that all these 3 mentioned fields are related to Kubernetes itself, not the Argoproj software 😉 )

We could also place a version label on the CRD or for you you could set --set crds.annotations.foo=bar and do the upgrade. Then you see that the CRD resource is updated :-)

[mkilchhofer]

Are there still open points @mailbox171 or can we close this issue?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.