Setting ARGOCD_ENABLE_DYNAMIC_CLUSTER_DISTRIBUTION results in error updating the sharding config map %!s(<nil>)

[morey-tech]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When using the dynamic cluster shard re-balancing feature, setting ARGOCD_ENABLE_DYNAMIC_CLUSTER_DISTRIBUTION to true results in the application-controller Deployment failing to start due to an error updating the sharding ConfigMap. It appears that the ConfigMap name becomes empty in the application-controller logic, as it's trying to update %!s(<nil>).

Interestingly, setting to false or simply removing the ARGOCD_ENABLE_DYNAMIC_CLUSTER_DISTRIBUTION env var results in the argocd-app-controller-shard-cm ConfigMap getting updated correctly.

To Reproduce

Install Argo CD with this kustomization, which includes the Deployment and the env var enabled.

Expected behavior

The application-controller Deployment should correctly determine the ConfigMap to use for the shard mapping.

Screenshots

The configmap being correctly used by the previous replicaset with the env var disabled.

Version

{
    "Version": "v2.9.1+58b04e5",
    "BuildDate": "2023-11-14T15:08:20Z",
    "GitCommit": "58b04e5e11d007b0518853029ff7612c24a2eb35",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/arm64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.12.1+gf32a527",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0"
}

Logs

time="2023-11-15T16:17:17Z" level=info msg="ArgoCD Application Controller is starting" built="2023-11-06T04:43:50Z" commit=9cf0c69bbe70393db40e5755e34715f30179ee09 namespace=argocd version=v2.9.0+9cf0c69
time="2023-11-15T16:17:17Z" level=fatal msg="unable to get shard due to error updating the sharding config map: %!s(<nil>)"

[morey-tech]

@ishitasequeira, I am hoping to get your insights! Maybe I've missed something in my setup.

[Thakurvaibhav]

I just deployed v2.9.3 and started running into this. Hoping for a fix soon.

[ishitasequeira]

Thanks @morey-tech for reporting the issue. I think I missed this issue. I will take a look.

[ishitasequeira]

I went through the error message. The error shown is an expected message when the configmap has been updated by another controller while the current controller is trying to update the configmap.

I will update the message so that it does not show up as an error and more as an expected message.

[morey-tech]

@ishitasequeira, when I install a fresh instance of Argo CD with this kustomization, which includes the Deployment and the env var enabled, the application-controller enter's a CrashLoopBackOff

NAME                                                READY   STATUS             RESTARTS      AGE
argocd-application-controller-7b4bffb558-2clmm      0/1     CrashLoopBackOff   4 (25s ago)   2m5s
argocd-application-controller-7b4bffb558-z4h94      0/1     CrashLoopBackOff   4 (14s ago)   2m5s

With the same error as in the issue.

time="2024-01-10T19:46:55Z" level=info msg="ArgoCD Application Controller is starting" built="2023-11-06T04:43:50Z" commit=9cf0c69bbe70393db40e5755e34715f30179ee09 namespace=argocd version=v2.9.0+9cf0c69
time="2024-01-10T19:46:55Z" level=fatal msg="unable to get shard due to error updating the sharding config map: %!s(<nil>)"

Despite the configmap has only been updated by these controller deployments:

$ k get cm -n argocd argocd-app-controller-shard-cm -o yaml
apiVersion: v1
data:
  shardControllerMapping: '[{"ShardNumber":0,"ControllerName":"argocd-application-controller-7b4bffb558-2clmm","HeartbeatTime":"2024-01-10T19:48:09Z"},{"ShardNumber":1,"ControllerName":"argocd-application-controller-7b4bffb558-z4h94","HeartbeatTime":"2024-01-10T19:48:26Z"}]'
kind: ConfigMap
metadata:
  name: argocd-app-controller-shard-cm
  namespace: argocd

Should the controller exit after that error message?

[esn89]

I am installing ArgoCD via helm-chart, how do I utilize that Kustomization?

[ishitasequeira]

Thanks @morey-tech !! I was able to reproduce the bug finally. The bug seems to come post the introduction of ARGOCD_ENABLE_DYNAMIC_CLUSTER_DISTRIBUTION flag. Working on the fix for the same.

[ishitasequeira]

@morey-tech I have raised the PR #16984 for the fix.