Notifications Controller Crashloops 2.13.0-rc1

[wmiller112]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

After update from 2.12.3 to 2.13.0-rc1, the notifications controller starts up and exits 0 immediately after processing the first item. Reverting the image of just the notifications controller allows it to run successfully. There is also mention of this here where users who were updated by the accidental change of the stable manifests/install.yaml to rc1 started to experience this issue. I've reviewed the upgrade guide and do not see anything related to notifications controller.

To Reproduce

Update an instance of Argocd from 2.12.3 to 2.13.0-rc1, wait for notifications controller to start up, crashloop backoff begins.

Expected behavior

Notifications controller continues to process all applications without exiting.

Version

argocd: v2.13.0-rc1+74a367d
BuildDate: 2024-09-16T08:51:02Z
GitCommit: 74a367d10e7110209610ba3ec225539ebe5f7522
GitTreeState: clean
GoVersion: go1.22.7
Compiler: gc
Platform: darwin/arm64

Logs

time="2024-09-17T17:55:51Z" level=info msg="ArgoCD Notifications Controller is starting" built="2024-09-16T08:29:44Z" commit=74a367d10e7110209610ba3ec225539ebe5f7522 namespace=argocd version=v2.13.0-rc1+74a367d
{"level":"info","msg":"serving metrics on port 9001","time":"2024-09-17T17:55:51Z"}
{"level":"info","msg":"loading configuration 9001","time":"2024-09-17T17:55:51Z"}
{"level":"info","msg":"invalidated cache for resource in namespace: argocd with the name: argocd-notifications-cm","time":"2024-09-17T17:55:51Z"}
{"level":"info","msg":"invalidated cache for resource in namespace: argocd with the name: argocd-notifications-secret","time":"2024-09-17T17:55:51Z"}
{"level":"warning","msg":"Controller is running.","time":"2024-09-17T17:55:55Z"}
{"level":"info","msg":"Start processing","resource":"argocd/<app-name>","time":"2024-09-17T17:55:55Z"}
{"app":"<app-name>","level":"debug","msg":"SyncStatus up-to-date (FinishedAt=2024-09-06 19:00:55 +0000 UTC, ReconciledAt=2024-09-17 17:46:02 +0000 UTC, Observed=0001-01-01 00:00:00 +0000 UTC","time":"2024-09-17T17:55:55Z"}

[pasha-codefresh]

I can confirm that it is related to this PR

#19368

Most likely because this line was removed

https://github.com/argoproj/argo-cd/pull/19368/files#diff-0a366e21a4f4ede84cf07a5a476ba20e61dfc6c24aa0be9e791bb589482bdb73L150

But once i reverting this commit it works properly

Will try to open PR with fix today

[rakesh0022]

@wmiller112 Did your issue resolved with the fix ?
because i am seeing the similar issue after updating to v2.13.0-rc1
@pasha-codefresh Can you help ?

[crenshaw-dev]

@rakesh0022 please try 2.13.0-rc3.