High CPU utilisation (5 cores) and spammy logs

[estahn]

Checklist:

• I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

High CPU utilisation (5 cores) and spamming logs:

W1114 12:35:34.096340       1 listers.go:77] can not retrieve list of objects using index : Index with name namespace does not exist

To Reproduce

N/A

Expected behavior

Less resource usage and not so many log entries about the same thing.

Screenshots

Version

argocd: v1.3.0-rc1+8a43840
  BuildDate: 2019-10-16T21:46:40Z
  GitCommit: 8a43840f0baef50946d3eadc1d52d6a2abc162d5
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.3.0-rc1+8a43840
  BuildDate: 2019-10-16T21:46:08Z
  GitCommit: 8a43840f0baef50946d3eadc1d52d6a2abc162d5
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.1.0
  Helm Version: v2.12.1
  Kubectl Version: v1.14.0

Logs

N/A

[alexec]

Is this a fresh install, upgrade or other please?

[estahn]

This has been running for a while. @alexec

[alexec]

OK. Can I ask what you're baselining against please?

[alexec]

Fixed by #2770 and #2761

[alexmt]

hello @estahn , does your argocd has a lot of git repos configured ?

[alexmt]

We've faced the same issue in our production cluster. The reason for high CPU usage appears to be in a not efficient implementation of setting parsing.

The ListRepositories method have n^2 complexity: it parses the whole content of repositories field from argocd-cm config map. So if argo cd instance has 100 repos when single execution of ListRepositories have to parse 30000 lines of yaml (100 repos * 3 lines per repo * 100 times) which uses a lot of CPU.

Working on PR which fixes it ASAP.

[alexmt]

Also got a response to my question from @estahn: around 80 repos are configured. That is definitely enough to cause high CPU usage.

[alexmt]

Deployed 1.3.2 + changes in #2810 internally. Fix improved performance but there I'm still seeing higher CPU usage comparing to 1.2 :(