argocd with crossplane: getting circular dependency warnings on each reconciliation attempt

[avoidik]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

every 180 seconds (a standard reconciliation interval) I'm getting few hundreds of unexpected warnings similar to one provided below

these errors are getting generated by the argocd-application-controller, relevant code is here:
https://github.com/argoproj/argo-cd/blob/v2.5.4/controller/cache/cache.go#L280

To Reproduce

• create standard EKS cluster
• deploy crossplane first
• deploy argocd next

Expected behavior

it would be great to find out what's going wrong here to be able to get rid of these warnings completely

Screenshots

here is an example of such a circular dependency composition

in this context Provider spec is referenced twice (from CRD and from ProviderRevision sides), I assume this is the reason to why argo-cd is throwing circular dependency warnings

Version

$ argocd version --port-forward --port-forward-namespace argocd
argocd: v2.5.4+86b2dde
  BuildDate: 2022-12-06T20:09:11Z
  GitCommit: 86b2dde8e4bf1187acd2b4294e94451cd104dad8
  GitTreeState: clean
  GoVersion: go1.18.8
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.5.4+86b2dde
  BuildDate: 2022-12-06T19:46:25Z
  GitCommit: 86b2dde8e4bf1187acd2b4294e94451cd104dad8
  GitTreeState: clean
  GoVersion: go1.18.8
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.1+g9f88ccb
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

Crossplane version:

• crossplane/crossplane:v1.9.0
• crossplanecontrib/provider-aws:v0.33.0
• crossplane/provider-kubernetes-controller:v0.3.0

Kubectl version:

$ kubectl version --short
Client Version: v1.23.13
Server Version: v1.23.13-eks-fb459a0

Logs

time="2022-12-13T17:18:28Z" level=warning msg="Circular dependency detected: map[{apiextensions.k8s.io CustomResourceDefinition  domainnames.apigateway.aws.crossplane.io}:true {pkg.crossplane.io Provider  provider-aws}:true {pkg.crossplane.io ProviderRevision  provider-aws-xyz}:true]."

[avoidik]

xref. crossplane/crossplane#3530

[jutley]

We also deploy crossplane using ArgoCD, and I am seeing these warnings as well. I'm not sure whether the issue has any implications or if it just an annoying log line.

It seems like the circular dependency detection doesn't anticipate that a resource can have multiple ownership paths to the same parent/grandparent resource. It's an unfortunate family tree but one that ArgoCD should probably be able to handle.

[avoidik]

there is an attempt to fix that via #12667