SharedResourceWarning Errors when Argo is self managing Umbrella type applications.

[rawc0der]

Summary

Encountered a setup which creates all sorts of problems.
I created an Umbrella application (Argo Application CRD referencing itself and other Applications from a folder path).
The reason for this setup is that Umbrella app is reading all the Application manifests from single folder, which further spawns the entire graph of dependencies.
I don't know if this type of setup is bad because it's poorly documented anywhere.
Has this been encountered by anyone else?

What happened/what you expected to happen?
I'm updating the umbrella app values which get read by downstream components.

After renaming Umbrella app by mistake, both applications are now conflicting and entire system is toast.
Argo components are stuck without possibility to sync/delete any resources.

• Any suggestions for recovery?
• I want to know if this type of tree like setup, with Argo self managing is going to be supported.

Argo Application resource status fields show the following:

status:
  conditions:
  - lastTransitionTime: "2021-02-23T10:47:15Z"
    message: Application/umbrella-stack is part of applications umbrella and umbrella-stack
    type: SharedResourceWarning

Diagnostics

What version of Argo Rollouts are you running?
argo-rollouts:v0.10.2

Paste the logs from the argo-server controller

│ time="2021-02-23T11:03:30Z" level=info msg="Deleting resources" application=umbrella                                                                                                                                                                     │
│ time="2021-02-23T11:03:30Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-02-23 11:00:30 +0000 UTC, expiry: 3m0s), level (2)" application=model-entrypoint-api                                                           │
│ time="2021-02-23T11:03:30Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2021-02-23 11:00:30 +0000 UTC, expiry: 3m0s), level (2)" application=umbrella-stack                                                                 │
│ time="2021-02-23T11:03:30Z" level=info msg="Ignore status for CustomResourceDefinitions"                                                                                                                                                                 │
│ time="2021-02-23T11:03:30Z" level=info msg="Ignore status for CustomResourceDefinitions"                                                                                                                                                                 │
│ time="2021-02-23T11:03:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: test)" application=model-entrypoint-api                                                                                             │
│ time="2021-02-23T11:03:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=umbrella-stack                                                                                                 │
│ time="2021-02-23T11:03:30Z" level=info msg="Deleting resources" application=umbrella-stack                                                                                                                                                               │
│ time="2021-02-23T11:03:30Z" level=info msg="2 objects remaining for deletion" application=umbrella                                                                                                                                                       │
│ time="2021-02-23T11:03:30Z" level=info msg="1 objects remaining for deletion" application=umbrella-stack                                                                                                                                                 │
│ time="2021-02-23T11:03:30Z" level=info msg="getRepoObjs stats" application=model-entrypoint-api build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=336 unmarshal_ms=335 version_ms=0                                                            │
│ time="2021-02-23T11:03:30Z" level=info msg="Deleting resources" application=umbrella                                                                                                                                                                     │
│ time="2021-02-23T11:03:30Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=umbrella                                                                                                                        │
│ time="2021-02-23T11:03:30Z" level=info msg="Ignore status for CustomResourceDefinitions"                                                                                                                                                                 │
│ time="2021-02-23T11:03:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=umbrella                                                                                                       │
│ time="2021-02-23T11:03:30Z" level=info msg="Update successful" application=model-entrypoint-api                                                                                                                                                          │
│ time="2021-02-23T11:03:30Z" level=info msg="Reconciliation completed" application=model-entrypoint-api dedup_ms=0 dest-name= dest-namespace=test dest-server="https://kubernetes.default.svc" diff_ms=17 fields.level=2 git_ms=336 health_ms=0 live_ms=2 │
│ 6 settings_ms=0 sync_ms=0 time_ms=414                                                                                                                                                                                                                    │
│ time="2021-02-23T11:03:30Z" level=info msg="2 objects remaining for deletion" application=umbrella                                                                                                                                                       │
│ time="2021-02-23T11:03:30Z" level=info msg="getRepoObjs stats" application=umbrella build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=28 unmarshal_ms=27 version_ms=0                                                                          │
│ time="2021-02-23T11:03:31Z" level=info msg="Skipping auto-sync: deletion in progress" application=umbrella                                                                                                                                               │
│ time="2021-02-23T11:03:31Z" level=info msg="No status changes. Skipping patch" application=umbrella                                                                                                                                                      │
│ time="2021-02-23T11:03:31Z" level=info msg="Reconciliation completed" application=umbrella dedup_ms=0 dest-name= dest-namespace=argocd dest-server="https://kubernetes.default.svc" diff_ms=18 fields.level=1 git_ms=29 health_ms=1 live_ms=2 settings_m │
│ s=0 sync_ms=0 time_ms=63                                                                                                                                                                                                                                 │
│ time="2021-02-23T11:03:31Z" level=info msg="getRepoObjs stats" application=umbrella-stack build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=578 unmarshal_ms=578 version_ms=0                                                                  │
│ time="2021-02-23T11:03:31Z" level=info msg="Skipping auto-sync: deletion in progress" application=umbrella-stack                                                                                                                                         │
│ time="2021-02-23T11:03:31Z" level=info msg="Update successful" application=umbrella-stack                                                                                                                                                                │
│ time="2021-02-23T11:03:31Z" level=info msg="Reconciliation completed" application=umbrella-stack dedup_ms=0 dest-name= dest-namespace=argocd dest-server="https://kubernetes.default.svc" diff_ms=7 fields.level=2 git_ms=580 health_ms=0 live_ms=0 sett │
│ ings_ms=0 sync_ms=0 time_ms=633                                                                                                                                                                                                                          │
│ time="2021-02-23T11:03:31Z" level=info msg="Deleting resources" application=umbrella-stack                                                                                                                                                               │
│ time="2021-02-23T11:03:31Z" level=info msg="1 objects remaining for deletion" application=umbrella-stack                                                                                                                                                 │
│ time="2021-02-23T11:04:30Z" level=info msg="Alloc=38329 TotalAlloc=239984 Sys=72720 NumGC=21 Goroutines=565"

Paste the logs from the rollout controller

│ time="2021-02-23T10:54:24Z" level=info msg="Creating event broadcaster"                                                                                                                                                                                  │
│ time="2021-02-23T10:54:24Z" level=info msg="Setting up event handlers"                                                                                                                                                                                   │
│ time="2021-02-23T10:54:24Z" level=info msg="Setting up experiments event handlers"                                                                                                                                                                       │
│ time="2021-02-23T10:54:24Z" level=info msg="Setting up analysis event handlers"                                                                                                                                                                          │
│ time="2021-02-23T10:54:24Z" level=info msg="Waiting for controller's informer caches to sync"                                                                                                                                                            │
│ time="2021-02-23T10:54:24Z" level=info msg="Enqueueing parent of test/api-model2-7df49f6496: Rollout test/api-model2"                                                                                                                                    │
│ time="2021-02-23T10:54:24Z" level=info msg="Enqueueing parent of test/api-model1-78b5c97b64: Rollout test/api-model1"                                                                                                                                    │
│ time="2021-02-23T10:54:24Z" level=info msg="Starting Controllers"                                                                                                                                                                                        │
│ time="2021-02-23T10:54:24Z" level=info msg="Started controller"                                                                                                                                                                                          │
│ time="2021-02-23T10:54:24Z" level=info msg="Starting Service workers"                                                                                                                                                                                    │
│ time="2021-02-23T10:54:24Z" level=info msg="Started Service workers"                                                                                                                                                                                     │
│ time="2021-02-23T10:54:24Z" level=info msg="Starting Rollout workers"                                                                                                                                                                                    │
│ time="2021-02-23T10:54:24Z" level=info msg="Started Rollout workers"                                                                                                                                                                                     │
│ time="2021-02-23T10:54:24Z" level=info msg="Starting Ingress workers"                                                                                                                                                                                    │
│ time="2021-02-23T10:54:24Z" level=info msg="Started Ingress workers"                                                                                                                                                                                     │
│ time="2021-02-23T10:54:24Z" level=info msg="Starting Experiment workers"                                                                                                                                                                                 │
│ time="2021-02-23T10:54:24Z" level=info msg="Started Experiment workers"                                                                                                                                                                                  │
│ time="2021-02-23T10:54:24Z" level=info msg="Starting analysis workers"                                                                                                                                                                                   │
│ time="2021-02-23T10:54:24Z" level=info msg="Started 30 analysis workers"                                                                                                                                                                                 │
│ time="2021-02-23T10:54:24Z" level=info msg="Started syncing rollout at (2021-02-23 10:54:24.551855 +0000 UTC m=+0.466125301)" namespace=test rollout=api-model2                                                                                          │
│ time="2021-02-23T10:54:24Z" level=info msg="Starting Metric Server at 0.0.0.0:8090"                                                                                                                                                                      │
│ time="2021-02-23T10:54:24Z" level=info msg="Started syncing rollout at (2021-02-23 10:54:24.5535611 +0000 UTC m=+0.467836201)" namespace=test rollout=api-model1                                                                                         │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling stable ReplicaSet 'api-model1-78b5c97b64'" namespace=test rollout=api-model1                                                                                                                     │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling new ReplicaSet 'api-model1-78b5c97b64'" namespace=test rollout=api-model1                                                                                                                        │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling pause" namespace=test rollout=api-model1                                                                                                                                                         │
│ time="2021-02-23T10:54:24Z" level=info msg="No status changes. Skipping patch" generation=3 namespace=test resourceVersion=3215949 rollout=api-model1                                                                                                    │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciliation completed" namespace=test rollout=api-model1 time_ms=28.8951                                                                                                                                  │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling stable ReplicaSet 'api-model2-7df49f6496'" namespace=test rollout=api-model2                                                                                                                     │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling new ReplicaSet 'api-model2-7df49f6496'" namespace=test rollout=api-model2                                                                                                                        │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling pause" namespace=test rollout=api-model2                                                                                                                                                         │
│ time="2021-02-23T10:54:24Z" level=info msg="No status changes. Skipping patch" generation=3 namespace=test resourceVersion=3215951 rollout=api-model2                                                                                                    │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciliation completed" namespace=test rollout=api-model2 time_ms=33.103699999999996                                                                                                                       │
│ time="2021-02-23T10:54:24Z" level=info msg="Started syncing rollout at (2021-02-23 10:54:24.5929872 +0000 UTC m=+0.507257801)" namespace=test rollout=api-model1                                                                                         │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling stable ReplicaSet 'api-model1-78b5c97b64'" namespace=test rollout=api-model1                                                                                                                     │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling new ReplicaSet 'api-model1-78b5c97b64'" namespace=test rollout=api-model1                                                                                                                        │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling pause" namespace=test rollout=api-model1                                                                                                                                                         │
│ time="2021-02-23T10:54:24Z" level=info msg="Started syncing rollout at (2021-02-23 10:54:24.5949637 +0000 UTC m=+0.509232801)" namespace=test rollout=api-model2                                                                                         │
│ time="2021-02-23T10:54:24Z" level=info msg="No status changes. Skipping patch" generation=3 namespace=test resourceVersion=3215949 rollout=api-model1                                                                                                    │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciliation completed" namespace=test rollout=api-model1 time_ms=2.8571                                                                                                                                   │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling stable ReplicaSet 'api-model2-7df49f6496'" namespace=test rollout=api-model2                                                                                                                     │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling new ReplicaSet 'api-model2-7df49f6496'" namespace=test rollout=api-model2                                                                                                                        │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciling pause" namespace=test rollout=api-model2                                                                                                                                                         │
│ time="2021-02-23T10:54:24Z" level=info msg="No status changes. Skipping patch" generation=3 namespace=test resourceVersion=3215951 rollout=api-model2                                                                                                    │
│ time="2021-02-23T10:54:24Z" level=info msg="Reconciliation completed" namespace=test rollout=api-model2 time_ms=1.5640999999999998              

Thank you, cheers!

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[jessesuen]

SharedResourceWarning is a warning you have two Argo CD applications managing the same set of resources. It becomes a problem if the two applications have different opinions about the spec of those resources, in which case, the last application to sync the resource "wins".

In any case, it's unrelated to Rollouts so transferred the issue

[rawc0der]

Hi,
Thanks for answering,

I found out that in this context, 2 different specs conflicting for the same resource did not actually resolve to the last one syncing.
Because both resources had finalizers set (I tried deleting the resources with --cascade as well), both apps were in Unknown state without anything happening.
Managed to recover only by patching the resources and removing the finalizers, which is the opposite of what the finalizers do in the first place.
Will try to see for what types of changes to the "original" resource this kind of problem arises and share my findings.
Thanks

[FireDrunk]

We've come across this issue, and after some debugging found out that on the spec of a Job we had a different app.kubernetes.io/instance annotation than on the pod template spec. That caused the error in question.

Leaving this here for future reference on googling people :)