new multi-source app diff cli feature: no diff when it should

[thomas-riccardi]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I tried the new multi-source app diff feature from #17650, and the diff is unexpectedly empty with --revisions and --source-positions, whereas we do have the correct diff with --revision.

On a private App with 3 sources, the diff is even stranger: as if it completely removed the resources for the first source (it's a helm release with value.yaml file from the 3rd source, using ref).

To Reproduce

setup:

• argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
• (argocd app diff guestbook shows the full manifest to add, because not yet sync: OK)
• argocd app sync guestbook
• (argocd app diff guestbook shows empty diff: OK)

tests: with an older revision:

$ argocd app diff guestbook  --revision fd70f16a43d8f14626329a4f687bc8f8f689e9f6

===== apps/Deployment default/guestbook-ui ======
--- /tmp/argocd-diff3285957672/guestbook-ui-live.yaml   2024-04-29 13:53:09.561992217 +0200
+++ /tmp/argocd-diff3285957672/guestbook-ui     2024-04-29 13:53:09.561992217 +0200
@@ -1,4 +1,4 @@
-apiVersion: apps/v1
+apiVersion: apps/v1beta2
 kind: Deployment
 metadata:
   annotations:

=> expected

$ argocd app diff guestbook  --revisions fd70f16a43d8f14626329a4f687bc8f8f689e9f6 --source-positions 1
$

=> empty diff

Expected behavior

• I expect to get the same output with --revision xxx vs --revisions xxx --source-positions 1 for mono-source app.
• I expect diff on multi-source app to work as intended: proper diff

side note:

• I expect --revisions xxx --source-positions 2 to raise an error when there is only one source
• argocd app manifests guestbook --revisions fd70f16a43d8f14626329a4f687bc8f8f689e9f6 --source-positions 1 returns the wrong manifest: same as live/HEAD

Version

argocd: v2.11.0-rc2+24ef777
  BuildDate: 2024-04-15T20:05:04Z
  GitCommit: 24ef7775e79a6ec20b441544aacf8d9d727a676f
  GitTreeState: clean
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.11.0-rc2+24ef777
  BuildDate: 2024-04-15T19:43:50Z
  GitCommit: 24ef7775e79a6ec20b441544aacf8d9d727a676f
  GitTreeState: clean
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.3+gf03cc04
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

[thomas-riccardi]

I reproduce the issue with the new rc3:

argocd: v2.11.0-rc3+20fd621
  BuildDate: 2024-04-29T20:18:11Z
  GitCommit: 20fd621aa20d546378d4643862054307773a1dd0
  GitTreeState: clean
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.11.0-rc3+20fd621
  BuildDate: 2024-04-29T19:57:20Z
  GitCommit: 20fd621aa20d546378d4643862054307773a1dd0
  GitTreeState: clean
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.3+gf03cc04
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

[ishitasequeira]

@thomas-riccardi , I am curious to see the app spec you are creating. Based on the step mentioned in setup, the command
argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default would create a single source app and not a multi-source app.

I also just tried the command on release 2.11.0-rc3 locally and below is what I see the diff as post sync:

 % ./dist/argocd app diff guestbook --revisions 4171e73dec711bde9a130398d3cfc93363b16339 --source-positions 1

===== apps/Deployment default/guestbook-helm-guestbook ======
133c133
<   replicas: 4
---
>   replicas: 3

Can you share an example of the application that I could use to see the issue?

[thomas-riccardi]

@ishitasequeira

I am curious to see the app spec you are creating

Indeed my reproduction here came after my initial issue with private app. My hope is that fixing the issue with mono-source apps would also fix my issue with multi-source app.

Here is an anonymized version of my private multi-source app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: foo
  namespace: argocd
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ApplicationSet
    name: foo-appset
spec:
  destination:
    namespace: foo
    server: https://kubernetes.default.svc
  project: myproject
  sources:
  - helm:
      releaseName: my-helm-release
      valueFiles:
      - $values/some/path/to/value.yaml
      values: |
        more: values
    path: charts/some-chart/
    repoURL: https://github.com/bar/foo-helm-charts
    targetRevision: main
  - path: /baz
    repoURL: https://github.com/bar/other-resources-to-load
    targetRevision: main
  - ref: values
    repoURL: https://github.com/bar/foo-values
    targetRevision: main
  syncPolicy:
    syncOptions:
    - CreateNamespace=true

---

I also just tried the command on release 2.11.0-rc3 locally and below is what I see the diff as post sync:

Maybe it's related to the way we deploy our argocd? (with the helm chart); so I tried with kind locally:

$ kind version
kind v0.22.0 go1.20.13 linux/amd64
$ kind create cluster
$ kubectl get node
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   22s   v1.29.2
$ kubectl apply -n argocd -f https://github.com/argoproj/argo-cd/raw/master/manifests/install.yaml
$ kubectl port-forward svc/argocd-server -n argocd 8080:443 &
$ argocd admin initial-password -n argocd
$ argocd login localhost:8080
# I then saw it was not the rc3, so I (down?)graded to 2.11.0-rc3:
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.11.0-rc3/manifests/install.yaml
# wait pod ready

=> Then I reproduce as in my initial comment: result: I still get the bug: no diff with the new options.

[jon-rei]

On a private App with 3 sources, the diff is even stranger: as if it completely removed the resources for the first source (it's a helm release with value.yaml file from the 3rd source, using ref).

I just upgraded to ArgoCD v2.11 and wanted to try out this new feature and I'm facing the same problem. In ArgoCD I have several apps with two sources, one a helm chart and the other values in a git repository. When I run argocd app diff my-app --revisions 1234 --source-positions 1 --revisions 1234 --source-positions 2 the diff shows me that everything is removed (the revisions I tested against where some where no changes should be expected).

My application spec is basically looking the same as the one posted by @thomas-riccardi, except I only use one additional repoURL source.

[jon-rei]

Unfortunately, the problem still exists with the latest ArgoCD v2.11.1. I also tested it with the guestbook single source application above and found the following:

It seems that the --revisions and --source-positions do not change the actual commit SHA compared to the deployed state in the diff. You can see this quite clearly by comparing it to the "old" --revision.

What I did:

1. Deployed the guestbook app: argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
2. Changed the replicas manually from 1 to 2
3. Executed: argocd app diff guestbook --revisions b0decab622473b6526a7025b2b52e2a6301dd83a --source-positions 1(Notice that the revisions is different to HEAD!)

===== apps/Deployment default/guestbook-ui ======
109c109
<   replicas: 2
---
>   replicas: 1

4. Executed the "old" diff check: argocd app diff guestbook --revision b0decab622473b6526a7025b2b52e2a6301dd83a

===== /Service default/guestbook-ui ======
1,56d0
< apiVersion: v1
< kind: Service
< metadata:
<   annotations:
<     kubectl.kubernetes.io/last-applied-configuration: |
<       {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"guestbook"},"name":"guestbook-ui","namespace":"default"},"spec":{"ports":[{"port":80,"targetPort":80}],"selector":{"app":"guestbook-ui"}}}
<   labels:
<     argocd.argoproj.io/instance: guestbook
<   managedFields:
<   - apiVersion: v1
<     fieldsType: FieldsV1
<     fieldsV1:
<       f:metadata:
<         f:annotations:
<           .: {}
<           f:kubectl.kubernetes.io/last-applied-configuration: {}
<         f:labels:
<           .: {}
<           f:argocd.argoproj.io/instance: {}
<       f:spec:
<         f:internalTrafficPolicy: {}
<         f:ports:
<           .: {}
<           k:{"port":80,"protocol":"TCP"}:
<             .: {}
<             f:port: {}
<             f:protocol: {}
<             f:targetPort: {}
<         f:selector: {}
<         f:sessionAffinity: {}
<         f:type: {}
<     manager: argocd-controller
<     operation: Update
<     time: "2024-05-22T07:27:42Z"
<   name: guestbook-ui
<   namespace: default
<   resourceVersion: "612483492"
<   uid: d326e40a-8ae6-4400-8085-b293e7d858ba
< spec:
<   clusterIP: 10.236.108.68
<   clusterIPs:
<   - 10.236.108.68
<   internalTrafficPolicy: Cluster
<   ipFamilies:
<   - IPv4
<   ipFamilyPolicy: SingleStack
<   ports:
<   - port: 80
<     protocol: TCP
<     targetPort: 80
<   selector:
<     app: guestbook-ui
<   sessionAffinity: None
<   type: ClusterIP
< status:
<   loadBalancer: {}

===== apps/Deployment default/guestbook-ui ======
1c1
< apiVersion: apps/v1
---
> apiVersion: apps/v1beta2
109c109
<   replicas: 2
---
>   replicas: 1

In this example, you can see that the new diff feature does not use the different commit SHA.

@ishitasequeira, in your example above, did you also run the diff command with a different commit SHA than the one from HEAD? Also, can you take a look at this issue

[tspadi]

Hi there,
I have a similar experience here with argocd v2.11.2+25f7504 (cli and server)
with a multi-source app (2 sources, 6 manifests from source 1 and 1 from source 2)
1/ argocd app manifests ns/app => show all (complete) live manifests (7 manifests originating from both sources, master ref)
2/ argocd app manifests ns/app --revision master => would expect same output as 1/ but only shows partial manifests originating from second source only (1 manifest)
3/ argocd app manifests ns/app --revisions master --source-positions 1 --revisions master --source-positions 2 => same partial output as 2/, would expect all 7 manifests instead
4/ argocd app manifests ns/app --revisions master --source-positions 1 => same partial output as 2/, would expect 6 manifests originating from source 1
5/ argocd app manifests ns/app --revisions master --source-positions 2 => same partial output as 2/, that's the only case I would be expecting the 1 manifest originating from second source, as asked.

This has consequences on the argocd app diff command too as it also relies on GetManifests function

[jon-rei]

I just updated to the newest version which was released today v2.11.3 the problem is fixed for me.
From the changelog I think #18395 has fixed the problem.

[tspadi]

It seems to work now with v2.11.3, thanks

[ishitasequeira]

@tspadi , thanks for confirming. This should be fixed by #18395 and is available in v2.11.3