feat: Multiple sources for applications (argoproj#2789) (argoproj#10432)

* feat: support multiple sources for application Signed-off-by: ishitasequeira <[email protected]> remove debug logging and unwanted code Signed-off-by: ishitasequeira <[email protected]> fix lint and unit test errors Signed-off-by: ishitasequeira <[email protected]> fix lint and unit test errors Signed-off-by: ishitasequeira <[email protected]> * fix bug introduced after rebase Signed-off-by: ishitasequeira <[email protected]> executed make codegen Signed-off-by: ishitasequeira <[email protected]> * rebase with master Signed-off-by: ishitasequeira <[email protected]> remove unwanted logging Signed-off-by: ishitasequeira <[email protected]> fix ci failures Signed-off-by: ishitasequeira <[email protected]> * fix index out of bounds error Signed-off-by: ishitasequeira <[email protected]> * ui fixes Signed-off-by: ishitasequeira <[email protected]> Signed-off-by: Michael Crenshaw <[email protected]> Signed-off-by: ishitasequeira <[email protected]> * add revisions to SyncOperation for rollback Signed-off-by: ishitasequeira <[email protected]> * change Source to *ApplicationSource in ApplicationSpec Signed-off-by: ishitasequeira <[email protected]> * fix env variable read logic for ValueFiles Signed-off-by: ishitasequeira <[email protected]> * update multiple sources doc Signed-off-by: ishitasequeira <[email protected]> * add repository lock and checkout target revision Signed-off-by: ishitasequeira <[email protected]> fix codegen Signed-off-by: ishitasequeira <[email protected]> * checkout all sources before generating manifest Signed-off-by: ishitasequeira <[email protected]> generate mock reposerverclient Signed-off-by: ishitasequeira <[email protected]> * update logic for returning ManifestResponse to avoid nil pointer issues Signed-off-by: ishitasequeira <[email protected]> * fix nil reference and key mismatch bugs; add more logs (argoproj#6) * fix nil reference and key mismatch bugs; add more logs * remove temporary comment * addressed the lint failure and added chart to RefTargeRevisionMapping * normalize git repo (argoproj#7) * do not leak lock releases * prevent deadlock * allow spec update * move settings fetch outside loop * cache busing * return err instead of logging it * no caching in test * fix cache key marshaling Signed-off-by: Michael Crenshaw <[email protected]> Rebase with master Signed-off-by: ishitasequeira <[email protected]> * update grpc field numbers Signed-off-by: ishitasequeira <[email protected]> * path resolution tests (argoproj#12) Signed-off-by: Michael Crenshaw <[email protected]> do things in better ways Signed-off-by: Michael Crenshaw <[email protected]> consolidate Signed-off-by: Michael Crenshaw <[email protected]> Signed-off-by: Michael Crenshaw <[email protected]> Signed-off-by: ishitasequeira <[email protected]> * add regex check for value of source.ref Signed-off-by: ishitasequeira <[email protected]> * add webhook tests Signed-off-by: Michael Crenshaw <[email protected]> change Source to *ApplicationSource in ApplicationSpec Signed-off-by: ishitasequeira <[email protected]> address PR comments Signed-off-by: ishitasequeira <[email protected]> rebase with master Signed-off-by: ishitasequeira <[email protected]> Retrigger CI pipeline Signed-off-by: ishitasequeira <[email protected]> rebased with master Signed-off-by: ishitasequeira <[email protected]> * fix env variable read logic for ValueFiles Signed-off-by: ishitasequeira <[email protected]> Address PR comments Signed-off-by: ishitasequeira <[email protected]> * add repository lock and checkout target revision Signed-off-by: ishitasequeira <[email protected]> fix codegen Signed-off-by: ishitasequeira <[email protected]> * checkout all sources before generating manifest Signed-off-by: ishitasequeira <[email protected]> * generate mock reposerverclient Signed-off-by: ishitasequeira <[email protected]> * address comments Signed-off-by: ishitasequeira <[email protected]> * update logic for returning ManifestResponse to avoid nil pointer issues Signed-off-by: ishitasequeira <[email protected]> * fix nil reference and key mismatch bugs; add more logs (argoproj#6) * fix nil reference and key mismatch bugs; add more logs * remove temporary comment * addressed the lint failure and added chart to RefTargeRevisionMapping * normalize git repo (argoproj#7) * do not leak lock releases * prevent deadlock * allow spec update * move settings fetch outside loop * cache busing * return err instead of logging it * no caching in test * fix cache key marshaling Signed-off-by: Michael Crenshaw <[email protected]> Rebase with master Signed-off-by: ishitasequeira <[email protected]> * update grpc field numbers Signed-off-by: ishitasequeira <[email protected]> * add regex check for value of source.ref Signed-off-by: ishitasequeira <[email protected]> * Rebase with master Signed-off-by: ishitasequeira <[email protected]> * Added unit tests (argoproj#15) * add unit tests 1 * fix lint Signed-off-by: ishitasequeira <[email protected]> * fix application parameters tab, rebased UI changes, tests Signed-off-by: ishitasequeira <[email protected]> * More tests (argoproj#16) * more tests Signed-off-by: Michael Crenshaw <[email protected]> fix lint error Signed-off-by: ishitasequeira <[email protected]> Test get ref sources (argoproj#17) * test GetRefSources Signed-off-by: Michael Crenshaw <[email protected]> * fix lint Signed-off-by: Michael Crenshaw <[email protected]> Signed-off-by: ishitasequeira <[email protected]> Signed-off-by: Michael Crenshaw <[email protected]> Co-authored-by: ishitasequeira <[email protected]> GenerateManifests test (argoproj#18) * GenerateManifests test Signed-off-by: Michael Crenshaw <[email protected]> Fix broken tests (argoproj#19) * fix broken tests Signed-off-by: Michael Crenshaw <[email protected]> Symlink test (argoproj#20) * check referenced sources for out-of-bounds symlinks Signed-off-by: Michael Crenshaw <[email protected]> * unlock the values file repo before doing a symlink check (argoproj#22) Signed-off-by: Michael Crenshaw <[email protected]> * multi source docs (argoproj#21) * multi source docs Signed-off-by: Michael Crenshaw <[email protected]> * fix warning title Signed-off-by: Michael Crenshaw <[email protected]> * clarify Signed-off-by: Michael Crenshaw <[email protected]> * clarify Signed-off-by: Michael Crenshaw <[email protected]> Signed-off-by: Michael Crenshaw <[email protected]> * add e2e tests for multiple sources and fix UI lint (argoproj#23) * add e2e tests for multiple sources and fix UI lint Signed-off-by: ishitasequeira <[email protected]> * add auto-sync and hard refresh to e2e tests Signed-off-by: ishitasequeira <[email protected]> * change refresh type to RefreshTypeNormal for e2e Signed-off-by: ishitasequeira <[email protected]> * update e2e testcase with helm data Signed-off-by: ishitasequeira <[email protected]> * add TestMultiSourceAppWithSourceOverride Signed-off-by: ishitasequeira <[email protected]> * add missing yaml file Signed-off-by: ishitasequeira <[email protected]> Signed-off-by: ishitasequeira <[email protected]> * rebase with master Signed-off-by: ishitasequeira <[email protected]> * rebase with master Signed-off-by: ishitasequeira <[email protected]> * fix lint Signed-off-by: ishitasequeira <[email protected]> Signed-off-by: ishitasequeira <[email protected]> Signed-off-by: Michael Crenshaw <[email protected]> Signed-off-by: Michael Crenshaw <[email protected]> Co-authored-by: Michael Crenshaw <[email protected]> Signed-off-by: emirot <[email protected]>
emirot · Jan 27, 2023 · 80e53e6 · 80e53e6
1 parent 9ef3bec
commit 80e53e6
Show file tree

Hide file tree

Showing 90 changed files with 27,950 additions and 4,186 deletions.
diff --git a/applicationset/controllers/applicationset_controller_test.go b/applicationset/controllers/applicationset_controller_test.go
@@ -684,7 +684,7 @@ func TestCreateOrUpdateInCluster(t *testing.T) {
 					Template: argov1alpha1.ApplicationSetTemplate{
 						Spec: argov1alpha1.ApplicationSpec{
 							Project:     "project",
-							Source:      argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
+							Source:      &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
 							Destination: argov1alpha1.ApplicationDestination{Server: "server", Namespace: "namespace"},
 						},
 					},
@@ -721,7 +721,7 @@ func TestCreateOrUpdateInCluster(t *testing.T) {
 					},
 					Spec: argov1alpha1.ApplicationSpec{
 						Project:     "project",
-						Source:      argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
+						Source:      &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
 						Destination: argov1alpha1.ApplicationDestination{Server: "server", Namespace: "namespace"},
 					},
 				},
@@ -741,7 +741,7 @@ func TestCreateOrUpdateInCluster(t *testing.T) {
 					},
 					Spec: argov1alpha1.ApplicationSpec{
 						Project:     "project",
-						Source:      argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
+						Source:      &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
 						Destination: argov1alpha1.ApplicationDestination{Server: "server", Namespace: "namespace"},
 					},
 					Status: argov1alpha1.ApplicationStatus{
@@ -918,7 +918,7 @@ func TestRemoveFinalizerOnInvalidDestination_FinalizerTypes(t *testing.T) {
 				},
 				Spec: argov1alpha1.ApplicationSpec{
 					Project: "project",
-					Source:  argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
+					Source:  &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
 					// Destination is always invalid, for this test:
 					Destination: argov1alpha1.ApplicationDestination{Name: "my-cluster", Namespace: "namespace"},
 				},
@@ -1080,7 +1080,7 @@ func TestRemoveFinalizerOnInvalidDestination_DestinationTypes(t *testing.T) {
 				},
 				Spec: argov1alpha1.ApplicationSpec{
 					Project:     "project",
-					Source:      argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
+					Source:      &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
 					Destination: c.destinationField,
 				},
 			}
@@ -1589,7 +1589,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
 					ObjectMeta: metav1.ObjectMeta{},
 					Spec: argov1alpha1.ApplicationSpec{
 						Project: "default",
-						Source: argov1alpha1.ApplicationSource{
+						Source: &argov1alpha1.ApplicationSource{
 							RepoURL:        "https://url",
 							Path:           "/",
 							TargetRevision: "HEAD",
@@ -1612,7 +1612,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
 					ObjectMeta: metav1.ObjectMeta{},
 					Spec: argov1alpha1.ApplicationSpec{
 						Project: "default",
-						Source: argov1alpha1.ApplicationSource{
+						Source: &argov1alpha1.ApplicationSource{
 							RepoURL:        "https://url",
 							Path:           "/",
 							TargetRevision: "HEAD",
@@ -1636,7 +1636,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
 					ObjectMeta: metav1.ObjectMeta{},
 					Spec: argov1alpha1.ApplicationSpec{
 						Project: "DOES-NOT-EXIST",
-						Source: argov1alpha1.ApplicationSource{
+						Source: &argov1alpha1.ApplicationSource{
 							RepoURL:        "https://url",
 							Path:           "/",
 							TargetRevision: "HEAD",
@@ -1659,7 +1659,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
 					ObjectMeta: metav1.ObjectMeta{},
 					Spec: argov1alpha1.ApplicationSpec{
 						Project: "default",
-						Source: argov1alpha1.ApplicationSource{
+						Source: &argov1alpha1.ApplicationSource{
 							RepoURL:        "https://url",
 							Path:           "/",
 							TargetRevision: "HEAD",
@@ -1682,7 +1682,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
 					ObjectMeta: metav1.ObjectMeta{},
 					Spec: argov1alpha1.ApplicationSpec{
 						Project: "default",
-						Source: argov1alpha1.ApplicationSource{
+						Source: &argov1alpha1.ApplicationSource{
 							RepoURL:        "https://url",
 							Path:           "/",
 							TargetRevision: "HEAD",
@@ -1811,7 +1811,7 @@ func TestReconcilerValidationErrorBehaviour(t *testing.T) {
 					Namespace: "argocd",
 				},
 				Spec: argov1alpha1.ApplicationSpec{
-					Source:      argov1alpha1.ApplicationSource{RepoURL: "https://github.com/argoproj/argocd-example-apps", Path: "guestbook"},
+					Source:      &argov1alpha1.ApplicationSource{RepoURL: "https://github.com/argoproj/argocd-example-apps", Path: "guestbook"},
 					Project:     "default",
 					Destination: argov1alpha1.ApplicationDestination{Server: "{{.url}}"},
 				},
@@ -1954,7 +1954,7 @@ func TestGenerateAppsUsingPullRequestGenerator(t *testing.T) {
 					},
 				},
 				Spec: argov1alpha1.ApplicationSpec{
-					Source: argov1alpha1.ApplicationSource{
+					Source: &argov1alpha1.ApplicationSource{
 						RepoURL:        "https://testurl/testRepo",
 						TargetRevision: "{{.head_short_sha}}",
 					},
@@ -1973,7 +1973,7 @@ func TestGenerateAppsUsingPullRequestGenerator(t *testing.T) {
 						},
 					},
 					Spec: v1alpha1.ApplicationSpec{
-						Source: v1alpha1.ApplicationSource{
+						Source: &v1alpha1.ApplicationSource{
 							RepoURL:        "https://testurl/testRepo",
 							TargetRevision: "089d92cb",
 						},
@@ -2113,7 +2113,7 @@ func TestPolicies(t *testing.T) {
 							},
 						},
 						Spec: argov1alpha1.ApplicationSpec{
-							Source:      argov1alpha1.ApplicationSource{RepoURL: "https://github.com/argoproj/argocd-example-apps", Path: "guestbook"},
+							Source:      &argov1alpha1.ApplicationSource{RepoURL: "https://github.com/argoproj/argocd-example-apps", Path: "guestbook"},
 							Project:     "default",
 							Destination: argov1alpha1.ApplicationDestination{Server: "https://kubernetes.default.svc"},
 						},

diff --git a/applicationset/utils/utils_test.go b/applicationset/utils/utils_test.go
@@ -41,7 +41,7 @@ func TestRenderTemplateParams(t *testing.T) {
 			Namespace:         "default",
 		},
 		Spec: argoappsv1.ApplicationSpec{
-			Source: argoappsv1.ApplicationSource{
+			Source: &argoappsv1.ApplicationSource{
 				Path:           "",
 				RepoURL:        "",
 				TargetRevision: "",
@@ -220,7 +220,7 @@ func TestRenderTemplateParamsGoTemplate(t *testing.T) {
 			Namespace:         "default",
 		},
 		Spec: argoappsv1.ApplicationSpec{
-			Source: argoappsv1.ApplicationSource{
+			Source: &argoappsv1.ApplicationSource{
 				Path:           "",
 				RepoURL:        "",
 				TargetRevision: "",
@@ -511,7 +511,7 @@ func TestRenderTemplateParamsFinalizers(t *testing.T) {
 
 	emptyApplication := &argoappsv1.Application{
 		Spec: argoappsv1.ApplicationSpec{
-			Source: argoappsv1.ApplicationSource{
+			Source: &argoappsv1.ApplicationSource{
 				Path:           "",
 				RepoURL:        "",
 				TargetRevision: "",

diff --git a/assets/swagger.json b/assets/swagger.json
@@ -5831,6 +5831,10 @@
         "plugin": {
           "$ref": "#/definitions/v1alpha1ApplicationSourcePlugin"
         },
+        "ref": {
+          "description": "Ref is reference to another source within sources field. This field will not be used if used with a `source` tag.",
+          "type": "string"
+        },
         "repoURL": {
           "type": "string",
           "title": "RepoURL is the URL to the repository (Git or Helm) that contains the application manifests"
@@ -6068,6 +6072,13 @@
         "source": {
           "$ref": "#/definitions/v1alpha1ApplicationSource"
         },
+        "sources": {
+          "type": "array",
+          "title": "Sources is a reference to the location of the application's manifests or chart",
+          "items": {
+            "$ref": "#/definitions/v1alpha1ApplicationSource"
+          }
+        },
         "syncPolicy": {
           "$ref": "#/definitions/v1alpha1SyncPolicy"
         }
@@ -6118,6 +6129,13 @@
           "type": "string",
           "title": "SourceType specifies the type of this application"
         },
+        "sourceTypes": {
+          "type": "array",
+          "title": "SourceTypes specifies the type of the sources included in the application",
+          "items": {
+            "type": "string"
+          }
+        },
         "summary": {
           "$ref": "#/definitions/v1alpha1ApplicationSummary"
         },
@@ -6415,6 +6433,13 @@
         },
         "source": {
           "$ref": "#/definitions/v1alpha1ApplicationSource"
+        },
+        "sources": {
+          "type": "array",
+          "title": "Sources is a reference to the application's multiple sources used for comparison",
+          "items": {
+            "$ref": "#/definitions/v1alpha1ApplicationSource"
+          }
         }
       }
     },
@@ -7757,8 +7782,22 @@
           "type": "string",
           "title": "Revision holds the revision the sync was performed against"
         },
+        "revisions": {
+          "type": "array",
+          "title": "Revisions holds the revision of each source in sources field the sync was performed against",
+          "items": {
+            "type": "string"
+          }
+        },
         "source": {
           "$ref": "#/definitions/v1alpha1ApplicationSource"
+        },
+        "sources": {
+          "type": "array",
+          "title": "Sources is a reference to the application sources used for the sync operation",
+          "items": {
+            "$ref": "#/definitions/v1alpha1ApplicationSource"
+          }
         }
       }
     },
@@ -8059,9 +8098,23 @@
           "description": "Revision is the revision (Git) or chart version (Helm) which to sync the application to\nIf omitted, will use the revision specified in app spec.",
           "type": "string"
         },
+        "revisions": {
+          "description": "Revisions is the list of revision (Git) or chart version (Helm) which to sync each source in sources field for the application to\nIf omitted, will use the revision specified in app spec.",
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
         "source": {
           "$ref": "#/definitions/v1alpha1ApplicationSource"
         },
+        "sources": {
+          "type": "array",
+          "title": "Sources overrides the source definition set in the application.\nThis is typically set in a Rollback operation and is nil during a Sync operation",
+          "items": {
+            "$ref": "#/definitions/v1alpha1ApplicationSource"
+          }
+        },
         "syncOptions": {
           "type": "array",
           "title": "SyncOptions provide per-sync sync-options, e.g. Validate=false",
@@ -8107,8 +8160,22 @@
           "type": "string",
           "title": "Revision holds the revision this sync operation was performed to"
         },
+        "revisions": {
+          "type": "array",
+          "title": "Revisions holds the revision this sync operation was performed for respective indexed source in sources field",
+          "items": {
+            "type": "string"
+          }
+        },
         "source": {
           "$ref": "#/definitions/v1alpha1ApplicationSource"
+        },
+        "sources": {
+          "type": "array",
+          "title": "Source records the application source information of the sync, used for comparing auto-sync",
+          "items": {
+            "$ref": "#/definitions/v1alpha1ApplicationSource"
+          }
         }
       }
     },
@@ -8163,6 +8230,13 @@
           "type": "string",
           "title": "Revision contains information about the revision the comparison has been performed to"
         },
+        "revisions": {
+          "type": "array",
+          "title": "Revisions contains information about the revisions of multiple sources the comparison has been performed to",
+          "items": {
+            "type": "string"
+          }
+        },
         "status": {
           "type": "string",
           "title": "Status is the sync state of the comparison"

diff --git a/cmd/argocd/commands/admin/app.go b/cmd/argocd/commands/admin/app.go
@@ -401,7 +401,12 @@ func reconcileApplications(
 			return nil, err
 		}
 
-		res := appStateManager.CompareAppState(&app, proj, app.Spec.Source.TargetRevision, app.Spec.Source, false, false, nil)
+		sources := make([]v1alpha1.ApplicationSource, 0)
+		revisions := make([]string, 0)
+		sources = append(sources, app.Spec.GetSource())
+		revisions = append(revisions, app.Spec.GetSource().TargetRevision)
+
+		res := appStateManager.CompareAppState(&app, proj, revisions, sources, false, false, nil, false)
 		items = append(items, appReconcileResult{
 			Name:       app.Name,
 			Conditions: app.Status.Conditions,

diff --git a/cmd/argocd/commands/admin/app_test.go b/cmd/argocd/commands/admin/app_test.go
@@ -80,6 +80,7 @@ func TestGetReconcileResults_Refresh(t *testing.T) {
 			Namespace: "default",
 		},
 		Spec: v1alpha1.ApplicationSpec{
+			Source:  &v1alpha1.ApplicationSource{},
 			Project: "default",
 			Destination: v1alpha1.ApplicationDestination{
 				Server:    v1alpha1.KubernetesInternalAPIServerAddr,