Applicationset creating Applications on an unexisting project does not get requeued for reconciliation

[zenitraM]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
When an applicationset is created that creates Applications in an unexisting project, it never gets reconciled until the applicationset gets recreated even if the project is created in the meantime.

We have a CI environment in which we bring up an entire environment and frequently create both AppProjects and ApplicationSets using the app-of-apps pattern, where they are created from two different apps-of-apps. This causes that, depending on the reconciliation order of both apps-of-apps, AppSets would never get synced if they happen to be created before the AppProjects.

To Reproduce

• Create an ApplicationSet that creates Applications on a project that does not exist yet.
• The ApplicationSet health ends up in this state:

{"applicationset":{"Namespace":"argocd","Name":"dev-appset"},"level":"error","msg":"validation error found during application validation: application references project my-project which does not exist","time":"2023-06-23T11:00:52Z"}
{"applicationset":{"Namespace":"argocd","Name":"dev-appset"},"level":"info","msg":"end reconcile","requeueAfter":0,"time":"2023-06-23T11:00:52Z"}

• Create the my-project project
• ApplicationSet does not try to reconcile/synchronize again ever, until applicationset-controller is restarted.

Expected behavior
The ApplicationSet controller should at least try to reconcile the application again after some time when it gets into this condition.

There's apparently a set timeout in the code for this to be retried after 3 minutes:

argo-cd/applicationset/controllers/applicationset_controller.go

Lines 129 to 150 in 2170f3a

	validateErrors, err := r.validateGeneratedApplications(ctx, desiredApplications, applicationSetInfo, req.Namespace)
	if err != nil {
	// While some generators may return an error that requires user intervention,
	// other generators reference external resources that may change to cause
	// the error to no longer occur. We thus log the error and requeue
	// with a timeout to give this another shot at a later time.
	//
	// Changes to watched resources will cause this to be reconciled sooner than
	// the RequeueAfter time.
	logCtx.Errorf("error occurred during application validation: %s", err.Error())
	_ = r.setApplicationSetStatusCondition(ctx,
	&applicationSetInfo,
	argov1alpha1.ApplicationSetCondition{
	Type: argov1alpha1.ApplicationSetConditionErrorOccurred,
	Message: err.Error(),
	Reason: argov1alpha1.ApplicationSetReasonApplicationValidationError,
	Status: argov1alpha1.ApplicationSetConditionStatusTrue,
	}, parametersGenerated,
	)
	return ctrl.Result{RequeueAfter: ReconcileRequeueOnValidationError}, nil
	}

but apparently it is only set when an error occurs in the validation evaluation itself, not when one of the validations fails which is the case here:

argo-cd/applicationset/controllers/applicationset_controller.go

Lines 189 to 208 in 2170f3a

	if len(validateErrors) > 0 {
	var message string
	for _, v := range validateErrors {
	message = v.Error()
	logCtx.Errorf("validation error found during application validation: %s", message)
	}
	if len(validateErrors) > 1 {
	// Only the last message gets added to the appset status, to keep the size reasonable.
	message = fmt.Sprintf("%s (and %d more)", message, len(validateErrors)-1)
	}
	_ = r.setApplicationSetStatusCondition(ctx,
	&applicationSetInfo,
	argov1alpha1.ApplicationSetCondition{
	Type: argov1alpha1.ApplicationSetConditionErrorOccurred,
	Message: message,
	Reason: argov1alpha1.ApplicationSetReasonApplicationValidationError,
	Status: argov1alpha1.ApplicationSetConditionStatusTrue,
	}, parametersGenerated,
	)
	}

Version
v2.7.4+a33baa3.dirty