Configuration Management Plugins chaining

[clementnuss]

Summary

It should be possible to chain configuration management plugins (CMP).
That would allow for workflows where a plugin templates some fields in the source manifests (e.g. ytt or a simpler envsubst-like plugin), before another plugin (e.g. Helm or kustomize) does the rendering.

source repository -> CMP plugin #1 (modifier) -> ... -> CMP plugin #n (renderer) -> stdout

Motivation

This would permits more advanced use cases for CMP, and could also greatly increase adoption rate of CMP.
Indeed, it might well be that I want to template some fields of the values.yaml (e.g. to do a DNS lookup, or to concatenate some variables and text together), but that I also need to render the templated values.yaml with Helm.
Currently, no solution to this approach exist, except maybe for my argocd-cmp-ytt plugin where I manually template with ytt before rendering with Helm (from within the config management plugin itself).

Proposal

When defining the plugins in an application spec, we would need a way to specify the order in which to apply plugins.

Also and more importantly, we would need a way to transmit the templated files/repository to the next plugin in the chain, as currently CMP's only output is the rendered manifests.

there might then be plugins of two sorts: modifier plugins, which edit a repository and produce a tar archive, and rendering plugins, which end the chain and render the manifests to stdout

[crenshaw-dev]

Ideally I'd like to make this even more generic than chaining plugins: it would be nice to be able to chain arbitrary sources.

@gczuczy wrote a PR allowing sources to reference other sources' pre-rendered files as plain old directories. #12508

I think the next logical step would be to let sources reference other sources post-rendered contents as a manifests.yaml file.

Aside: I haven't tried it myself, but I understand argocd-lovely-plugin provides a mechanism for chaining plugins. https://github.com/crumbhole/argocd-lovely-plugin