Investigate getting infrastructure metrics and logs using OTel collectors

[petewall]

Perhaps something in the values file like:

collector: grafana-agent
OR
collector: otel-collector

And that controls the dependent-chart and the generated config file.

[petewall]

Some technical concerns:
To get logs, per these instructions, we need a daemonset, so there's a collector on each node. However, for metrics, having multiple collectors means the metric endpoints will be scraped multiple times

Also need to see how to run both pod logs and cluster events at the same time (since they both go to Loki).

[petewall]

Also, cluster events needs to set a job label.

[petewall]

Proposal to use the K8s API to get logs, rather than requiring a daemonset: open-telemetry/opentelemetry-collector-contrib#24641

This would be nice, because it'd match what grafana agent is doing.