deployment_environment / deployment.environment

[znd4]

the problem

Hi, I've found this to be a bit of a tricky thing to implement piecemeal, so it should probably in the chart proper -- OTEL has a "deployment environment" resource attribute, and grafana's prometheus and loki integrations seem to support some variation of deployment_environment labels.

proposed solution

Add a deploymentEnvironment option that adds discovery relabeling rules to the log service, and deployment.environment / deployment.environment.name to all of the otel objects in the main alloy collector.

caveat

I guess some orgs might have multiple "environments" in a single cluster, but my guess is that most clusters that k8s-monitoring would be installed into will be a part of at most one. With that in mind though, this should probably be optional