Explore Vector as a log and metric exporter

[achimnol]

Vector (made by Datadog) is a high-performance observability pipeline tool, which can collect and process events, logs, and metrics. It's written in Rust and some of my friends in "big techs" reported that it works successfully in very large-scale deployments.

https://vector.dev/docs/about/concepts/

Let's inspect how we can integrate this into our platform, particularly in on-premise customer sites. Since there are multiple "vector sink" implementation, we could attach Datadog in the cloud while attaching other open-source dashboards in the on-prem setups.

• We could provide some "vector source" configuration samples (e.g., capturing the stderr stream of service daemons).
• We could write additional logging backends in ai.backend.common.logging to feed structured logs to the vector pipeline directly.
• We could extend our stats subsystem in ai.backend.agent.stats to feed metrics to the vector pipeline. If we write our own prometheus exporter interfaces, we could reuse them for the vector sources.
• We could extend Add a init/health check for mountpoints #1639 and our event bus to feed health check events to the vector pipeline.
• We could provide some "vector transform" configuration samples to detect anomalies in the manager and agents.
• We could update our internal dahsboard project to take the role of "vector sinks" to accept and display the processed observability data.
• Q: Could we use Graylog as a sink? What about Grafana Loki as the log viewer?

(taken from https://vector.dev/docs/setup/deployment/)