[Usage Collection] [Meta] Enhance usage collection with richer API's

[TinaHeiligers]

In an effort to simplify the usage data that's collected in Kibana, we are making an effort to improve the existing usage collectors.

Documentation/Helper guides:

• https://github.com/elastic/kibana/tree/master/src/plugins/usage_collection#kibana-usage-collection-service
• https://github.com/elastic/kibana-team/blob/master/telemetry.md#the-hitchhikers-guide-to-telemetry

There are 2 options for capturing and reporting usage data to the telemetry service:

1. Public/Browser side: ui_metrics
2. Server side: usage_collection service

In addition, we want to enrich the APIs:

1. Replace callCluster with { callCluster (with the deprecated flag), esClient (the new one), savedObjects }
2. (to be filled in)

This issue serves as a meta ticket to reference related issues that need to be addressed.

Linked issues:

- [ ] Migrate Usage Collection plugin to Core as a Service (#69306)

[elasticmachine]

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

[afharo]

I’m not sure Standardize is the word we are after. If a piece of telemetry is “standard” for all plugins, we’ve already done it as a general one: like application_usage or custom uiSettings, and any more we can add if needed (like if we want to report every time an element containing a certain HTML property is displayed as part of the application_usage).

The custom UsageCollectors are meant to be used for custom telemetry: for anything that is very plugin-specific.

We can improve our APIs to report event-based telemetry (the repeated 7d/30d/90d/total approach), and I def think we should put some efforts into that. UI Metrics is an example that could benefit from that.

But in both scenarios, as far as I understand, it's just about adding support to event-based telemetry (which we already do with the existing ui_metric API, we simply store it and report it in a way that it's meaningless).

Is there any other type of standardisation we can do?

[TinaHeiligers]

@afharo we need to see if there are custom usage collectors that have similar enough implementations to figure out if we can develop other types of standard/general collectors to suit those needs. If we could offer a customizable general usage collector that other plugins could "tweak" to get their job done and offer that through core too, that would be great! If, however, the custom collectors differ sufficiently from each other, then we can't really offer anything else right now.

[afharo]

I think the only missing bit is #81645. Let's close this issue for the sake of housekeeping :)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)