Provide a way to gather and store server side telemetry data

[TinaHeiligers]

Description
Provide a way to gather and store server-side telemetric data. The methods should allow for basic data gathering (e.g. a counter or a server-side event), aggregating and storing data in Saved Objects or Elasticsearch and have an easy to use API.

Describe a specific use case for the feature:
While retrieving and shipping telemetric data is disabled when telemetry is disabled, data gathering and aggregation isn't. This means that we're storing data that we're ultimately not using, which is a waste.

We want to stop collecting that data when telemetry is disabled, without having to manually go through each task and check on the telemetry status first.

Why is this needed?
At the moment, there isn't a standardized approach to gathering and storing data for telemetry purposes. This means that every time we want to start storing and shipping new server-side telemetric data, teams need to instrument the code themselves, aggregate it and save that data somewhere.

[elasticmachine]

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

[TinaHeiligers]

@Bamieh @afharo I'd love to discuss what the API would look like but for now, I was thinking along the lines of creating, editing (through aggregation) and deleting. e.g a way to delete the data we've already stored after it's been shipped to the remote service or if the opt-out status changes and we don't need the data anymore. The usage collector's fetch method will handle reading the data from where it is stored.

[afharo]

I love this proposal! Thank you Tina for creating this issue so we can tackle it! I sincerely think that we need to provide an easy way for other plugins to count events.

As far as I understand, I can see the server-side telemetry in 2 ways:

1. The current situation: The plugin registers the fetch method that is called when the telemetry payload is about to be sent to the server. Usually, the implementation in that fetch method can resolve count of objects (like visualisations, map layers, number of registered alerts, ...), or config parameters we may want to track (i.e.: certain features are enabled, the default behaviour is overwritten, ...).
2. Count of events (i.e.: an API has been used, an alert has been registered/disabled/enabled/triggered).

The first use case is very ad-hoc and we need to maintain the flexibility of the fetch method, but the 2nd use case seems fairly straight-forward and common to me. In this case, I would propose a 2-step process:

1. Register the counter: during the plugin's setup lifecycle method, it would call usageCollection.registerCounter('APP_ID', ['NAME_OF_COUNTER_1', 'NAME_OF_COUNTER_2', ...], counterFormatter?), so we know we have to maintain a count of those ones (internally, we'll ensure the related SOs exist and they have, at least, the zero-values). counterFormatter is an optional method that would be called when retrieving the telemetry payload and it would receive the values { 'NAME_OF_COUNTER_1': { total, last7days, last30days, last90days }, 'NAME_OF_COUNTER_2': { total, last7days, last30days, last90days } } and can modify the payload into any other output (i.e.: flatten, make any math operation with those counters to come up with any other one, ...).
2. Increase the counter: In the start contract, the plugin will be able to call usageCollection.increaseCounter('APP_ID', 'NAME_OF_COUNTER', byThisAmount?) (if byThisAmount is not provided, it will increase it by 1) every time the event happens.

Later on, when the telemetry payload needs to be sent to the server, a general collector in kibana_usage_collection or usage_collection will bundle up all the registered counters, reporting their total and last 7/30/90 days counts.

Possible further improvements:

• If we move usageCollection to core, we will know which plugin ID is calling the API and we can scope the request automatically (no need for the APP_ID parameter).
• If counterFormatter is provided in the method registerCounter, we could also request the schema to know it beforehand for the benefit of the remote telemetry service.

I think that implementing these APIs can help us with UI metrics and Application Usage as well.

What do you think?

[afharo]

I think this could follow the same structure as UI Counters does.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)