Telemetry: last fixes for v1

[adizere]

Proposal

There's a few things left:

• There is an inconsistency between send packets count and acknowledgements count in the displayed metrics.

  • Ensure the documentation on send_packet_count, acknowledgement_count, cleared_send_packet_count and cleared_acknowledgment_count is clear, as these four metrics seem to be difficult to use

• Discovery phase: Some operators do a discovery phase to understand the Prometheus schema, but Hermes upon starting up does not populate most of the metric fields

  • the most important fields that we should initially populate with some zero value would be the backlog_* ones and the ones related to activity (msg_num and send_packet_count for example).

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[AlianBenabdallah]

Discovery phase: Some operators do a discovery phase to understand the Prometheus schema, but Hermes upon starting up does not populate most of the metric fields

• the most important fields that we should initially populate with some zero value would be the backlog_* ones and the ones related to activity (msg_num and send_packet_count for example).

The branch ali/metrics_discovery now implements a discovery phase when exact filters are given in the config file. Every metric should be initialized except for tx_latency_*. I'm still unsure about how to init these metrics as they use histograms.

If no exact filter is given, the relayer needs to query every channel. I'll see if it's possible to do it with one query but I guess that it might hurt performances if there are too many channels.

[AlianBenabdallah]

Regarding the metrics, we have acknowledgment_packets and acknowledgement_count. I think the latest should be renamed as acknowledgement_events.

Also, would it be useful to add a metric timeout_events just like for acknowledgements ?