Monitor bridge GRANDPA transactions

[svyatonik]

Related to #2496

#1882 introduces some constants, related to weight and size (soft) limits of pallet-bridge-grandpa transactions. We shall add monitoring stuff (loki + relay logs + alerts OR relay metrics + dashboards + alerts?) to our substrate-relay (or some separate process). The relay must look at all pallet transactions and:

• search transactions with invalid headers in votes_ancestries. If there are such headers, it means that the GRANDPA authority is cheating;
• search transactions with unknown headers in votes_ancestries. Unknown means that we can't find header on other nodes. Such transaction means that the BABE+GRANDPA authority/authorities are possibly cheating;
• search transactions that are above our limits. We may need to increase limits if we see such transactions

[EmmanuellNorbertTulbure]

@svyatonik to recheck

[svyatonik]

search transactions that are above our limits. We may need to increase limits if we see such transactions

There are two limits:

• REASONABLE_HEADERS_IN_JUSTIFICATON_ANCESTRY set to 2. Normally it is zero. If it is larger (given optimal justification), it means that GRANDPA is lagging;
• AVERAGE_HEADER_SIZE_IN_JUSTIFICATION is set to 20kb. Regular Kusama header is ~300b, but on epoch change it is 40Kb;
• MAX_HEADER_SIZE is set to 90_000b.

So we expect to see:

• 2 or less headers in votes ancestry;
• header+justification, which is near to 90_000 + 2*20kb ~ 130_000 bytes.

If we see transaction over those limits, then:

• if that's a standalone submit_finality_proof for mandatory header, the relayer pays regular fee. Normally mandatory headers submission is free;
• if such header+justification is submitted in batch with delivery/confirmation transaction, we partially refund the tx cost. I.e. if there are more than REASONABLE_HEADERS_IN_JUSTIFICATON_ANCESTRY headers in ancestry, we only refund for REASONABLE_HEADERS_IN_JUSTIFICATON_ANCESTRY headers and so on.

So we need a metrics for actual values in source chain headers and if they are above the limits, we'll need to change constants OR ELSE relayers will keep (partially) spending funds. I'd add this either in v1, or soon after the launch. OTOH, we have enough reserve in those constants, so we unlikely to overflow. So maybe "soon after the launch" is a way to go here. @EmmanuellNorbertTulbure Please triage :)