Prometheus Instrumentation

[onelapahead]

Similar to hyperledger/firefly#189, it'd be great if the tokens connector had Prometheus metrics for API call count and durations, tokens transferred / minted / burned and etc.

We can leverage https://github.com/siimon/prom-client to get some basic metrics added quickly and then build from there.

[matthew1001]

I've started working on this under a PR (#131).

I've used https://github.com/willsoto/nestjs-prometheus which is a NestJS prometheus plugin

Initially I've added the following metrics:

1. ff_apiserver_rest_requests_total (counter)
2. ff_apiserver_rest_request_size_bytes (histogram)
3. ff_apiserver_rest_request_duration_seconds (histogram)
4. ff_event_batch_size (gauge)
5. ff_event_batch_interval_ms (histogram)
6. ff_event_batch_ack_interval_ms (histogram)
7. ff_blockchain_calls_total (counter)

The ff_ prefix is in keeping with other FireFly metric names. The ff_apiserver_* metrics all have labels method, route, and code, again in keeping with other FireFly components.

The ff_event_batch_size metric is useful for tracking how efficiently the blockchain connector is batching up events for token connector to parse. It seems like gauge is the right type of metric for this, but has the downside that if a batch isn't received for some time due to inactivity its value remains at the last recorded value. I've used a timeout to reset it to 0 after 60 seconds, which is cancelled/reset whenever a new batch size is recorded.

The ff_event_batch_ack_interval metric is useful for tracking how long it takes FireFly core to ACK an event batch. Ultimately this ACK rate will determine how frequently the blockchain connector will flow new batches up to TC/FF core.

The ff_blockchain_calls_total metric has a single label, type, which denotes the type of blockchain call being made. These are currently query, sendTransaction and getReceipt.