Expose metrics

[carlpett]

Following discussions on Slack:
We'd like to be able to monitor our Weave Net, so that we know that all is well. Preferrably, we'd like this exposed as a prometheus metrics endpoint.

Some suggested metrics:

• Number of connections, active and total, per type (sleeve/fastdp)
• Number of locally active ips (possibly split between container allocations and expose allocations?)
• IPAM allocation "fullness"
• Active flows (Would be interesting to know how this could be broken down?)
• Bytes in/out

It would be interesting to maintain the source/dest information in the last two, but from a prometheus perspective it might lead to generating too many timeseries?

[awh]

'Flow misses' is also interesting if it is increasing rapidly.

[carlpett]

Good point!
I've implemented prometheus instrumentation in a few applications, and I'd be willing to work a bit on this, but I'd appreciate some initial structural discussions before jumping in (unless you are already working on this?).

[awh]

@carlpett we're going to be working on this as a matter of priority for kubecon (~4 weeks from now) 😄

[carlpett]

Excellent! 😆

[bboreham]

I have done a bit of this, in #2547. Some points to note:

• Separating fastdp from sleeve is nontrivial
• Weave IPAM doesn't actually know weave:expose is special; it's just another ID

[awh]

I have done a bit of this, in #2547. Some points to note:

These are the specific metrics implemented in #2547:

"weave_connections", "Number of peer-to-peer connections."
"weave_connection_termination_count", "Number of peer-to-peer connections terminated."
"weave_ips", "Number of IP addresses.", "state"
"weave_dns_entries", "Number of DNS entries.", "state"
"weave_flows", "Number of FastDP flows."
"weave_packets_total", "Number of packets transferred."
"weave_bytes_total", "Number of bytes transferred."

The last two are struck through because they were subsequently removed from master to a branch in preparation for #2557, where they will undergo further refinement before release. The residual suggestions made by the OP will be given further consideration as part of #2557.