Purge offline validators

[garious]

Avalanche will split segments between all validators. If very few are offline, erasure codes can be used to fill the gaps, but if too many, the validators will need to request the missing segments, which is very slow. Nodes should actively identify offline nodes and purge them.

[garious]

@aeyakovenko, can you recommend an entrypoint for this code?

[aeyakovenko]

we should see a RequestUpdate from nodes in the network.
https://github.com/solana-labs/solana/blob/master/src/crdt.rs#L565
and we can mark them as stale if we don't see them often enough

[carllin]

@aeyakovenko, right now the validators pick random peers to RequestUpdate from, so should we just set the timeout to be sufficiently long so that the probability that two peers will connect is very high? Or we could set a timeout period after which a validator can send an explicit heartbeat request if they suspect a node on the network might be offline.

[aeyakovenko]

@carllin initially we can set the timeout long enough, and eventually we can start gossip availability stats between nodes and aggregate those values. The latter is tricky because we can’t trust rumors from adversarial nodes on the network