Statement Distribution Per Peer Rate Limit

[Overkillus]

• Drop requests from a PeerID that is already being served by us.
• Don't sent requests to a PeerID if we already are requesting something from them at that moment (prioritise other requests or wait).
• Tests
• Add a small rep update for unsolicited requests (same peer request) not included in original PR due to potential issues with nodes slowly updating
• Add a metric to track the amount of dropped requests due to peer rate limiting
• Add a metric to track how many time a node reaches the max parallel requests limit in v2+

Helps with but does not close yet: https://github.com/paritytech-secops/srlabs_findings/issues/303

[sandreim]

Shouldn't this be handled in the context of a candidate/relay parent ? Otherwise we would drop legit requests.

[Overkillus]

This is definitely a fair point. Rob mentioned it https://github.com/paritytech-secops/srlabs_findings/issues/303#issuecomment-1588185410 as well.

It's a double-edged sword. If we make it 1 per context then the request/response protocol is more efficient, but with async backing enabled (which to my understanding broadens the scope of possible contexts) the mpsc channel (size 25-ish) can still be easily overwhelmed. The problem of dropped requests gets somewhat a bit better after the recent commit [Rate limit sending from the requester side](https://github.com/paritytech/polkadot-sdk/pull/3444/commits/58d828db45b7c023e8a8266d3180be4361723519) which also introduces the rate limit on the requester side so effort is not wasted.

If we are already requesting from a peer and want to request something else as well we simply wait for the first one to finish.

What could be argued is the limit as a small constant 1-3 instead of 1 per peerID. I would like to do some testing and more reviews and changing it to a configurable constant instead of a hard limit to 1 is actually a trivial change. (I don't think it's necessary. Better have a single completely sent candidate than two half sent candidates.)

Truth be told this whole solution is far from ideal. It's a fix but the final solution is to manage DoS at a significantly lower level than in parachain consensus. It will do for now but DoS needs to be looked on in more detail in near future.

[sandreim]

This is definitely a fair point. Rob mentioned it paritytech-secops/srlabs_findings#303 (comment) as well.

It's a double-edged sword. If we make it 1 per context then the request/response protocol is more efficient, but with async backing enabled (which to my understanding broadens the scope of possible contexts) the mpsc channel (size 25-ish) can still be easily overwhelmed. The problem of dropped requests gets somewhat a bit better after the recent commit [Rate limit sending from the requester side](https://github.com/paritytech/polkadot-sdk/pull/3444/commits/58d828db45b7c023e8a8266d3180be4361723519) which also introduces the rate limit on the requester side so effort is not wasted.

If we are already requesting from a peer and want to request something else as well we simply wait for the first one to finish.

I would rather put the constraint on the responder side rather than the requester. The reasoning is to avoid any bugs in other clients implementation due to the subtle requirement to wait for first request to finish.

What could be argued is the limit as a small constant 1-3 instead of 1 per peerID. I would like to do some testing and more reviews and changing it to a configurable constant instead of a hard limit to 1 is actually a trivial change. (I don't think it's necessary. Better have a single completely sent candidate than two half sent candidates.)

I totally agree with DoS protection as low as possible in the stack, but it seems to be sensible. Maybe instead of relying on a hardcoded constant we can derive the value based on the async backing parameters.

[Overkillus]

I would rather put the constraint on the responder side rather than the requester. The reasoning is to avoid any bugs in other clients implementation due to the subtle requirement to wait for first request to finish.

I agree that constraint on the responder side is more important, but having it in both places seems even better.

The reasoning is to avoid any bugs in other clients implementation due to the subtle requirement to wait for first request to finish.

Even without this PR the risk seems exactly the same. We already were checking for the number of parallel requests being handled in the requester so even if there would be bugs in other clients they could surface there with or without this PR.

The only really bad scenario would be if for some reason we keep a PeerID marked as still being active when it really isn't effectively blacklisting that peer. If that would happen it means that the future connected to that PeerID is still in the pending_responses which would eventually brick the requester anyway since we cannotgo over the max limit of MAX_PARALLEL_ATTESTED_CANDIDATE_REQUESTS. Both situations are just as fatal, but the new system at least has extra protections against DoS attempts so it's a security gain.

By adding the constraint on the requester side we limit the wasted effort and can potentially safely add reputation updates if we get those unsolicited requests which protects us from DoS even further. (Some honest requests might slip through so the rep update needs to be small.)

Maybe instead of relying on a hardcoded constant we can derive the value based on the async backing parameters.

We potentially could but also I don't why this value would need to change a lot. Even if we change some async backing params we should be fine. It seems to be more sensitive to channel size as we want to make it hard to dominate that queue as a malicious node and the MAX_PARALLEL_ATTESTED_CANDIDATE_REQUESTS.

[burdges]

You need not choose a uniform distribution over recent relay parents either. Instead the full value on the first couple, then 1/2 and 1/4 for a few, and then 1 one for a while. You need to figure out if the selected distribution breaks collators, but maybe works.

[alexggh]

Added some comments, what is not clear to me is why can_request from answer_request is not considered fast enough, that already has the benefit that we punish peers that are misbehaving.

Just my2cents, if what we want is to prevent peers from exhausting the buget for the number of connections, I would think the proper fix needs to go deeper into the layer network, rather than here, since there is a lot time spent until the message reaches here.

[Overkillus]

re: @alexggh

Added some comments, what is not clear to me is why can_request from answer_request is not considered fast enough, that already has the benefit that we punish peers that are misbehaving.

Technically fair point but I think there's a reason. It's not about speed but technically yeah respond_task is a bit quicker than answer_request so it seems like a minimal gain here. It also would logically make sense for this logic to be in something called can_request.

Even before this change we tracked the parallel requests up to the limit (MAX_PARALLEL_ATTESTED_CANDIDATE_REQUESTS) in respond_task and my guess why is simply that respond_task encapsulates the async component of the request filtering procedure while can_request focuses on the simpler sync filtering. To properly filter out the most current PeerID this logic here needed to be async as well hence it's also in the async respond_task.

The can_request is primarily doing the filtering based on the knowledge we have about what was sent/received previously while the async respond_task is very time sensitive as we are operating on stuff that is currently being sent over.

Just my2cents, if what we want is to prevent peers from exhausting the buget for the number of connections, I would think the proper fix needs to go deeper into the layer network, rather than here, since there is a lot time spent until the message reaches here.

I 100% agree with you here. I said this myself and me and Robert have already spoken about it and this approach will be pursued. That's why this solution is not ideal but it at least makes the attack vector much harder to pull off with minimal changes to the current code-base. Not ideal but worth the tiny effort before we tackle the wider more general issue at the lower level.

[sandreim]

This might be true for now, but can be easily broken in the future. Why not loop through all incoming requests ?

[Overkillus]

Handled multiple requests 👍

[sandreim]

It should work without needing peer_to_duplicate and payload_to_duplicate clones:

Suggested change

	for _ in 0..self.spam_factor - 1 {
	let (new_outgoing_request, _) = OutgoingRequest::new(
	peer_to_duplicate.clone(),
	payload_to_duplicate.clone(),
	);
	let new_request = Requests::AttestedCandidateV2(new_outgoing_request);
	requests.push(new_request);
	}
	requests.extend(0..self.spam_factor - 1).into_iter().map(|_|
	Requests::AttestedCandidateV2(OutgoingRequest::new(
	peer_to_duplicate.clone(),
	payload_to_duplicate.clone(),
	)))

[Overkillus]

Borrowed the extend syntax but still need the clones to satisfy the borrow checker.

[sandreim]

Nice work!

[sandreim]

We can dedup this line by moving it above the match.

[Overkillus]

This one is somewhat of a conscious trade-off but if you see a graceful way of doing it feel free to suggest it.

Problem is Request does not implement copy/clone. I could possibly track the index where I push it but this will make it even less readable imo.

[alexggh]

Good job!