Re-process early sampling requests

[dapplion]

Issue Addressed

Allow data availability samplers to send requests early as a network-wide optimization to complete sampling faster. A custodial node will queue the requests and resolve them once the block has been imported.

Next steps

This is a basic (non-tested) implementation, next steps:

• Resolve requests when column arrives, not the whole block
• Separate requests for known block and unknown block, the former can be DOS protected better
• Only allow retry mechanism for recent slots
• Add more metrics

Design

two phases
a) requests at the start of the start of the slot have a long timeout — e.g. 3s (expected time for most of the network to get samples on gossip)
b) then shift into fast timeout mode

At start of the slot, send your sample requests to peers you expect to get your needed samples in (a) mode (long timeout). If they don't respond by timeout, send further queries in (b) mode to various peers, trying to quickly get the final samples

Forwarded from codex (credit to them) 👇

The usage of (a) allows for samples queries to be responded to immediately when a peer gets it from gossip, instead of the node waiting until some arbitrary time at the slot that they think people will likely have gotten the sample to start the requests

The messaging primitive I proposed is a request with an explicit timeout. This simple primitive allows us to build a nice dynamic behavior, as you see in the example on the slide. It allows us to have the first period, beneficial for two reasons:

• reduce traffic explosion on the happy path: if things go well, every sample query will only consume two messages. A request and a response. This is beneficial both in the peerDAS case, and in the later stages, where we would go for individual segments on the 2D encoding grid.
• reduce sampling delay close to the minimum: sending the sampling query at the start of the slot allows us to minimize the sampling delay to a single 1-hop latency on top of to diffusion delay (the delay with which the node we queried received the segment or column).

The minimum sampling latency would be what you could achive if you ask, right when you know what samples you need, all your peers that could have the sample. That would however mean that you receive lots of responses, adding considerable extra load. Here we only ask one of them as a compromise between bandwidth efficiency and latency, and our simulations currently show the latency compromise is very limited on the happy path. Obviusly, if we are not on the happy path, we should start doing something. On the slide you find a few proposals, considering that we should still avoid traffic explosion.

There a few other points there, improving dynamic behaviour:
1, anticipate the hunt for peers that could provide samples. We have a few options here:

• a node could try to keep a few peers (but at least one) for each column. This can be done independent of slots, continously
• a node can notice at the beginning of the slot, that it misses peers for a column selected for sampling. The node has almost 4 seconds to find a peer and send a query
• a node can also act if a request was sent out, but a response is not coming close to the 4s mark. If there are other peers for the same sample, send another request. If not, start searching for a peer to request from.

2, to avoid traffic explosion, we also recommend controlling the number of outstanding queries, together with timeouts. This practically means that we can avoid traffic explosion.
3, we've also proposed the extension of the sample. This is something I'm writing up in an ethresear.ch post 🙂

Regarding the number of queries is stage (a): I think 1 per sample is fine, but of course 2 or 3 can also be used. These are comprimises that can even be made locally, we can't really mandate them anyway. (actually we could, using RLN and similar spam protection, but that would require stake to send DAS queries, which is not what we want now).

Finally, there was another idea on my previous slide on messaging primitives. I would include a have list in the query response message. The reason behind this is that it gives extra information on the state of diffusion that we would not have otherwise. Given the 1D or 2D structure, a have list can be compressed very well to a small bitmap, so it is not much extra bytes, so I think it's worth to add. Eventually, to stay generic on the message primive, we can make it a flag in the request whether you want a have list in the response.

[jimmygchen]

I think this is the same as block_exists

[jimmygchen]

Fixed in 4adbd29

[jimmygchen]

Looks good!

@pawanjay176 mentioned that we would also need to send first byte so that the connection doesn't timeout (TTFB = 5s). I can follow this up in a separate PR.

[jimmygchen]

@dapplion I've made some minor fixes and I think this PR is good to go! I'll address the TTFB issue in a separate PR. Thanks!

[jimmygchen]

We probably don't need to worry too much about TTFB timeout for now - as we send sampling request after we receive the block (instead of start of slot), so in most cases 5s would be good enough. Will leave this for later if we need it.