Add config option tenure_last_block_proposal_timeout_secs and do not allow reorgs at tenure boundary before it is exceeded

[jferrant]

Closes #5423

[obycode]

LGTM!

[jcnelson]

Approving for now, but I'm not sure this is guaranteed to prevent reorgs at tenure boundaries? It could be that some singers' slow networks cause them to:

• Accept and broadcast a signature on the last block of tenure N B[N][-1]
• Wait for the allotted timeout to pass, but never see B[N][-1] get broadcast and processed
• Accept and broadcast signatures for the first block of tenure N+1, B[N+1][0], which conflicts with B[N][-1] (i.e. has the same height).

Am I understanding this right? Is this okay?

[jferrant]

Approving for now, but I'm not sure this is guaranteed to prevent reorgs at tenure boundaries? It could be that some singers' slow networks cause them to:

* Accept and broadcast a signature on the last block of tenure N `B[N][-1]`

* Wait for the allotted timeout to pass, but never see `B[N][-1]` get broadcast and processed

* Accept and broadcast signatures for the first block of tenure N+1, `B[N+1][0]`, which conflicts with `B[N][-1]` (i.e. has the same height).

Am I understanding this right? Is this okay?

Yep, your understanding is correct! it is not meant to prevent reorgs at the boundary. just to minimize them. Unfortunately, I do not think there is a way to prevent it without causing a chainstall :( We previously agreed that this was okay (see this comment)

This chain stall case happens in the following scenario:

Tenure A starts

• ...some blocks happen...
• block N proposed
• x signers receive this proposal and approve it (70% > x > 30%)

Tenure B starts

• y signers receive bock N and reject it (but due to non participating signers, y < 30%)
• block N' proposed
• x (> 30%) signers reject the proposal because it is a sibling of N which they have already approved

Chain Halts

• No valid block can be built because majority signers (x) will always reject it as a sibling

(@kantai @obycode am I remembering the chainstall case correctly?)

Our response to this chainstall due to poorly timed block commits case was to enable signers to reorg the last block of the prior tenure. However, this caused legit globally accepted blocks due to network latency issues to be reorged when they really shouldn't have.

Maybe this default should be much higher? However, in the actual chainstall case, the chain will halt for up to that config amount so prob shouldn't set it too much higher.

[obycode]

Almost got it 😆