Introduce Follower Replication

[Fullstop000]

Relate to tikv/raft-rs#249
Signed-off-by: Fullstop000 [email protected]

[siddontang]

I think this can also reduce the pressure of the leader when it has many followers or learners.

[siddontang]

can you paste your previous benchmark results here, so others can see the benefit intuitively.

[Fullstop000]

@siddontang I'll make a more concrete benchmark for this. Some previous results are hard to explain why. But I'm ok to post them here :).

[siddontang]

how do we do by the upper layer for the group which contains the leader?

[Fullstop000]

It's unnecessary for the upper layer to acknowledge which group contains the leader because only the leader can choose delegates. And the leader itself must know which group it belongs to.

The description here might be somewhat confused. which is able to be controlled by upper layer means that the upper layer can decide whether the leader can choose a delegate in the group itself belongs to or not. I'll make it more clear.

[abbccdda]

Thanks for the detailed design. Could we have a compatibility section, to discuss things such as some nodes in a group are still in old version, and could not recognize a delegate's request for entry replication?

[Fullstop000]

@abbccdda A node in a group will send its group_id in msg and the receiver will update the sender's group info based on it. The leader only picks a delegate when the group info is enough. In a rolling-upgrade/downgrade situation, This can introduce several cases:

Upgrade

• If only the leader uses follower replication, it can only know the group info until followers finish upgrading and send their group_id so that the leader uses origin log replication at this point
• If only a follower or part of them use follower replication, the leader will just ignore the group_id in the msg so no delegate will be picked and origin log replication keeps processing

Downgrade

• If only the leader use origin log replication, the case is just like common raft cluster and nothing special happens (pick a delegate, broadcast appends)
• If a follower is downgraded and stop informing the leader its group_id, the leader will remove it from the group system and send entries directly to it

By such a design, the compatibility can be guaranteed when nodes in the cluster use either origin log replication or follower replication

It seems this feature description is missing in the RFC. I'll add it soon.

[abbccdda]

@Fullstop000 Thanks for the reply. It would be good to add such details to RFC for sure :)

[abbccdda]

I was actually thinking whether we could make the delegation group support native in the first version. Suppose we want to form another delegation group in runtime, we need to change the static configs and do the cluster rolling bounce.

While it makes sense for the first version to focus on static configs, we may also propose runtime formulation of delegation group through third-party control such as PD. When the leader load goes up, it makes sense to distribute its load to other up-to-date followers. We could briefly talk about how we could keep the door open for such a future improvement, with the first version only supporting static group_id.

[Fullstop000]

Good idea! Actually it's the first design of how the leader manages all the group info.