How to debug the requests sent from raft leader to its peers. I have previous leader step down and doesn't get notified about new term and index.

[sumithreddy]

The one stuck in precandidate state (probe state in leader view) keep sending prevote requests which end up on current leader and thats it. They don't get notified about the new term & leader and thus remain stuck. How to get to the bottom of this issue ?

Leader rejecting peer voting:

2023-01-12T05:57:50.925	INFO	raft	[email protected]/raft.go:905	a597b44beff820f [logterm: 5, index: 227338, vote: a597b44beff820f] rejected MsgPreVote from 17d32fa7d00728cb [logterm: 2, index: 94096] at term 5
2023-01-12T05:57:51.180	INFO	raft	[email protected]/raft.go:969	a597b44beff820f [logterm: 5, index: 227338, vote: a597b44beff820f] rejected MsgPreVote from 3e0c9e98694e57b4 [logterm: 4, index: 98246] at term 5

Raft state:

root@service-pod-objectscale:/# for i in `seq 0 4`; do echo "asuka-cluster-atlas-$i"; curl asuka-cluster-atlas-$i.asuka-cluster-atlas-headless.atlantic.svc.cluster.local:8080/debug/replication/raftstatus | jq; done
asuka-cluster-atlas-0
{
  "id": "3e0c9e98694e57b4",
  "term": 4,
  "vote": "3e0c9e98694e57b4",
  "commit": 98231,
  "lead": "0",
  "raftState": "StatePreCandidate",
  "applied": 98231,
  "progress": {},
  "leadtransferee": "0"
}

asuka-cluster-atlas-1
{
  "id": "f3052edbb6a03283",
  "term": 5,
  "vote": "a597b44beff820f",
  "commit": 204524,
  "lead": "a597b44beff820f",
  "raftState": "StateFollower",
  "applied": 204524,
  "progress": {},
  "leadtransferee": "0"
}

asuka-cluster-atlas-2
{
  "id": "a597b44beff820f",
  "term": 5,
  "vote": "a597b44beff820f",
  "commit": 204524,
  "lead": "a597b44beff820f",
  "raftState": "StateLeader",
  "applied": 204524,
  "progress": {
    "a597b44beff820f": {
      "match": 204524,
      "next": 204525,
      "state": "StateReplicate"
    },
    "17d32fa7d00728cb": {
      "match": 0,
      "next": 98247,
      "state": "StateProbe"
    },
    "3db33c0b5107ef0c": {
      "match": 204524,
      "next": 204525,
      "state": "StateReplicate"
    },
    "3e0c9e98694e57b4": {
      "match": 0,
      "next": 98247,
      "state": "StateProbe"
    },
    "f3052edbb6a03283": {
      "match": 204524,
      "next": 204525,
      "state": "StateReplicate"
    }
  },
  "leadtransferee": "0"