Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PD may repeatedly add learner to a region #5786

Closed
HunDunDM opened this issue Dec 15, 2022 · 3 comments
Closed

PD may repeatedly add learner to a region #5786

HunDunDM opened this issue Dec 15, 2022 · 3 comments
Labels
affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 affects-6.1 affects-6.5 severity/minor type/bug The issue is confirmed as a bug.

Comments

@HunDunDM
Copy link
Member

Bug Report

What did you do?

  • Start a cluster with 5 tikv(mark as tikv-1, tikv-2, tikv-3, tikv-4, tikv-5)
  • Pause rule-checker curl -X POST -d '{"delay":3000}' "http://{PD_ADDRESS}/pd/api/v1/checker/rule"
  • Find a Region whose Peer is on tikv-1, tikv-2, tikv-3
  • Add Learner to this Region on tikv-4, tikv-5
  • Kill tikv-2, and wait for it to become Down and the Region recognizes pending-peer and down-peer
  • Resume rule-checker curl -X POST -d '{"delay":0}' "http://{PD_ADDRESS}/pd/api/v1/checker/rule"

What did you expect to see?

The other Regions of tikv-2 were migrated away, and the Region also returned to normal.

What did you see instead?

Other Regions return to normal, but this Region cannot return to normal.

What version of PD are you using (pd-server -V)?

v6.4.0

Note

It can be restored manually.

Solving it requires two steps:

  • If the redundant Learner is also Pending or Down, delete it first.
  • If a redundant Learner is available, promote it directly.
@HunDunDM HunDunDM added the type/bug The issue is confirmed as a bug. label Dec 15, 2022
@HunDunDM HunDunDM mentioned this issue Dec 15, 2022
Merged
@HunDunDM
Copy link
Member Author

Because of #4045 , we need more discussion.

ti-chi-bot added a commit that referenced this issue Dec 30, 2022
@HunDunDM @ti-chi-bot
checker: when there are too many orphan peers, try to delete them (#5787
531d9f3 
)

ref #5786

Signed-off-by: HunDunDM <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Dec 30, 2022
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5787
c656c9a 
ref tikv#5786

Signed-off-by: ti-chi-bot <[email protected]>
This was referenced Dec 30, 2022
Closed
Closed
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Dec 30, 2022
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5787
f5d36a1 
ref tikv#5786

Signed-off-by: ti-chi-bot <[email protected]>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Dec 30, 2022
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5787
bd67293 
ref tikv#5786

Signed-off-by: ti-chi-bot <[email protected]>
This was referenced Dec 30, 2022
Closed
Closed
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Dec 30, 2022
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5787
226f360 
ref tikv#5786

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Dec 30, 2022
Closed
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Dec 30, 2022
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5787
6a542a4 
ref tikv#5786

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Dec 30, 2022
Merged
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Dec 30, 2022
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5787
bc1602a 
ref tikv#5786

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Dec 30, 2022
Merged
nolouch pushed a commit to nolouch/pd that referenced this issue Dec 30, 2022
@HunDunDM @ti-chi-bot @nolouch
checker: when there are too many orphan peers, try to delete them (ti…
7da03eb 
…kv#5787)

ref tikv#5786

Signed-off-by: HunDunDM <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
ti-chi-bot added a commit that referenced this issue Jan 6, 2023
@ti-chi-bot @HunDunDM
checker: when there are too many orphan peers, try to delete them (#5787
979c460 
) (#5818)

ref #5786, ref #5787

Signed-off-by: HunDunDM <[email protected]>

Co-authored-by: HunDunDM <[email protected]>
ti-chi-bot added a commit that referenced this issue Jan 11, 2023
@ti-chi-bot @HunDunDM @nolouch
checker: when there are too many orphan peers, try to delete them (#5787
93f49ea 
) (#5817)

* This is an automated cherry-pick of #5787

ref #5786

Signed-off-by: ti-chi-bot <[email protected]>

* fix conflicts

Signed-off-by: nolouch <[email protected]>

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: nolouch <[email protected]>
Co-authored-by: 混沌DM <[email protected]>
Co-authored-by: nolouch <[email protected]>
@nolouch nolouch mentioned this issue Mar 7, 2023
Merged
16 tasks
@rleungx
Copy link
Member

rleungx commented Aug 10, 2023

Can we close it now? @nolouch

@nolouch
Copy link
Contributor

nolouch commented Aug 28, 2023

yes. I think this fixed.

@nolouch nolouch closed this as completed Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 affects-6.1 affects-6.5 severity/minor type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports
Status: Closed
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nolouch @HunDunDM @rleungx