Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

balance data failed - Transfer leader failed, status Unknown code -3503 #3672

Closed
HarrisChu opened this issue Jan 10, 2022 · 1 comment · Fixed by #3668
Closed

balance data failed - Transfer leader failed, status Unknown code -3503 #3672

HarrisChu opened this issue Jan 10, 2022 · 1 comment · Fixed by #3668
Assignees
@liwenhui-soul
Labels
type/bug Type: something is unexpected
Milestone
v3.0.0

Comments

@HarrisChu
Copy link
Contributor

HarrisChu commented Jan 10, 2022 •
edited
Loading

steps:

  1. CREATE SPACE s1(vid_type=int, replica_factor=4, partition_num=100) on "z1","z2","z3","z4","z5";
  2. MERGE ZONE "z1","add_z6" INTO "z1"
  3. USE s1;SUBMIT JOB BALANCE IN ZONE;
  4. MERGE ZONE "z1","add_z7","add_z8" INTO "z1"
  5. USE s1;SUBMIT JOB BALANCE IN ZONE;

expect result:

  1. balance successfully in second job.

actual result:

  1. job failed.
    -3503 means E_RAFT_TERM_OUT_OF_DATE
I20220110 15:04:51.799562 77140 Host.cpp:174] [Port: 14708, Space: 0, Part: 0] [Host: 127.0.0.1:15839] AppendLog request sent successfully
I20220110 15:04:51.799585 77140 Host.cpp:174] [Port: 14708, Space: 0, Part: 0] [Host: 127.0.0.1:19073] AppendLog request sent successfully
E20220110 15:04:51.799587 79808 BalanceTask.cpp:79] 4, 1:71,127.0.0.1:18692->127.0.0.1:11739 Transfer leader failed, status Unknown code -3503

(root@nebula) [s1]> show job 4
+------------------------+------------------------------------+-------------+---------------------------------+---------------------------------+
| Job Id(spaceId:partId) | Command(src->dst)                  | Status      | Start Time                      | Stop Time                       |
+------------------------+------------------------------------+-------------+---------------------------------+---------------------------------+
| 4                      | "DATA_BALANCE"                     | "FAILED"    | "2022-01-10T07:04:51.000000000" | "2022-01-10T07:05:03.000000000" |
| "4, 1:1"               | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:3"               | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:4"               | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:5"               | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:54.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:6"               | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:8"               | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:9"               | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:10"              | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:11"              | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:54.000000      |
| "4, 1:13"              | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:14"              | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:15"              | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:16"              | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:18"              | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:19"              | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:20"              | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:21"              | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:23"              | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:24"              | "127.0.0.1:13072->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:25"              | "127.0.0.1:13072->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:51"              | "127.0.0.1:18692->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:55.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:53"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:58.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:54"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:52.000000      | 2022-01-10T07:04:55.000000      |
| "4, 1:55"              | "127.0.0.1:18692->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:56"              | "127.0.0.1:18692->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:58.000000      | 2022-01-10T07:05:03.000000      |
| "4, 1:58"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:58.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:59"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:58.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:60"              | "127.0.0.1:18692->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:61"              | "127.0.0.1:18692->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:63"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:64"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:65"              | "127.0.0.1:18692->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:66"              | "127.0.0.1:18692->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:68"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:52.000000      |
| "4, 1:69"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:70"              | "127.0.0.1:18692->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:71"              | "127.0.0.1:18692->127.0.0.1:11739" | "FAILED"    | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "4, 1:73"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:58.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:74"              | "127.0.0.1:18692->127.0.0.1:17958" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:58.000000      |
| "4, 1:75"              | "127.0.0.1:18692->127.0.0.1:11739" | "SUCCEEDED" | 2022-01-10T07:04:51.000000      | 2022-01-10T07:04:51.000000      |
| "Total:40"             | "Succeeded:39"                     | "Failed:1"  | "In Progress:0"                 | "Invalid:0"                     |
+------------------------+------------------------------------+-------------+---------------------------------+---------------------------------+
Got 42 rows (time spent 3507/14999 us)
@HarrisChu HarrisChu added the type/bug Type: something is unexpected label Jan 10, 2022
@Sophie-Xie Sophie-Xie added this to the v3.0.0 milestone Jan 10, 2022
@Sophie-Xie Sophie-Xie added the duplicate Solution: this issue or pull request already exists label Jan 13, 2022
@liwenhui-soul
Copy link
Contributor

fix in #3668

@Sophie-Xie Sophie-Xie linked a pull request Jan 13, 2022 that will close this issue
Balance fix #3668
Merged
11 tasks
@Sophie-Xie Sophie-Xie removed the duplicate Solution: this issue or pull request already exists label Jan 13, 2022
@jamieliu1023 jamieliu1023 mentioned this issue Jan 15, 2022
Closed
@critical27 critical27 mentioned this issue Jan 20, 2022
Merged
11 tasks
@jamieliu1023 jamieliu1023 mentioned this issue Jan 22, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liwenhui-soul liwenhui-soul

Labels
type/bug Type: something is unexpected
Projects
None yet
Milestone
v3.0.0
Development

Successfully merging a pull request may close this issue.

Balance fix liwenhui-soul/nebula
3 participants
@HarrisChu @liwenhui-soul @Sophie-Xie