Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

back up sf300 failed, rpc error #463

Closed
jinyingsunny opened this issue Mar 1, 2024 · 3 comments
Closed

back up sf300 failed, rpc error #463

jinyingsunny opened this issue Mar 1, 2024 · 3 comments
Assignees
@MegaByte875
Labels
affects/master PR/issue: this bug affects master version. process/done Process of bug severity/major Severity of bug type/bug Type: something is unexpected

Comments

@jinyingsunny
Copy link

sf300 backup failed，if we shall do some retry
image
the failed two jobs have same rpc error, as follows:

$ kubectl -n harris logs -f backup-full-nb20240301f300-3-5gqr9
+ exec /usr/local/bin/br-ent backup full --meta nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559 --concurrency 5 --storage s3://nebula-br-test2 --s3.region us-west-2 --s3.endpoint https://s3.us-west-2.amazonaws.com --s3.access_key AKIAXVNKNXDHKRV47J7K --s3.secret_key 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
{"level":"info","meta address":"nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T17:43:07.126Z"}
{"level":"info","meta address":"nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T17:43:07.129Z"}
{"level":"info","meta address":"nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T17:43:07.130Z"}
{"level":"info","meta address":"nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T17:43:07.133Z"}
{"level":"info","meta address":"nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T17:43:07.134Z"}
{"level":"info","meta address":"nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T17:43:07.137Z"}
{"level":"info","meta address":"nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T17:43:07.138Z"}
{"level":"info","meta address":"nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T17:43:07.140Z"}
{"level":"info","meta address":"nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T17:43:07.140Z"}
{"level":"info","meta address":"nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T17:43:07.143Z"}
{"host info":"map[nebulasf300-graphd-0.nebulasf300-graphd-headless.harris.svc.cluster.local:nebulasf300-graphd-0.nebulasf300-graphd-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-graphd-0.nebulasf300-graphd-headless.harris.svc.cluster.local:9669[GRAPH]: (data: , root: /usr/local/nebula) nebulasf300-graphd-1.nebulasf300-graphd-headless.harris.svc.cluster.local:nebulasf300-graphd-1.nebulasf300-graphd-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-graphd-1.nebulasf300-graphd-headless.harris.svc.cluster.local:9669[GRAPH]: (data: , root: /usr/local/nebula) nebulasf300-graphd-2.nebulasf300-graphd-headless.harris.svc.cluster.local:nebulasf300-graphd-2.nebulasf300-graphd-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-graphd-2.nebulasf300-graphd-headless.harris.svc.cluster.local:9669[GRAPH]: (data: , root: /usr/local/nebula) nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebulasf300-storaged-0.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-0.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-0.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-1.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-1.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-1.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-2.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-2.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-2.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-3.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-3.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-3.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-4.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-4.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-4.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-5.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-5.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-5.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: )]","level":"info","msg":"Get cluster topology from the nebula.","time":"2024-03-01T17:43:07.144Z"}
{"level":"info","msg":"Start to full backup cluster...","time":"2024-03-01T17:43:07.144Z"}
{"level":"info","msg":"Ensure backup root dir.","name":"BACKUP_2024_03_01_17_43_07","root":"s3://nebula-br-test2/BACKUP_2024_03_01_17_43_07","time":"2024-03-01T17:43:07.455Z"}
{"level":"info","meta":"s3://nebula-br-test2/BACKUP_2024_03_01_17_43_07/meta","msg":"Upload meta successfully.","name":"BACKUP_2024_03_01_17_43_07","time":"2024-03-01T17:43:09.029Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 1/24","time":"2024-03-01T17:49:10.540Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 2/24","time":"2024-03-01T17:49:37.087Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 3/24","time":"2024-03-01T17:49:38.749Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 4/24","time":"2024-03-01T17:49:41.003Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 5/24","time":"2024-03-01T17:49:45.252Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 6/24","time":"2024-03-01T17:54:36.717Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 7/24","time":"2024-03-01T17:54:57.642Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 8/24","time":"2024-03-01T17:55:05.413Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 9/24","time":"2024-03-01T17:55:59.461Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 10/24","time":"2024-03-01T17:56:05.058Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 11/24","time":"2024-03-01T18:00:09.726Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 12/24","time":"2024-03-01T18:00:19.219Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 13/24","time":"2024-03-01T18:00:23.194Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 14/24","time":"2024-03-01T18:02:13.731Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 15/24","time":"2024-03-01T18:02:25.292Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 16/24","time":"2024-03-01T18:05:32.881Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 17/24","time":"2024-03-01T18:06:06.661Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 18/24","time":"2024-03-01T18:06:34.348Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 19/24","time":"2024-03-01T18:08:11.786Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 20/24","time":"2024-03-01T18:08:40.186Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 21/24","time":"2024-03-01T18:11:17.007Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 22/24","time":"2024-03-01T18:11:36.142Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 23/24","time":"2024-03-01T18:12:04.674Z"}
{"level":"info","msg":"full upload storaged partition finished, progress: 24/24","time":"2024-03-01T18:13:57.732Z"}
{"level":"error","msg":"Full backup failed: upload storage failed upload /usr/local/nebula/data/storage/nebula/1/11/checkpoints/BACKUP_2024_03_01_17_43_07 to s3://nebula-br-test2/BACKUP_2024_03_01_17_43_07/data/1/11 failed:rpc error: code = Unknown desc = upload from /usr/local/nebula/data/storage/nebula/1/11/checkpoints/BACKUP_2024_03_01_17_43_07/data/002933.sst to BACKUP_2024_03_01_17_43_07/data/1/11/data/002933.sst failed: upload from /usr/local/nebula/data/storage/nebula/1/11/checkpoints/BACKUP_2024_03_01_17_43_07/data/002933.sst to BACKUP_2024_03_01_17_43_07/data/1/11/data/002933.sst failed: MultipartUpload: upload multipart failed\n\tupload id: sxjZw50Vnp9z8o1ROMOILd56Iop.goGwXMEgHOY2aWdKCmEFDDn31ImtkNq9Zw1V2hQuYmoLp7eA3.Xd4ROXOwux7jXES_yptfAcLb4EIQLpG0DTwFr1k_7tJrgxqQnl\ncaused by: RequestError: send request failed\ncaused by: Put \"https://nebula-br-test2.s3.us-west-2.amazonaws.com/BACKUP_2024_03_01_17_43_07/data/1/11/data/002933.sst?partNumber=3\u0026uploadId=sxjZw50Vnp9z8o1ROMOILd56Iop.goGwXMEgHOY2aWdKCmEFDDn31ImtkNq9Zw1V2hQuYmoLp7eA3.Xd4ROXOwux7jXES_yptfAcLb4EIQLpG0DTwFr1k_7tJrgxqQnl\": read tcp 192.168.51.246:37466-\u003e3.5.80.186:443: read: connection reset by peer, will try to clean the remaining garbage...\n","time":"2024-03-01T18:13:57.732Z"}
{"level":"info","meta address":"nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T18:13:57.733Z"}
{"level":"info","meta address":"nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T18:13:57.736Z"}
{"level":"info","meta address":"nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T18:13:57.737Z"}
{"level":"info","meta address":"nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T18:13:57.740Z"}
{"level":"info","meta address":"nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T18:13:57.741Z"}
{"level":"info","meta address":"nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T18:13:57.744Z"}
{"level":"info","meta address":"nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T18:13:57.745Z"}
{"level":"info","meta address":"nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T18:13:57.748Z"}
{"level":"info","meta address":"nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2024-03-01T18:13:57.749Z"}
{"level":"info","meta address":"nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2024-03-01T18:13:57.751Z"}
{"host info":"map[nebulasf300-graphd-0.nebulasf300-graphd-headless.harris.svc.cluster.local:nebulasf300-graphd-0.nebulasf300-graphd-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-graphd-0.nebulasf300-graphd-headless.harris.svc.cluster.local:9669[GRAPH]: (data: , root: /usr/local/nebula) nebulasf300-graphd-1.nebulasf300-graphd-headless.harris.svc.cluster.local:nebulasf300-graphd-1.nebulasf300-graphd-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-graphd-1.nebulasf300-graphd-headless.harris.svc.cluster.local:9669[GRAPH]: (data: , root: /usr/local/nebula) nebulasf300-graphd-2.nebulasf300-graphd-headless.harris.svc.cluster.local:nebulasf300-graphd-2.nebulasf300-graphd-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-graphd-2.nebulasf300-graphd-headless.harris.svc.cluster.local:9669[GRAPH]: (data: , root: /usr/local/nebula) nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-metad-0.nebulasf300-metad-headless.harris.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-metad-1.nebulasf300-metad-headless.harris.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebulasf300-metad-2.nebulasf300-metad-headless.harris.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebulasf300-storaged-0.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-0.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-0.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-1.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-1.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-1.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-2.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-2.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-2.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-3.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-3.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-3.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-4.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-4.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-4.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: ) nebulasf300-storaged-5.nebulasf300-storaged-headless.harris.svc.cluster.local:nebulasf300-storaged-5.nebulasf300-storaged-headless.harris.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebulasf300-storaged-5.nebulasf300-storaged-headless.harris.svc.cluster.local:8888[AGENT]: (data: , root: )]","level":"info","msg":"Get cluster topology from the nebula.","time":"2024-03-01T18:13:57.751Z"}
{"backup name":"BACKUP_2024_03_01_17_43_07","level":"info","msg":"Start to cleanup data in cluster self.","time":"2024-03-01T18:13:57.751Z"}
{"backup name":"BACKUP_2024_03_01_17_43_07","level":"info","msg":"Start cleanup data in external storage.","time":"2024-03-01T18:13:57.872Z"}
{"backup name":"BACKUP_2024_03_01_17_43_07","level":"info","msg":"Clean up backup data successfully.","time":"2024-03-01T18:22:46.279Z"}
{"level":"error","msg":"Cleanup full backup BACKUP_2024_03_01_17_43_07 successfully after backup failed.","time":"2024-03-01T18:22:46.279Z"}
Error: upload storage failed upload /usr/local/nebula/data/storage/nebula/1/11/checkpoints/BACKUP_2024_03_01_17_43_07 to s3://nebula-br-test2/BACKUP_2024_03_01_17_43_07/data/1/11 failed:rpc error: code = Unknown desc = upload from /usr/local/nebula/data/storage/nebula/1/11/checkpoints/BACKUP_2024_03_01_17_43_07/data/002933.sst to BACKUP_2024_03_01_17_43_07/data/1/11/data/002933.sst failed: upload from /usr/local/nebula/data/storage/nebula/1/11/checkpoints/BACKUP_2024_03_01_17_43_07/data/002933.sst to BACKUP_2024_03_01_17_43_07/data/1/11/data/002933.sst failed: MultipartUpload: upload multipart failed
	upload id: sxjZw50Vnp9z8o1ROMOILd56Iop.goGwXMEgHOY2aWdKCmEFDDn31ImtkNq9Zw1V2hQuYmoLp7eA3.Xd4ROXOwux7jXES_yptfAcLb4EIQLpG0DTwFr1k_7tJrgxqQnl
caused by: RequestError: send request failed
caused by: Put "https://nebula-br-test2.s3.us-west-2.amazonaws.com/BACKUP_2024_03_01_17_43_07/data/1/11/data/002933.sst?partNumber=3&uploadId=sxjZw50Vnp9z8o1ROMOILd56Iop.goGwXMEgHOY2aWdKCmEFDDn31ImtkNq9Zw1V2hQuYmoLp7eA3.Xd4ROXOwux7jXES_yptfAcLb4EIQLpG0DTwFr1k_7tJrgxqQnl": read tcp 192.168.51.246:37466->3.5.80.186:443: read: connection reset by peer
Usage:
  br-ent backup full [flags]

Flags:
  -h, --help   help for full

Global Flags:
      --ca-path string          Specify CA file path (default "/usr/local/certs/ca.crt")
      --cert-path string        Specify cert file path (default "/usr/local/certs/client.crt")
      --concurrency int         Max concurrency for download data and data playback (default 5)
      --debug                   Output log in debug level or not
      --enable-ssl              Enable SSL connection
      --gs.credentials string   GS Option: set google credentials with the given service account or refresh token JSON, the input json is a single long string.
      --insecure-skip-verify    Verify the server's certificate chain and host name
      --key-path string         Specify key file path (default "/usr/local/certs/client.key")
      --log string              Specify br detail log path (default "br.log")
      --meta string             Specify meta server
      --s3.access_key string    S3 Option: set access key id
      --s3.endpoint string      S3 Option: set the S3 endpoint URL, please specify the http or https scheme explicitly
      --s3.region string        S3 Option: set region or location to upload or download backup
      --s3.secret_key string    S3 Option: set secret key for access id
      --server-name string      The subject alternative name (SAN) of the peer server to verify
      --spaces stringArray      (EXPERIMENTAL)space names.
                                    By this option, user can specify which spaces to backup. Now this feature is still experimental.
                                    If not specified, will backup all spaces.

      --storage string          backup target url, format: <SCHEME>://<PATH>.
                                    <SCHEME>: a string indicating which backend type. optional: local, s3.
                                    now only s3-compatible is supported.
                                    example:
                                    for local - "local:///the/local/path/to/backup"
                                    for s3  - "s3://example/url/to/the/backup"
                                    fort gs - "gs://example/url/to/the/backup"

{"level":"fatal","msg":"upload storage failed upload /usr/local/nebula/data/storage/nebula/1/11/checkpoints/BACKUP_2024_03_01_17_43_07 to s3://nebula-br-test2/BACKUP_2024_03_01_17_43_07/data/1/11 failed:rpc error: code = Unknown desc = upload from /usr/local/nebula/data/storage/nebula/1/11/checkpoints/BACKUP_2024_03_01_17_43_07/data/002933.sst to BACKUP_2024_03_01_17_43_07/data/1/11/data/002933.sst failed: upload from /usr/local/nebula/data/storage/nebula/1/11/checkpoints/BACKUP_2024_03_01_17_43_07/data/002933.sst to BACKUP_2024_03_01_17_43_07/data/1/11/data/002933.sst failed: MultipartUpload: upload multipart failed\n\tupload id: sxjZw50Vnp9z8o1ROMOILd56Iop.goGwXMEgHOY2aWdKCmEFDDn31ImtkNq9Zw1V2hQuYmoLp7eA3.Xd4ROXOwux7jXES_yptfAcLb4EIQLpG0DTwFr1k_7tJrgxqQnl\ncaused by: RequestError: send request failed\ncaused by: Put \"https://nebula-br-test2.s3.us-west-2.amazonaws.com/BACKUP_2024_03_01_17_43_07/data/1/11/data/002933.sst?partNumber=3\u0026uploadId=sxjZw50Vnp9z8o1ROMOILd56Iop.goGwXMEgHOY2aWdKCmEFDDn31ImtkNq9Zw1V2hQuYmoLp7eA3.Xd4ROXOwux7jXES_yptfAcLb4EIQLpG0DTwFr1k_7tJrgxqQnl\": read tcp 192.168.51.246:37466-\u003e3.5.80.186:443: read: connection reset by peer","time":"2024-03-01T18:22:46.280Z"}

Your Environments (required)

snap-1.42

Expected behavior

perhaps do some retry. since when upload huge data, rpc error may always happen.
@jinyingsunny jinyingsunny added the type/bug Type: something is unexpected label Mar 1, 2024
@github-actions github-actions bot added affects/none PR/issue: this bug affects none version. severity/none Severity of bug labels Mar 1, 2024
@MegaByte875 MegaByte875 self-assigned this Mar 1, 2024
@MegaByte875
Copy link
Contributor

MegaByte875 commented Mar 2, 2024 •
edited
Loading

I will fix the issue in vesoft-inc/nebula-agent#58

@jinyingsunny jinyingsunny added affects/master PR/issue: this bug affects master version. and removed severity/none Severity of bug affects/none PR/issue: this bug affects none version. labels Mar 3, 2024
@github-actions github-actions bot added the severity/none Severity of bug label Mar 3, 2024
@jinyingsunny jinyingsunny added severity/major Severity of bug and removed severity/none Severity of bug labels Mar 3, 2024
@MegaByte875
Copy link
Contributor

vesoft-inc/nebula-agent#59

@jinyingsunny
Copy link
Author

checked source code

@github-actions github-actions bot added the process/fixed Process of bug label Mar 4, 2024
@jinyingsunny jinyingsunny added process/done Process of bug and removed process/fixed Process of bug labels Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MegaByte875 MegaByte875

Labels
affects/master PR/issue: this bug affects master version. process/done Process of bug severity/major Severity of bug type/bug Type: something is unexpected
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MegaByte875 @jinyingsunny