Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Miner should not try to dial the client to restart a data transfer #4991

Closed
dineshshenoy opened this issue Nov 24, 2020 · 3 comments · Fixed by #5210
Closed

Miner should not try to dial the client to restart a data transfer #4991

dineshshenoy opened this issue Nov 24, 2020 · 3 comments · Fixed by #5210
Assignees
Labels
P2 P2: Should be resolved

Comments

@dineshshenoy
Copy link

From @dirkmc

It appears that on restart the miner is trying to dial the client to restart a data transfer.
The miner shouldn't dial the client - the client should restart when the connection goes down.

error restarting data transfer: unable to send restart request: exhausted %!d(float64=5) attempts but failed to open stream, err: failed to dial 12D3KooWHAN67HgLoqrygQnouFF5uVhwM4kP7zurpR65Powtfgsa: all dials failed

  • [/ip4/127.0.0.1/tcp/33695] dial tcp4 127.0.0.1:33695: connect: connection refused
  • [/ip6/::1/tcp/39135] dial tcp6 [::1]:39135: connect: connection refused
  • [/ip4/10.240.0.46/tcp/33695] dial tcp4 0.0.0.0:9999->10.240.0.46:33695: i/o timeout
  • [/ip4/13.68.239.146/tcp/33695] dial tcp4 0.0.0.0:9999->13.68.239.146:33695: i/o timeout

Recommendation from @magik6k and @hannahhoward

  1. Miner shouldn't try to restart the request.
  2. Client should auto-retry when it sees the network connection drop (every 5 minutes)
@dineshshenoy dineshshenoy added this to the 💹Storage Deal Success milestone Nov 24, 2020
@hannahhoward
Copy link
Contributor

Follow-up for @dirkmc

Specs for this ticket:

  • remove auto restart for data transfers from provider in go-fil-markets - just leave deal as is, in transferring state
  • in go-fil-markets, when a transfer becomes stalled (see event / message) on the client only start a go routine with a timer that attempts to restart the data transfer every 5 minutes -- I would rather NOT put this in go-data-transfer itself - not all users (including, most importantly, the provider) will want to attempt to restart

@dineshshenoy dineshshenoy added the P2 P2: Should be resolved label Nov 25, 2020
@dineshshenoy
Copy link
Author

The miner retry is causing the deal to fail:

From @magik6k

bafyreig53wbhyafwc4nvdnupjadb3r24v6xlvsizpgbnnays7zqaec5kzq 0 StorageDealError f3rab4mhsr7f2gdg6ozgj44mtny6bhcy3rvlftc5xyki6t4hvhdkaji3ahfaid6fh2o3wff4yfrzag4lge2r2q 8GiB 0.004161616 FIL 520202 error restarting data transfer: unable to send restart request: exhausted %!d(float64=5) attempts but failed to open stream, err: routing: not found


@kernelogic
Copy link

kernelogic commented Nov 27, 2020

My client lotus freeze and it keep printing error messages like this. By "freeze" I mean the daemon still prints something, but any cli command for example lotus sync wait will freeze and cannot terminate by CTRL+C.

2020-11-27T08:07:27.307Z WARN storagemarket_network network/libp2p_impl.go:127 exhausted 5 attempts but failed to open stream, err: dial backoff
2020-11-27T08:07:27.307Z WARN storagemarket_impl clientstates/client_states.go:176 error when querying provider deal state: %!w(*xerrors.wrapError=&{failed to open stream to miner 0xc29c4054c0 {[9796235 25912814 25925750]}})
2020-11-27T08:07:27.372Z INFO markets loggers/loggers.go:14 storage event {"name": "ClientEventWaitForDealState", "proposal CID": "bafyreibglvcufodk3t4djxa37tsqamqdar27wnpqzbeo3npt3tyjsdfibq", "state": "StorageDealCheckForAcceptance", "message": ""}
2020-11-27T08:07:28.308Z INFO markets loggers/loggers.go:14 storage event {"name": "ClientEventWaitForDealState", "proposal CID": "bafyreierge755zb3wpx57o4mnhzsozf3bu3rpnaaidlbjvi7psozizevza", "state": "StorageDealCheckForAcceptance", "message": ""}
2020-11-27T08:08:13.273Z WARN storagemarket_network network/libp2p_impl.go:127 exhausted 5 attempts but failed to open stream, err: dial backoff
2020-11-27T08:08:13.273Z WARN storagemarket_impl clientstates/client_states.go:176 error when querying provider deal state: %!w(*xerrors.wrapError=&{failed to open stream to miner 0xc28c345000 {[9796235 25912814 25925750]}})
2020-11-27T08:08:14.273Z INFO markets loggers/loggers.go:14 storage event {"name": "ClientEventWaitForDealState", "proposal CID": "bafyreibsbbtv6a56coy7bybd6z33i73ro5a3ycnzyf3rkxaypt2rnj6upa", "state": "StorageDealCheckForAcceptance", "message": ""}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 P2: Should be resolved
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants