[Deal Making Issue] Retrieval Deals Error: deal data transfer failed: incomplete response

[William8Work]

I wanted to see if there are any online retrieval deals happening in my miner. Found an issue below.

Describe the problem

Many deals failed at "deal data transfer failed: incomplete response" error.

Expected Behavior

data transfer gets retries automatically, or provide more meaningful error details for further investigation.

Version

The output of lotus --version.

lotus-miner version 1.5.3+mainnet+git.3e6dcb578
lotus version 1.5.3+mainnet+git.3e6dcb578

Lotus daemon and miner logs

2021-04-03T08:25:34.880-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventPaymentRequested", "deal ID": "2097", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusFundsNeededUnseal", "message": ""}
2021-04-03T08:25:34.889-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDealAccepted", "deal ID": "2097", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusFundsNeededUnseal", "message": ""}
2021-04-03T08:29:00.647-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventPaymentReceived", "deal ID": "2097", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusUnsealing", "message": ""}
2021-04-03T08:35:01.852-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "2097", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}

And there are quite a few of these errors:

2021-03-28T10:19:46.531-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "714", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "No channel for channel ID --0"}
2021-03-28T10:23:15.304-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "1429", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "No channel for channel ID --0"}
2021-04-01T16:26:17.310-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "1973", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-01T16:26:17.325-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "1919", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-01T16:26:17.326-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "1918", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-01T16:26:17.326-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "1841", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-01T16:26:17.326-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "1858", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-01T16:26:17.326-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "1952", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-03T00:47:45.240-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "2086", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-03T08:35:01.852-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "2097", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-03T15:34:39.147-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "2152", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-03T19:33:10.385-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "2210", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-04T00:28:38.457-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "2314", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-04T11:38:06.136-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "2350", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-04T19:38:07.915-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "2393", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}
2021-04-05T04:43:29.288-0700	INFO	markets	loggers/loggers.go:30	retrieval provider event	{"name": "ProviderEventDataTransferError", "deal ID": "2462", "receiver": "12D3KooWSrVTGNQP7UaXJYB8r6a2bfEcHzt2gnrHrrqwu5zYK1AD", "state": "DealStatusErrored", "message": "deal data transfer failed: incomplete response"}

[jennijuju]

What's the issue exactly?

[William8Work]

updated with more info.

[jennijuju]

updated with more info.

Ah I see - you want an equivalence of lotus client inspect-deal feature for the miner? if so please change the title of the ticket or create a new feature request. Otherwise, the issue itself is duplicated with #5829 and you may try the minerx/staging-v1.6.0 branch which has the new market release that contains a fix for the issue.

[arajasek]

data transfer gets retries automatically

I hope we already do this / are planning on doing this -- @dirkmc do you know?

[dirkmc]

We do automatic retries already for storage deals. We're currently building it out for retrieval deals.

[dkkapur]

@dirkmc - is this addressed as part of the markets 1.3 / resilient retrievals related work? #6149.

If so - might be worth closing this out and asking @William8Work to test once possible.

[dirkmc]

Agreed, closing this ticket.