Automatically restart push channel

[dirkmc]

Implementation of auto-restart behaviour described in filecoin-project/go-fil-markets#463 (comment)

If a "push" data transfer channel stalls while transferring data, attempt to reconnect to the other party and send a "restart" request for the channel.

Note that the backoff behaviour on dial already exists in the network layer.

This PR adds a pushChannelMonitor to data-transfer manager. Each time the data-transfer manager opens a "push" data transfer channel, it adds the channel ID to the monitor.

Graphsync queues up data to be sent. The pushChannelMonitor periodically checks the amount of data queued against the amount sent. If the amount of pending data (queued - sent), is greater than the configured minimum amount over the configured interval (eg 1MB over 1s), the pushChannelMonitor assumes the transfer has stalled and attempts to send a "restart" request.

[nonsense]

lgtm

[hannahhoward]

Something you should be aware of:
Graphsync has it's own notion of backpressure -- it will stop queuing when a certain among of memory has been allocated but not sent. We built this to resolve graphsync hogging memory if the network slowed down. So, that means, for this to work as a detection mechanism, the value of the diff between queued & sent must be lower than the backpressure default in graphsync. (I believe 16MB per peer). There aren't any defaults listed here, but I think you should inspect the Graphsync defaults: https://github.com/ipfs/go-graphsync/blob/master/impl/graphsync.go#L32, and your defaults which I assume are in the PR on markets, plus make sure Lotus isn't overriding the graphsync defaults (which would not surprise me)

[hannahhoward]

Follow-up: One thing just to factor for the future is that while retrievals seem to be almost no one's priority, we're gonna have to deal with this problem eventually for Pulls too. And, while the ipfs/go-graphsync#129 might help, we may also want to do some kind of keep alive in go-graphsync? I guess maybe go-yamux does that already.

[dirkmc]

@hannahhoward for pulls I'm not sure we need this mechanism, I think it may make more sense to rely on go-graphsync itself to do retries. go-graphsync has some retry capability but it would be nice if it worked the same way as the retries in go-fil-markets and go-data-transfer

[hannahhoward]

LGTM -- just keep an eye out for the back pressure issue when you get to LOTUS

[dirkmc]

I realized that the data-rate monitoring mechanism I implemented was not very granular. If for example the pending data suddenly increased right before checking the data rate, it would appear that the data rate was too low and the monitor would trigger a channel restart.

The latest commit instead adds functionality such that:

• the data-rate is checked and recorded multiple (configurable) times per interval, eg 10
• each check compares the sent amount between the start and end of the interval against the minimum required data rate

[nonsense]

The new algorithm looks correct to me.