Parallel fetch for chain sync

[vyzo]

Also fixes stream leaks during sync.

[vyzo]

Marked as ready to review as it seems to be working; I'll wait for my node to do a full sync and figure out how long it takes.

Please make suggestions for improvements, this is step 1.

[vyzo]

rebased on master.

[Kubuxu]

I concurrent reslice and copy into a slice safe?

[vyzo]

It might be safe, I am just not sure -- hence the lock.

[vyzo]

Or do you mean the copy operation on the reslice? That should be perfectly fine, we make a new slice and then we copy.

[Kubuxu]

I 100% missed the lock ;p

[vyzo]

fixed a minor issue in the end of sync, there was an off by 1 in the batch size.

note that it didn't prevent from syncing, it just required an extra call to fetchMessages.

[whyrusleeping]

running this i seem to get a lot of:

2020-09-22T22:21:14.558-0700	WARN	chainxchg	exchange/client.go:112	could not connect to peer 12D3KooWCrx8yVG9U9Kf7w8KLN3Edkj5ZKDhgCaeMqQbcQUoB6CT: failed to read chainxchg response: i/o deadline reached
2020-09-22T22:21:16.885-0700	WARN	chainxchg	exchange/client.go:112	could not connect to peer 12D3KooW9yeKXha4hdrJKq74zEo99T8DhriQdWNoojWnnQbsgB3v: failed to read chainxchg response: i/o deadline reached
2020-09-22T22:21:18.232-0700	WARN	chainxchg	exchange/client.go:112	could not connect to peer 12D3KooWKYSsbpgZ3HAjax5M1BXCwXLa6gVkUARciz7uN3FNtr7T: failed to read chainxchg response: i/o deadline reached
2020-09-22T22:21:18.240-0700	WARN	chainxchg	exchange/client.go:112	could not connect to peer 12D3KooWQYzqnLASJAabyMpPb1GcWZvNSe7JDcRuhdRqonFoiK9W: failed to read chainxchg response: i/o deadline reached
2020-09-22T22:21:18.408-0700	WARN	chainxchg	exchange/client.go:112	could not connect to peer 12D3KooWPkL9LrKRQgHtq7kn9ecNhGU9QaziG8R5tX8v9v7t3h34: failed to read chainxchg response: i/o deadline reached
2020-09-22T22:21:18.648-0700	WARN	chainxchg	exchange/client.go:112	could not connect to peer 12D3KooWKdy1XigjH7mJ8ZM4bULqSJnUYV8UJ6AggGvRSGzWCoSV: failed to read chainxchg response: i/o deadline reached

[vyzo]

Yeah, that's kind of expected, we use more peers so at startup we might select some bad ones.

[whyrusleeping]

LGTM, and works well on my machine!