Update graphsync & fix in-progress request memory leak by consuming responses #109
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goals
Resolve major in process request memory usage by updating graphsync (work done in repo) and consuming in progress graphsync response channel.
Implementation
For discussion
Here are heap profiles of in use memory objects assembled by running:
Essentially, we're running a 1GB data transfer over a connection gated at 16MB per second -- it will stop in the middle after 10 seconds, and we take the heap profile here.
First, with graphsync v0.3.1:
Note the two areas in red -- the one on the left is memory use on the responder side and the one on the right is memory use on the requestor side.
Now, with graphsync v0.4.1:
The responder side memory usage is massively reduced (the worst issue) but we still have accumulated data on the requestor side.
Now, adding consuming the response channel in go-data-transfer:
The requestor side memory usage is gone.