Response Assembler Refactor

[acruikshank]

Goal

A common complaint in this code base is tracking the flow from one module to another. This is a proposal to simplify that flow, at least from the standpoint of concurrency issues.

Implementation

Each response in go-graphsync touches several independent threads of execution, which creates additional confusion.
In the first thread, a selector traversal is executed. (see responsemanager/queryexecutor) as blocks load, we queue them up into blocks of response data to send over the wire with the PeerResponseSender (see responsemanager/peerresponsemanager/peerresponsesender.go)

In a second thread in PeerResponseSender, we take the blocks of response data and serialize them down to components that go into an actual graphsync wire message, and queue the message data to go over the wire with the MessageQueue
In a third thread in MessageQueue, we read the next message, serialize to protobuf and send it.

The second thread is the source of much difficulty, especially since it's not entirely independent of the third thread. Currently, the PeerResponseSender takes care of chunking up response data so we don't ever go over a certain size for the resulting message, but then has to stay in sync with the MessageQueue so that each block of response data a separate message goes over the wire.

Now that we've added the notification system, this adds a whole other rats nest of complexity, because we have to subscribe to notifications from the MessageQueue, and then republish them in the PeerResponseSender.

The goal of this refactor is simple: get rid of the second thread.

I started by attempting to do exactly that, and convert the AddResponse method of MessageQueue to operate like buildResponse method of PeerResponseSender. I attempted to move the chunking of responses into the MessageQueue while leaving the AddRequest method unchanged.

However, then I found that created a bunch of complexity and potential problems, which led to a different intuition: what if we made messages immutable, and forced them to be build entirely with a builder class when you wanted to mutate them (side note: this will be of much use should we at some point move to serializing these messages with github.com/ipld/go-ipld-prime). So I converted the ResponseBuilder to a general purpose message Builder, and then removed the AddRequest method from message queue so now we only have a single BuildMessage method.

I renamed the PeerResponseManager/PeerResponseSender to the ResponseAssembler as that's now all it does -- provide a nice DSL for assembler responses and it keeps track of block duplicates as it always has.

Few other extras:

• graphSyncOptions struct in impl/graphsync.go to simplify the construction of the graphsync instance -- it was getting complicated when we had to construct graphsync itself before setting any options.
• Refactored the peerProcess manager a bit to account for the fact that now the ResponseAssembler not longer has a thread. I may remove that abstraction entirely.

To Do:

• renaming pass (check naming - possibly improve, clean up old shortened variable names, correct var names referenced in other modules)
• code commenting pass (make sure public functions documented, add code comments in complicated logic -- i.e. message queue
  Failing tests:
• github.com/ipfs/go-graphsync/messagequeue HARD
• github.com/ipfs/go-graphsync/network EASY
• github.com/ipfs/go-graphsync/peermanager EASY
• github.com/ipfs/go-graphsync/requestmanager MEDIUM
• github.com/ipfs/go-graphsync/responsemanager MEDIUM
• github.com/ipfs/go-graphsync/responsemanager/responseassembler HARD

[acruikshank]

Looks good, a nit and a naming suggestion.

[acruikshank]

I like ResponseBuilder or MessageBuilder more for this. I can see how this 'adds' to a transaction, but the name suggests it 'builds' a transaction in the sense that a message builder builds a message, but it doesn't really. I can also see TransactionContext, TransactionMutator or TransactionAPI as this represents the limited set of message building functionality available within a transaction.

[acruikshank]

Suggested change

	// Transaction build a response, and queues it for sending in the next outgoing message
	// Transaction builds a response, and queues it for sending in the next outgoing message