[WIP] Add Filestore support for the trickle DAG builder

[schomatis]

This needs to be updated after #5135 is done. Any help from the community is welcome.

---

This is a work in progress, not ready to be merged.

As commented in #4052, the support for Filestore will be added only to the Layout function for now, not to Append. This means that ipfs add --trickle will support --nocopy (Filestore), but that added node when copied to MFS won't support ipfs files write (which uses Append).

It builds on top of the #4730 that should be merged first.

Closes #4052.

[schomatis]

Due to this partial Filestore support (for Layout but not for Append) I'm not sure I can just add the SetPosInfo functionality (and add the offset parameter) to the support functions fillTrickleRec and FillNodeLayer as it was done for the balanced builder because they are also used in Append (and related), see this commit of this PR as an example (which doesn't build because the calls from Append weren't adapted). I'm thinking of a few possibilities:

1. Just add SetPosInfo for Append as it won't make a difference because it won't be called for Filestore nodes. This makes it more homogeneous (Layout and Append can use the same support functions) but it will be unnecessarily modifying Append (as it's not defined how, and if, Append will support Filestore).

2. Overload fillTrickleRec (and similarly FillNodeLayer) to call them from Append with "unset" parameters (e.g., offset: -1) that will cause it not to use SetPosInfo in that case. It won't be very nice, it will add many if clauses.

3. Split the functionality of fillTrickleRec in two separate functions, one with Filestore support (with calls to SetPosInfo) and one without (do the same for FillNodeLayer). This will likely cause some code duplication.

@kevina WDYT?

[kevina]

@schomatis I don't know enough about the trickle DAG builder to make to know for sure. I would add some test to t0270-filestore.sh that use the add --chunker=rabin --nocopy ....

[schomatis]

Another (4th) possibility is to refactor the Filestore functionality inside DagBuilderHelper, taking this responsibility away from the functions that rely upon the helper.

The function SetPosInfo is already contained by it, and (following the balanced DAG builder logic) SetPosInfo is always called after DagBuilderHelper.NewUnixfsNode() (see 1, 2), so the offset could be a variable inside the DagBuilderHelper structure and it's value would be updated after each call to DagBuilderHelper.GetNextDataNode (instead of getting it indirectly from the node's FileSize).

This option seems more organic to me because it's the helper the centralized point where data nodes are obtained from, and regardless of how they are organized (balanced, trickle, etc.) their offset will be the same (as long as the nodes are processed in a FIFO order). I would still need some pointers on how to organize this refactoring though.

[schomatis]

@whyrusleeping WDYT? Could you give me some recommendations here please?

[whyrusleeping]

Hey @schomatis, I'm a bit busy working on the 0.4.14 release. If I don't review this in the next couple days please do ping me again.

Also maybe @Kubuxu can help

[Kubuxu]

I will try to take a look at this after the weekend.

[schomatis]

This code has been moved to go-unixfs and with the trickle importer now abandoning the UnixfsNode structure this PR adds little value to justify an extraction there.