Should format exist in leaves?

[btc]

ipfs block put problem

@jbenet

[whyrusleeping]

Could you elaborate a little more on this? I worked on most of the formatting stuff (unixfs) and could probably provide some insight

[btc]

@jbenet can elaborate

[jbenet]

This is just a note to myself to figure out whether leaves in the merkledag should be allowed without a format. We discussed before and opted for forcing format. The issue that emerged is this:

anyone could publish a block on ipfs that doesnt follow the merkledag format

Even if most nodes check it and discard non-properly-formatted blocks, we want to be able to store encrypted blocks, which would dispense with the format.

1. if we keep forcing the format, we'd have to setup encryption such that the link table and data are encrypted separately and the format is visible. This leaks information about the file. bad.
2. force the format for in-the-clear nodes and deal with it if nodes store non-formatted encrypted blocks.
3. dispense with the format entirely, allowing merkledag leaves (they'd have to be leaves) to be any opaque data.

I don't like 3. because it breaks self-description. We might be able to contrive a format -- if we do define our own format, and not protobuf, for the merkledag node -- where the link table is optional, but recognizable. if parsing fails on the link table part, implementations assume it's a leaf with no links.

[whyrusleeping]

👍

[daviddias]

Let's bring this discussion to the IPLD thread ipfs/specs#37.

[whyrusleeping]

This should be resolved by CIDv1

Ref #3059

[whyrusleeping]

progress #3307

[whyrusleeping]

We now have the ability to create files that have no formatting in the leaves via ipfs add --raw-leaves. This option may be made the default at some point in the future.