Switch to raw multihashes in datastores

[Stebalien]

Part of #4143.

Currently, we store blocks by CID in the datastore. However, a single block can have multiple CIDs:

• CIDv0/CIDv1 -- blocks created as cidv0 can be referenced with v1 CIDs
• CIDv1-*/CIDv1-raw -- we can treat any block as a "raw" block
• CIDv1-cbor/CIDv1-dag-cbor -- theoretically, we have non-"dag" versions of our codecs.

Plan:

TODO: put this in the right spot: ipfs/fs-repo-migrations#95

• Create a repo migration to switch from CIDs to multihashes. Luckily, all CIDv0 blocks (most blocks) are valid multihashes so we don't have to move them.
  • Migration 11-12: migrate CIDsv1 to raw multihashes fs-repo-migrations#95
  • This migration should iterate over all blocks in the datastore:
    • If the block is stored using CIDv1, we should add it again using the raw multihash (if it's not already there).
    • Delete the CIDv1 block.
    • If the datastore offers transactions, we should ideally perform each add/delete in a transaction.
    • We should probably have multiple threads doing this for better performance.
    • We should do this with synchronous writes enabled. Alternatively, we could do this in two passes (add all then delete all) but that's going to double the amount of disk space used.
  • Optional: Iterate over all provider records and migrate them as well. This is not critical as these will expire anyways.
• Convert CIDs to/from multihashes in the blockstore layer. At this point, the external blockstore interfaces will still use CIDs.
  • feat: switch to raw multihashes for blocks go-ipfs-ds-help#18
  • feat: switch to raw multihashes for blocks go-ipfs-blockstore#38
• Fix GC (in go-ipfs) to work when the datastore no longer stores codec information.
  • feat: switch to raw multihashes for blocks #6816
• Remove the previous CIDv0/CIDv1 blockstore hacks.
  • feat: switch to raw multihashes for blocks #6816
• Eventually, switch the blockstore API to use multihashes directly instead of CIDs.
• 2021-12-14: this will get handled as part of blockstore: switch from Cid to Multihash boxo#361

---

2021-12-02 context on why were' doing this:

• Helps us to move towards CIDv1 by default
• Deduplicates data that’s stored multiple times (e.g. as CIDv0, CIDv1-DAG-PB, CIDv1-Raw)
• Removes the technical debt and tribal knowledge around the different interfaces used by go-ipfs and anything newer (lotus, venus, estuary, …)
  • Unlocks bigger refactors around blockstores going forward
• Enables any pinning services backed by go-ipfs to serve content for unknown IPLD codecs
  • e.g. all those CAR files filled with Bitcoin blockchain data could be stored by services that don’t have the Bitcoin codec
  • Allows for greater experimentation from groups making their own codecs even before they land their codec in go-ipfs by default

[Stebalien]

@lidel if you find yourself with some time, it would be really nice to get this in before 0.5.0. We're already doing one migration, so doing a second isn't a big deal.

[Stebalien]

The final pieces here are to:

1. Write the actual migration: https://github.com/ipfs/fs-repo-migrations
2. Take this the rest of the way (write a migration test, write a sharness test to make sure we don't end up storing blocks twice, etc.).
3. Go over everything I've done to make sure it makes sense. This was thrown together rather quickly.

[ianopolous]

Just a note that this affects us at Peergos as we have just implemented our own blockstore which needs to be compatible with IPFS's. I see your logic and it shouldn't be a problem for us.

[ianopolous]

One small comment is that GC can significantly benefit in speed by knowing that a block is raw and hence has no links in it - i.e. it doesn't need to retrieve the block at all to follow the links. This benefit is lost in this model. This affects remote blockstores like S3 a lot, especially as the genuinely raw blocks tend to be much larger than cbor blocks.

Although I might be wrong because you are still theoretically starting from a GC root which is a cid, so I think that means you don't lose the benefit I just stated?

[Stebalien]

Thanks for the heads up.

Although I might be wrong because you are still theoretically starting from a GC root which is a cid, so I think that means you don't lose the benefit I just stated?

Correct. GC already implements this optimization, as far as I know. It does two things.

1. It'll start at the typed root (CID) and walk the graph. If it encounters a "raw" node, it won't bother fetching it.
2. It'll list all blocks in the datastore (but won't fetch them).
3. It'll delete everything from step 2 not found in step 1.

[Stebalien]

Update: @lidel is busy.

@hsanjuan how would you feel about picking this up? The main pieces left are in #6815 (comment).

[Stebalien]

Note: #6817 should make fixing the tests easier.

[danimesq]

@Stebalien , could IPFS do inspiration in git for data deduplication?

[Stebalien]

IPFS does do data deduplication, like Git. (if you'd like to discuss this, let's talk on the forums)

[hsanjuan]

Sorry @Stebalien, this only came now to my attention. I would not block release for it or make it depend on me, as I have very little availability in the next two weeks. That said, I could work on it (and would like to), but we are looking at the end of january...

[Stebalien]

That's fine. We'll see if someone has time before then but that's unlikely.