repo/fsrepo/migrations: verified HTTP migrations

[hacdias]

Closes #9159. This changes the current HttpFetcher to use CAR files. Because we have /ipns/dist.ipfs.tech as default (even though not used), I also implemented DNSLink resolution, as well as IPNS record fetching and verification.

Go tests were updated such that the test server now uses a CAR file backed gateway. This CAR file is generated on the fly before the tests begin. The reasoning behind this versus a static CAR is that the migration file names depend on the platform that they are being run on.

You can also test this locally by spawning a repository, downgrading the version, and running the migrations:

# Build local version (this PR)
make build

# New Kubo repo
export IPFS_PATH=$(mktemp -d)
./cmd/ipfs/ipfs init

# Change version to older version
echo 13 > $IPFS_PATH/version

# Start daemon and accept migrations
./cmd/ipfs/ipfs daemon

[hacdias]

I can't figure out a way of "fixing" the Docker test that checks if Docker downloads the correct version. The problem here is that uses the hardcoded distribution CIDs in the code. But I don't have that CAR file to just serve with socat. And the file would be massive nevertheless.

I would suggest removing this test unless someone has a very good idea.

[aschmahmann]

@hacdias removing that test (or reworking it to not use docker) might be reasonable.

However, if you wanted to keep this test working roughly as is you might be able to work around the large CAR issue by setting the IPFS_DIST_PATH env var to the root CID of a small DAG that you can control + embed.

kubo/repo/fsrepo/migrations/fetcher.go

Line 19 in 4d3cc96

envIpfsDistPath = "IPFS_DIST_PATH"

[hacdias]

@aschmahmann I think the test uses docker on purpose: the idea seems to be that the Docker image fetches the correct migration for the correct arch and platform. I will try what you said and see how it goes. Otherwise, I will just remove it.

[hacdias]

@aschmahmann the environment variable seems to do the trick 😄

[hacdias]

I think we should extract this code into either of:

1. As a Boxo example on "how to download a CAR from the gateway and verify its contents"; or
2. Make a package that does it, or perhaps just inside gateway. There's a lot of "bootstrap" code here that feels a bit annoying to write and having an exported function that does this somewhere in Boxo could be very helpful. Maybe gateway.Fetcher?

[lidel]

Creating an example is good enough, but this is so popular use case we should have a high level boxo lib/API.

Unsure about putting this in gateway package, feels like something that deserves own.

Given we have https://www.npmjs.com/package/@helia/verified-fetch that basically is a similar abstraction, maybe we could have boxo/fetch and VerifiedFetch that works in similar way?

We don't have to do it now, ok to do example now + fill issue for the API.

[lidel]

lgtm, works as expected, thank you for making migrations fully trustless. ❤️
Merge when it feels appropriate.

ps. @hacdias I've been thinking about reusing the CAR fetch code, and turning it into a utility.

Filled issues for follow-up work in:

• Implement trustless and verified HTTP retrieval ipfs-update#179
• Implement trustless and verified HTTP retrieval fs-repo-migrations#188

Doing these is a good opportunity to flesh out how Fetch API library in go should look like.