Plan proposal for future of fs-repo-migrations

[hsanjuan]

State of the art

fs-repo-migrations is a binary that knows how to migrate repository versions since version 1 to soon-to-be version 9.

Each individual migration is a go-module (and a go executable) but they are all wired together in fs-repo-migrations, which is published on dist.ipfs.io and consumed by ipfs and ipfs-update (in very similar ways) when a migration is necessary (either by http-download or by ipfs-download).

Each migration module needs to be built offline, and they use vendored dependencies. Due to the long history of migrations, these includes library copy-pastes added as extra modules, gxed-vendored dependencies added as submodules and go-mod vendored dependencies.

Migrations are mostly tested with single, non vendored sharness tests, with some deps depending on iptb.

Current problems

• The fs-repo-migrations binary is now 54M (to version 7), soon to grow
• There are some libraries (opencensus tracing related) that only allow to be init()ed ONCE. Multiple subpackages may vendor different versions and then fail badly. This needs patching across all migrations to prevent init() from happening or to align the libraries.
• Things like /net/trace (if I remember well) do not allow multiple versions and fail. This requires alignment between all submodules.
• Resolving the above involves touching very legacy code and potentially breaking very old migrations. It is horribly annoying, any new contributor will need a core dev to give a hand if this happens.
• It has been discussed to deprecate old migrations (Consider deprecating old migrations #93) but apparently we do not want that.

Proposal

We should make it easy for anyone to drop in and write a migration. Migration code should be fully isolated and testable in isolation. We should prevent having to touch any of the old migrations, or to having to adjust test things which affect all migrations and end up becoming a time sink on the developer.

Thus:

1. fs-repo-migrations binary is deprecated and we stop using it. We build migration-specific binaries for all migrations.
2. The migration specific binaries are produced in isolation and fully, correctly vendored, without the implementer having to fix retroactively older migrations when a dependency issue comes up. This binaries are in the form ipfs-version-to-version. Tests for new migrations (sharness or not) are added in the submodule directly for new migrations
3. Binaries are built and published separately in go-migrations.ipfs.io (using the same mechanism as dist.ipfs.io).
4. A common library to download and install the right migration binary (and run it) is created as submodule in this repository, using the current migration code from ipfs-update and go-ipfs. It should be able to pick the right binary to run every migration and execute them. For new migrations this should speed things up a bit. ipfs-update and go-ipfs are switched to use it.

This is not much work but, I hope, will facilitate the life of anyone needing to write a migration in the future a lot.

[gammazero]

Modifications to the proposal:

1. fs-repo-migrations will not be deprecated, but will be changed so that it knows how to download the necessary migration binaries from the distribution site, unpack and run them in sequence to perform a migration. This will be done using logic that is common to go-ipfs and ipfs-update. This way a user will not have to download and run individual binaries in the correct sequence for migrating or reverting a migration.

The code for the common library described by 4 should live in go-ipfs/repo/fsrepo/migrations instead of in this repo.

[gammazero]

This feature (overhaul of migrations) is implemented in the following PRs:

• Migrations distributions#327
• Fix/separate migration bins kubo#7857
• Migration downloads individual migration binaries ipfs-update#134
• Give each migration its own binary #116

All tests are currently passing using a temporary IPFS distribution (/ipfs/QmXt92hFRuvQgFhgHoaMxC4wLFcvKsCywQPTNmPYCGfEV4). After the distributions PR is merged, we can then publish a new official distribution, pin that, and point our distribution site at it.

[gammazero]

Work on this feature is completed and deployed. Closing.