Distribute repo migrations as set of binaries

[gammazero]

Currently, there is a single monolithic binary responsible for all repo migrations, growing with each release that needs repo migration. This will no longer be the case going forward. Repo migrations will be available as a set of smaller binaries, one for each version migration (e.g. ipfs-9-to-10, ipfs-10-to-11). These will be downloaded and run individually by go-ipfs, ipfs-update, and fs-repo-migration. To facilitate this, some changes to dist.ipfs.io are needed.

Directory structure changes

The plan is to store the store the migrations under a common root, dist.ipfs.io/go-ipfs-repo-migrations/:

go-ipfs-repo-migrations/
|-- ipfs-10-to-11/
|   |-- versions
|   `-- v1.8.0/
|       |-- ipfs-10-to-11_v1.8.0_darwin-amd64.tar.gz
|       |-- ipfs-10-to-11_v1.8.0_linux-amd64.tar.gz
|       `-- ipfs-10-to-11_v1.8.0_windows-amd64.zip
`-- ipfs-9-to-10/
    |-- versions
    `-- v1.8.0/
        |-- ipfs-9-to-10_v1.8.0_darwin-amd64.tar.gz
        |-- ipfs-9-to-10_v1.8.0_linux-amd64.tar.gz
        `-- ipfs-9-to-10_v1.8.0_windows-amd64.zip

The URL to retrieve a 10-to-11 migration for darwin is https://ipfs.io/ipns/dist.ipfs.io/go-ipfs-repo-migrations/ipfs-10-to-11/v1.8.0/ipfs-10-to-11_v1.8.0_linux-amd64.tar.gz

Website changes

There should not be any visible changes to the dist.ipfs.io web pages, since the current plan is to not show the users any links to individual migration binaries. New versions of fs-repo-migrations will continue to be available on the dist site, and will know how/where to download the necessary migration binary archives, unpack and install them. There will not be a need for users to download individual migration binaries themselves.

Should it be necessary to download a specific migration binary, a URL like the example above could be used. If there ever is a need for users to do this, then a link from the main page to the top-level go-ipfs-repo-migrations could be added, but this does not appear to be needed for now.

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[aschmahmann]

This seems like a reasonable plan. In order to support it we'd need to add some syntax into the scripts for creating/adding binaries that live inside of folders.

WDYT @Stebalien @olizilla @hsanjuan

Note: If we wanted to we could make this easier by putting the binaries like ipfs-10-to-11 at the top level and just having a flag for hiding them. This folder approach seems like it might be helpful to us going forward (e.g. if we wanted to add a link that references all the ipfs-x-to-y binaries together), but if it's a huge pain we could potentially forgo it.

This is related to ipfs/fs-repo-migrations#98

[olizilla]

Sounds good. I am mildy in favour of hiding the many migrations under it's own directory.

[jessicaschilling]

@aschmahmann @olizilla Is this something either of you is able to follow up on? Afraid the GUI team is a bit slammed at the moment.

[aschmahmann]

@jessicaschilling I suspect we'd hide all the GUI changes for now and just change the behind-the-scenes things first (i.e. add the data to dist, but not expose it on the home page).

We can then circle back to the UI side of this later

[gammazero]

In the interest of not making a one-off distribution that contains all the individual migrations, another option is to make a new distribution for each migration, but without adding it to the web GUI.

@aschmahmann @olizilla Does this approach seem better? Would this allow us to only have to publish the latest migration with each release that needs one?

[gammazero]

After experimenting with organizing the individual migrations under their own distribution root directory, or keeping them each as individual distributions at the top level of distributions, the later seemed like a better choice for now. The reason for this is:

• There is no web content for migrations, so no need to clean up visual clutter on the distributions page
• Easier, more intuitive, to find because their ipfs path structure is the same as any other distribution
• Does not introduce (so far unneeded) complexity into build scripts

[gammazero]

Each migration is an individual distribution that is not displayed on the web page of the distributions site. For example: https://dist.ipfs.io/fs-repo-10-to-11. ipfs-update, fs-repo-migrations, and go-ipfs know how to download necessary migration binaries from the new distributions.

All work on this issue is completed and deployed. Closing issue.