[Design/discussion/decision] Whether to "migrate" or "copy" repos into go-libipfs

[BigLep]

Purpose

Given concerns raised by others within PL EngRes about the IPFS repo consolidation effort, this issue is being used to solicit/design/discuss/decide alternatives that can be followed while still meeting the primary goals of the consolidation effort being spearheaded by the EngRes IPFS Stewards. The EngRes IPFS Stewards is planning to push forward with more repo consolidation in 2023Q1, but are up for considering alternative approaches before doing so.

Background (on the existing migrate approach)

The limited effort so far of consolidating various ipfs/* repos into go-libipfs has used https://github.com/guseggert/repo-migration-tools largely following the Example Workflow section of the README.

This has included:

• Copying code from one repo into a subdir of go-libipfs (while preserving commit history, fixing broken merge commit links, adding a note at the bottom of each commit with where the commit originally came from)
• Transfering GitHub issues from one repo into go-libipfs, adding the repo as a prefix to the issue name
• Leave a note on open PRs in the original repo and closing them (since it is not feasible to transfer PRs).
• Making it a more graceful migration with:
  • stubbing out types in the old repo by making type aliases that point to types in go-libipfs
  • tag & release a final version of the old repo
• Archiving the repo
• In some cases proactively updating consumers of the migrated repo

Pros of the existing migrate approach

1. Clear signal for consumers of the migrated repo where the new development is happening
2. Easier maintenance burden from the regard of bug fixes or security fixes only need to be made in one place (go-libipfs)

Cons of the existing migrate approach

1. Consumers of existing repos need to do work including changing their imports and go.mod dependencies.
2. Consumers of existing repos who haven't updated other key dependencies of go-libipfs like go-libp2p may be forced into larger dependency upgrade issues as go-libp2p has had various API breakages over the year.
3. Existing PL EngRes engineers who are used to bug-fixing/patching/releasing repos are now dependent on reviews and releases happening through go-libipfs. SLAs for reviews and release processes haven't been well defined yet as of 2023-03-02, which leaves ambiguity and uncertainty on when a given PR will be reviewed/merged/released. Before a PL EngRes engineer felt like they could own their destiny in a more self-service fashion.

Alternative approach 1: copy (don't migrate)

Leave the repos as they were (i.e., don't archive) and instead just copy them into go-libipfs. Specifically, this would mean:

1. Copying code from one repo into a subdir of go-libipfs (while preserving commit history, fixing broken merge commit links, adding a note at the bottom of each commit with where the commit originally came from)
2. Copying (not transferring) GitHub issues over.
3. Adding a clear README notice to the existing repo that:
4. There is an actively maintained version in ipfs/go-libipfs/subdr
5. The existing version does not have a maintainer.

Pros of copy approach

1. No disruption for existing consumers of these ipfs/* repos that have moved into go-libipfs. They can opt-in at a later point if they want/need the maintained version in go-libipfs.

Cons of copy approach

1. Security / bug fixes need to be made multiple places. It is unclear if/who will make the change for the original/non-go-libipfs version.
2. Consumers of the repos don't have as clear of a signal that likely the repo that they're depending on is without maintenance.

If there is a security issue in a repo, what will happen?

EngRes IPFS Stewards will certainly handle patching go-libipfs. If there are maintainers for the existing repos, they will need to handle patching/disclosing. EngRes IPFS Stewards will certainly coordinate and share work, but won't handle communication with or updating of the affected consumer. If there are no maintainers for the existing repos, there will likely be internal private PL EngRes chats analyzing which projects are impacted, and it will then be up to those projects to determine whether they want to patch the existing repos or migrate to go-libipfs.

Alternative approach 2: copy (don't migrate) but deprecate the types

This is the same as "Alternative approach #1: copy (don't migrate)", but also deprecates the Go types with a clear signal as to where the maintained version is. This has the same pros before in addition to providing a better signal for consumers.

What do we gain from these approaches

EngRes IPFS Stewards primary goals are still satisfied. TODO: link to what these are (since I don't want to duplicate them here).

Next Steps

• Flush out the options
• Surface this discussion with parties who have been concerned about this repo consolidation effort to give feedback on which approach(es) they'd prefer.
  2023-03-09: @BigLep didn't present this as a vote given taking a more lax approach and have a clear path for anyone who doesn't want the deprecated messages.
  • Consolidate IPFS Repositories kubo#8543 (comment)
  • [Milestone] Getting boxo "over the hump" #196 (comment)
• EngRes IPFS Stewards take the input and make a decision (reflected with a comment back in this issue)

Note: regardless of what path is chosen, EngRes IPFS Stewards still have key work they can do, specifically around:

1. Document the planned import scope and order of ipfs repos #174
2. Document the release process #170
   as these are needed for all paths.

Cases where concerns about the existing migration approach have been raised

1. All through the original IPFS repo consolidation issue: Consolidate IPFS Repositories kubo#8543
2. #go-libipfs-maintainers thread: https://filecoinproject.slack.com/archives/C04M8232QRW/p1677103751480109
3. PL EngRes internal #ipfs-nexus thread: https://filecoinproject.slack.com/archives/C03FPDYG6T0/p1677326540132039

[BigLep]

Note: the above issue description is just a starting point. @ipfs/kubo-maintainers please update/correct/add a will. Other interested parties and folks expected are welcome to comment as well.

[BigLep]

My vote is to go with "Alternative approach 2: copy (don't migrate) but deprecate the types". I don't think we have the TPM communication power for the "existing migration approach", we're not being as disruptive, and we're still giving consumers clarity on where they can find maintained code. 6 months from now if this experiment was fruitful we can come through and archive the old repos.

[BigLep]

@ipfs/kubo-maintainers met on 2023-03-07 about this. Here is the outcome of that conversation:

What approach are we planning to take?

We're planing to do "Alternative approach 2: copy (don't migrate) but deprecate the types" from above.

When is this going to happen?

We expect to make tangible headway the week of 2023-03-13 but we are ensuring we have all our ducks in a row first. Updates will be posted in the tracking issue #196. #174 will be creating the tracking checklist/table where the incremental progress will be reported.

What if someone is annoyed by the deprecation warnings?

They have a path forward. They have to explicitly add a commit/PR that includes:

• Revert of the deprecation types
• Remove the "not maintained" notice from the README
• Add themselves to CODEOWNERS

This satisfies the requirement about being clear to users about who owns and its status.

Are we archiving repos?

No, we're not archiving repos. If 4 months from now there hasn't been any activity on the repo, we'll go ahead and archive the repo. This is being tracked in #201

What are we going to do with the issues in the repos that we're copying in?

To not disrupt the repo, we’re not going to transfer the issues. Instead, we’ll read the repo's issues and for the relevant items, create a new issue in go-libipfs and link to the original.

What messaging are we putting on the repo READMEs?

This will get handled in #174

What about repos we’ve already migrated into go-libipfs?

We're going to leave these repos as is (fully migrated over and original archived):

• bitswap
• files
• tar

We will "revert" the migration of https://github.com/ipfs/go-block-format meaning:

1. Unarchive the repo
2. Copy back issues
3. Adjust the messaging to say it isn't maintained.

What are we importing explicitly?

This will be answered in #174

How will someone find the repos that have been imported?

This will be self-service from the tooling being developed in #181. That tooling will end up living in go-lifipfs main.

[BigLep]

#174 has been updated based on the plan in #191 (comment)

[BigLep]

I pointed at this from ipfs/kubo#8543 (comment)

I'm not planning to do any additional communication on this. Additional comms will happen in the parent tracking issue (#196 ) and the other issues therein.

[BigLep]

It was briefly stated in the 2023-03-09 PL EngRes All Hands

It is surfaced in the 2023-03-10 IP Stewards sitrep: https://www.notion.so/pl-strflt/2023-03-10-IP-Stewards-SitRep-51463e1d1283495c91289dfe576596b9?pvs=4#a25873a00eab427282f50a1a6f59fd4e

[BigLep]

2023-05-10: this content has been moved into https://github.com/ipfs/boxo/wiki/Copied-or-Migrated-Repos-FAQ

This comment serves as the pointer in the "not maintained README notice" for repos that were copied into Boxo. This can be updated as needed.

A more durable location will be found, but currently the best source of information about these "not maintained" repos is in #190. The FAQ covers the following questions as of 2023-03-27:

• What repos have been copied into Boxo?
• What will happen if there is a security issue in one of the legacy ipfs/* repos that has been copied into boxo?
• I don't like the "deprecated" warnings in "not maintained" repos that have been moved to boxo. What can be done about this?
• How does one claim ownership of a "not maintained" repo?
• Will the "not maintained" ipfs/* repos be left around to rot?

If you have other questions or need help, please reach out to @ipfs/kubo-maintainers at https://github.com/ipfs/boxo#help

[BigLep]

Given the decision to copy, having a mechanism for knowing if the source has made "important updates" will be handled in #270