Create a replaced_by relationship to allow seamless continuation of deprecated mods

[netkan-bot]

This was 2 very different issues, 1 revolved around how to de-index deprecated mods, and one is about introducing a replaced_by relationship. I've pruned this conversation to be directed toward the latter topic as the former has a relatively well established workflow.

• @plague006

[dbent]

What I'd like to see is a new relationship replaced_by that can be added to deprecated mods. replaced_by would be special in that it's evaluated before pretty much anything else.

The logic should be that before a mod is added to the list of mods to install or upgrade it's checked for replaced_by relationships. If they do not exist, we continue as normal, if they do exist however, then the current mod is ignored and all mods listed in its replaced_by are added instead.

For example, say we have AwesomeMod and the maintainer eventually abandons it (no!). New maintainers decide to continue it with AwesomeModContinued. To AwesomeMod we would add:

{
    "spec_version": "1.x",
    "identifier": "AwesomeMod",
    "replaced_by": [
        { "name": "AwesomeModContinued" }
    ]
}

However, eventually the maintainers decide that AwesomeModContinued is just too awesome, and needs to be split into multiple mods (AmazingMod, FantasticMod) to contain its awesome. To AwesomeModContinued we could then add:

{
    "spec_version": "1.x",
    "identifier": "AwesomeModContinued",
    "replaced_by": [
        { "name": "AmazingMod" },
        { "name": "FantasticMod" }
    ]
}

replaced_by would be a fairly strong relationship, in that when adding a replaced_by you're asserting that the replacing mods are a substitute for the replaced mod. In that way mods in a replaced_by stanza can be used to satisfy depends of other mods. This relationship should also be recursive so let's say MediocreMod depends on AwesomeMod and is never updated. After the first update, MediocreMod would transparently become dependent on AwesomeModContinued and then after the next update would transparently become dependent on AmazingMod and FantasticMod.

Potentially you could weaken the relationship and not require it to be a substitute and make use of provides when it is but that only works for 1-to-1 substitutions. In the second case, only the set of AmazingMod and FanstaticMod is a substitute for AwesomeModContinued and neither of them by themselves can be said to provide AwesomeModContinued.

replaced_by should also support the versioning (version, min_version, max_version) fields that other relationship types support.

Thoughts? A slight concern I have is that my favorite package manager (Arch Linux's pacman) has the exact inverse relationship with replaces. I intuitively feel the replaced_by relationship is more robust (allowing things like 1-to-N replacements) but I fear I may be missing something that makes replaces advantageous or replaced_by disadvantageous.

[dbent]

Adding @hakan42 and @Dazpoet, since with the migration of the original issue I don't think you were subscribed to this issue.

[hakan42]

One thing I could think of is:

"replaces" would go into the new ckan/netkan so it would be in control of whoever writes the replacing addon.

"replaced_by" would need cooperation from the author of ckan/netkan of the previous addon.

This issue is purely academical at the moment because all metadata authors are still alive and kicking (and, more importantly, cooperating with each other) at this point in time.

[dbent]

@hakan42 I don't think that's much of a concern, since ultimately the CKAN team has full control of the metadata, even with something like my proposal in #865, the CKAN team can still override metadata if necessary. I imagine it may be a concern with something like multiple repositories, but that's an advanced enough use case that I trust users doing that would be able to figure it out.

A few other things to keep in mind:

• After a ckan update, a mod with replaced_by relationships should be listed as out of date, or perhaps have another special indicator.
• This should work without a version bump. We should be able to add the replaced_by relationship to the latest (or more as necessary) version of the deprecated mod. After an update CKAN should notice the new replaced_by relationship and mark the mod out of date.
• Mods in a replaced_by relationship should implicitly conflict with the replaced mod.
• This should interact gracefully with provides. For example, say MediocreMod depends on AwesomeMod, which is replaced_by AwesomeModContinued. But AnotherAwesomeMod exists and has a provides of AwesomeMod. Now when installing MediocreMod the user should be prompted to either install AwesomeModContinued or AnotherAwesomeMod.

[Dazpoet]

As I see it this is two somewhat seperate issues that got thrown into one. I think the initial issue was the fact that people might have installed mod X in their system was negatively effected if we removed said mod from the metadata no matter how incorrectly it was placed there. Baiscally CKAN (primarily the GUI) hated any mods not having metadata in the upstream. From what I gather that part will be solved by #1005 and great rejoicement shall be had!

As for the continuing issue of mods being taken over, revisited, continued etc I agree that a replaced_by keyword sounds rather amazing, esspecially considering the 1-to-N relationships which are bound to occur. Acctually I could have used one just the other day for KSP-CKAN/NetKAN#1457.

However imaging a future with multiple repositories and perhaps varying updatecycles to those repositories a replaces keyword might be better since that means that adding a mod which replaces will always replace for every user who gets access to that mods metadata. This would prevent them from somehow ending up with both AwesomeMod and AwesomeModContinued due to metadata not being perfectly synced across all repositories. Not sure that's a real issue but it's the only thing I can come up with at this time which would make replaces better than replaced_by.

[mheguy]

I think this issue is timely once again since #1730 will actually allow a replaced_by function.

[politas]

I'd like to get this into the next release; it will solve many issues over people changing the name of their mods and wanting the CKAN identifier changed, too.

[politas]

• After a ckan update, a mod with replaced_by relationships should be listed as out of date, or perhaps have another special indicator.

Yep. I think a new filter and maybe a new column for Replaced mods is probably required. Might be worth creating a status column with icons. This should be easy to add to the CLI list, on the other hand.

• This should work without a version bump. We should be able to add the replaced_by relationship to the latest (or more as necessary) version of the deprecated mod. After an update CKAN should notice the new replaced_by relationship and mark the mod out of date.

I disagree with this. I think we should create a new release, with _lastversion_-obsolete as the version number. Then to accept the replacement, users simply upgrade the mod.

[politas]

On the question of replaced_by versus replaces, I think the important point is that we are saying the new mod is the proper replacement for a mod the user has already chosen. I understand why other packaging systems use replaces, but I think our provides combined with the ageing out of mods due to KSP versions solves those cases adequately.

[dbent]

I disagree with this. I think we should create a new release, with lastversion-obsolete as the version number. Then to accept the replacement, users simply upgrade the mod.

This feels like a hack and the type of thing #1730 was designed to prevent (thanks @ayan4m1 for rebasing that). Ideally there should be a 1-to-1 mapping between CKAN metadata and releases of a mod. Creating "psuedo-releases" just for CKAN is a design flaw in my opinion that should be fixed within CKAN (hence #1730).

[politas]

The trouble then is that we are effectively forcing users to cross-grade to the replacement. They may not want to "upgrade" to the replacement.

[dbent]

Nobody should be forced to do anything. replaced_by should just create a virtual link in the version ordering between one mod and another.

AwesomeMod 1.0 --> AwesomeMod 2.0 -(replaced_by)-> AwesomeModContinued 1.0

If a user doesn't want to use AwesomeModContinued 1.0 they simply don't upgrade from AwesomeMod 2.0. Same as if they don't want to use AwesomeMod 2.0 they don't upgrade from AwesomeMod 1.0.

[politas]

But CKAN isn't modeling "links" between versions. The data objects we have are "Modules" and "Versions". We can fairly easily create a virtual "Version" which points to a different "Module", but #1730 forces reinstallation in order to meet new relationships. So anyone having the version with "replaced_by" would be forced to upgrade, because their registry would show up as inconsistent.

[dbent]

So anyone having the version with "replaced_by" would be forced to upgrade, because their registry would show up as inconsistent.

Only if it's written that way and I don't see why it would be. replaced_by is an entirely new relationship. If it was added today, #1730 would have no effect on it.

#1730 forces reinstallation in order to meet new relationships.

Not quite, both with #1730 and prior to it the user is prompted to reinstall. At some point, the decision was made that (for whatever reason) GUI interactions would automatically accept confirmation dialogs while the CLI would actually prompt the user for a decision (GUI easy mode? CLI advanced mode? /me shrugs). Again, there's no reason why replaced_by has to behave the same. You can add a ForSuperSrslyGuiRaiseYesNoDialogAndIReallyMeanItThisTime() method to the IUser interface explicitly for this case.

Links between versions are modeled implicitly by they're ordering. 1.0 < 2.0 therefore 1.0 and 2.0 are "linked". Right now this ordering only considers modules with the same identifier. What replaced_by should do is create a mechanism that effectively slides in a mod with a different identifier into the ordering.

AwesomeMod 1.0 < AwesomeMod 2.0 < AwesomeModContinued 1.0 < AwesomeModContinued 2.0

But again, CKAN should notice this and prompt the user explicitly.

And as a matter of policy as I said originally:

replaced_by would be a fairly strong relationship, in that when adding a replaced_by you're asserting that the replacing mods are a substitute for the replaced mod.

The number of people using AwesomeMod that would not want to use AwesomeModContinued should be very, very low if it's a true substitute and not just an "alternative".

[politas]

But if you have AwesomeMod 1.0 installed and AwesomeMod 3.0 is listed as the latest, CKAN will ignore AwesomeMod 2.0 and go straight to AwesomeMod 3.0. I'm really not seeing the concept of a "link" showing up in the CKAN architecture. It's not how CKAN models it. We have the installed version and the latest version, or using the CLI, users can specify any specific version. There's no chain of releases, just a collection that happen to sort into a particular order.

How do we show that a mod has been replaced in the modlist? What do we put as AwesomeMod's latest version if AwesomeMod 3.0 is replaced_by AwesomeModContinued 1.0? Under my proposal it would show as 3.0-obsolete and then a "Replaced By" Column would list "AwesomeModContinued". This would also mean that if the original mod returned with say AwesomeMod 3.1, it would become the new latest version and the Replaced_by would automatically disappear. No one would be sitting in a state where they have AwesomeMod 3.0 installed and be unable to upgrade to AwesomeMod 3.1 until we make a change to AwesomeMod-3.0.ckan.

[dbent]

But if you have AwesomeMod 1.0 installed and AwesomeMod 3.0 is listed as the latest, CKAN will ignore AwesomeMod 2.0 and go straight to AwesomeMod 3.0. I'm really not seeing the concept of a "link" showing up in the CKAN architecture. It's not how CKAN models it. We have the installed version and the latest version, or using the CLI, users can specify any specific version. There's no chain of releases, just a collection that happen to sort into a particular order.

Like I said, the link is implied, it's not explicit anywhere.

How do we show that a mod has been replaced in the modlist? What do we put as AwesomeMod's latest version if AwesomeMod 3.0 is replaced_by AwesomeModContinued 1.0? Under my proposal it would show as 3.0-obsolete and then a "Replaced By" Column would list "AwesomeModContinued".

Not sure how that's materially different from showing AwesomeMod 3.0 as the latest version with a column with Replaced by AwesomeModContinued 1.0.

This would also mean that if the original mod returned with say AwesomeMod 3.1, it would become the new latest version and the Replaced_by would automatically disappear. No one would be sitting in a state where they have AwesomeMod 3.0 installed and be unable to upgrade to AwesomeMod 3.1 until we make a change to AwesomeMod-3.0.ckan.

It would only disappear in the sense that it's being ignored, the metadata would still be there for all time as a source of confusion: "Why is there a 3.0-obsolete if there's a perfectly good 3.1 right here?".

Anyway, if the original mod returns with a 3.1 it seems perfectly reasonable to take a moment and go "Woah, what's going on here?" replaced_by shouldn't be added willy-nilly. replaced_by should only be added in cases where we're pretty darn sure the original mod isn't coming back. Really the only situations replaced_by should be used are:

• A mod has been truly abandoned (not just a no new releases in two weeks kind of thing) and another maintainer is reviving it.
• The original author is deprecating/renaming it.

In both those cases if the original mod suddenly pops back into existence, that seems like a good time to reevaluate the situation and not inadvertently split the user base between two mods, causing a potential host of confusion.

But really, that situation is super rare. The only time I can think of a mod being "replaced" and then the original coming back was when kOS was adopted by new maintainers and then the original maintainer came back after some time and made like one new release (kOS-Classic) before it was abandoned again.

[politas]

Well, we have a perfect case at the moment, where BahamutoD is taking a break from KSP modding, and various people are taking over his mods, but it is expected that BahamutoD will return at some stage.

[HebaruSan]

Well, we have a perfect case at the moment...

How did that turn out? It could be useful to examine now that the code is moving forward again.

[politas]

How did that turn out? It could be useful to examine now that the code is moving forward again.

I believe BahamutoD is still on break from KSP modding. I think #1888 caters for it, in any case. The replaced_by applies only to the specific version. If an upgrade appears after a replaced_by has been set, the users will have the option to either upgrade or replace (and after upgrading, they will no longer see the replacement option unless it is re-applied to the new version's metadata)