[RFC 0054] Support levels and tiers

[CrystalGamma]

This RFC describes a scheme for expressing and managing support of packages and for platforms in nixpkgs by defining support levels for packages and support/acceptance tiers for platforms.

Rendered version

[shlevy]

This RFC is now open for nominations. Any volunteers?

[7c6f434c]

Maybe something in the direction «every package of support level X that has a source, must correspond to a depepndency of the support-level-defining expression with the same source»?

[CrystalGamma]

Sorry, I'm having trouble understanding the suggestion … what do you mean by "source"? the src attribute? How would you check this?

[7c6f434c]

We can enumerate all the derivations (say, using release.nix). Most derivations have src or srcs. In a single Nixpkgs checkout, «same» source usually means exact match of the source derivation output.

[CrystalGamma]

So you'd check if the src is somewhere in the (build) closure of the defining expression? But doesn't closure-info only include the closure of the output?

[7c6f434c]

Does this require PMG to be large enough for the next support level?

[CrystalGamma]

I haven't thought about that so far, but I'd say it makes more sense to only require the requisite number when raising the support tier (not level, support levels are for packages – maybe we should find a different name for one of them …).

[7c6f434c]

Is the suport-acceptance gap allowed to become two in this case?

[CrystalGamma]

Good question. I'd argue it shouldn't, since the purpose of having a small gap is making maintainers focus on the basics first by denying PRs for minor packages. In practice I can't imagine a platform mature enough to warrant raising the acceptance level suddenly being abandoned. I suppose it might make sense to prescribe revoking the elevated acceptance level before dropping the support level.

[7c6f434c]

@CrystalGamma I can remember such a thing in Nixpkgs, actually. Depending on the details of the defnitions, i686-linux might have eventually dropped from the flagship all-tier platform to a platform where part of system-level packages might be broken without too many people noticing.

[CrystalGamma]

i686-linux in particular might be gnarly because its main use nowadays is as a dependency for wine on x86_64-linux, so it doesn't need everything from system but does need a lot of things like sound libraries that are out of scope for system … we could get around that by defining support levels/tiers not as total orders, but lattices, with an additional wine support tier/level as a sibling of system, but that sounds like it could easily lead to exactly the kind of complexity I was trying to avoid :(

[7c6f434c]

You have an option to say that there is an x86_64-linux Wine package (with multiarch support) that still depends on the same set of things, so these are supported in the interest of x86_64-linux by Wine package maintainers.

I do hope that the reference system configuration for the system package includes some kind of sound support, though!

But I think that we cannot exclude that some other platform goes the same way.

[CrystalGamma]

Sure, system should probably include things like libpulse and whatever the low-level libraries for ALSA are called … I was thinking of libraries like OpenAL or cubeb (then again I don't actually know what wine uses, I was just recalling some dependencies of random video games I've seen run on Ubuntu and assumed wine might use those as well, kind of silly of me, thinking about it now).

[7c6f434c]

vkd3d might be a good example of a thing useful for Wine and really not making sense in system

[CrystalGamma]

I suppose it makes sense to say that i686-linux inherits x86_64-linux's acceptance tier where it makes sense for multilib.
It's probably sensible to have a provision for multilib somewhere in the RFC, in case that, as you say, it becomes relevant for another platform.

[Ericson2314]

@7c6f434c do you wish to withdraw #46? That RFC already has a shepeard team from a previous MR. @shlevy I would suggest we just carry over the same band of people again.

To be clear, the slow process is almost entirely my own fault. I think it's very good that as much RFCs have at least one meeting before NixCon, so I will try to not let the ball down with whatever I end up shepherding.

[7c6f434c]

Due to conflict of interest, my first round of comments here assumes that «streamlined is obviously better»; outside this context I do not hold this conviction strongly enough to withdraw my alternative proposal immediately.

It does make sense to have an interesecting (or same) group of shepherds so that there can be a detailed discussion which proposal to push to completion.

[CrystalGamma]

I've been thinking … would it make sense to rename what I named "support level" (for packages) to "scope"?

There's already been confusion in the discussion between that and the "support tier" of platforms.

Renaming it "scope" would avoid that and no longer carries the assumption that the scopes are totally ordered, which would simplify the categorization of platforms mainly used for multilib (which was something I didn't think about when writing the RFC).

[7c6f434c]

I've been thinking … would it make sense to rename what I named "support level" (for packages) to "scope"?

Regardless of the structure, it does make make the two notions less similar-sounding

[7c6f434c]

So you'd check if the `src` is somewhere in the (build) closure of the defining expression? But doesn't closure-info only include the closure of the output?

If we check the closure of the derivation file after instantiating the defining expression, it will include all the derivation files of all the build steps. You are right, that if we check the closure of the output after building an expression, indeed only the runtime closure is included. But we do not want the check to cost a full rebuild anyway.

[CrystalGamma]

That would need some code outside of Nix expressions to check, right? I was thinking of the closureInfo function, which AFAICT will only output the closure of the output path. But if we made a script that collected the closure of the derivation described by the defining expression and extracted the output paths of all the fixed-output derivations contained therein, we could add another nixpkgs option called, say, restrictSrc which gets the set of those paths and for each package/stdenv.mkDerivation checks if src/srcs contains that path.

I don't know the details of how Hydra is run, but this does sound reasonably easy to implement.

[alyssais]

I think a worked example would be useful in explaining this. Suppose I wanted to reintroduce illumos support. What would and would not be expected at each point in the journey? I'd having trouble figuring that out from just reading the RFC.

[globin]

The @NixOS/rfc-steering-committee has decided to ask for new nominations here, since #46 will remain open. Please do come forward with nominations and also feel free to nominate yourself!

Quoting RFC #36 for the responsibilities of the shepherd team:

Shepherd Team

A team of 3-4 community members defined unanimously by the RFC Steering Committee, responsible for accepting or rejecting a specific RFC. This team is created per RFC from community members nominated in the discussion on that RFC.

This team should be people who are very familiar with the main components touched by the RFC. The author cannot be part of the Shepherd Team. In addition, at most half of the Shepherd Team can be part of the RFC Steering Committee.

The responsibility of the team is to guide the discussion as long as it is constructive, new points are brought up and the RFC is iterated on and from time to time summarise the current state of discussion. If this is the case no longer, then the Shepherd Team shall step in with a motion for FCP.
Shepherd Leader

The person in charge of the RFC process for a specific RFC, and responsible for ensuring the process is followed in a timely fashion. The Shepherd Leader has no special responsibility with regard to moving an undecided Shepherd Team to a certain decision.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/shepherds-wanted-for-rfc-54-support-levels-and-tiers/4406/1

[shlevy]

@ryantm @zimbatm @grahamc @Ericson2314 Do you want to carry over to this one? Also @ehmry did I understand correctly you put yourself forward for nomination?

[ehmry]

@shlevy yes, I volunteer for #54.

[Ericson2314]

@shlevy I could, but first me prove myself a non-shit shepherd leader by finally scheduling a meeting for #46.

[ryantm]

@shlevy yes, I'm okay with doing this one too.

[globin]

The @NixOS/rfc-steering-committee has decided on @ehmry, @Ericson2314 and @ryantm as shepherds, with @ryantm as leader.

[ehmry]

This is a minor nitpick, but I think the phrase "OS ABI" is more appropriate than kernel. For example, FreeBSD and Fuchsia can emulate the Linux ABI, CloudABI binaries are basically kernel-agnostic.

[CrystalGamma]

I apologize for being a terrible author by disappearing without a word for 2½ months :(

Anyway, @alyssais : assuming you are aiming for system level support, you'd do the following:

• Add illumos to lib/systems, add whatever toolchain support a cross-stdenv targeting illumos needs and make a PR. Unless you are doing major things wrong (i. e. your changes aren't 'technically sound' as I called it in the RFC), this should be accepted as part of the 'accept-core-by-default' policy proposed by the RFC.
• Nominate the illumos-based platforms for core support, naming yourself (and possibly others) as maintainers.
• You will now be expected to check in somewhere to report on the support state of your platforms when the next release comes around.
• Assuming you are responsive enough and you can report that core functionality isn't broken or is at least fixed quickly, in the next release illumos will be considered core-supported and bootstrap-scoped changes will be accepted.
• You can now start upstreaming changes needed to create bootstrap tools for illumos within Nix and bootstrap a stdenv with them.
• Automated cross-builds of the bootstrap files via Hydra should be installed at some point in this phase, and, if possible, some installation (e. g. a VM) that can check that bootstrapping works on the platform.
• When you feel confident enough, you can apply for bootstrap support. You'll need to have found at least one other maintainer for the platform before this, as the PMG for bootstrap-supported platforms are required have at least two active members.
• Again, you (and the other maintainer(s)) are expected to report in on the state of the platform for the next release.
• Assuming no major breakage, illumos can be classified as a bootstrap-supported platform for the next release, and acceptance tier may be raised to system.
• Upstream system-scoped changes (depending on your goal this may be just a collection of relevant programs and libraries, or even a full port of NixOS to the illumos kernel – though the latter might be difficult considering the current reliance of NixOS on systemd)
• When you feel ready, you can apply for system support. By this point you should have:
  • at least 5 active PMG members
  • Hydra building packages on the supported illumos platforms
• As above, when the PMG reports that system support is stable for some release, that support tier can be awarded for that release.

It seems that during my prolonged absence, #46 has gained traction again. I'm personally not invested in which one of these wins (actually, the arguments for #46's less bureaucratic process of tier advancement is quite convincing), so if my work has inspired improvements in that RFC that helped it become ready, then my primary purpose for making this submission (providing clarity about how non-mainstream platforms are handled) is fulfilled.

[ryantm]

@CrystalGamma I'm also sorry that I haven't done my duty as shepherd leader in pushing this along. It wasn't until I voted to move #46 to FCP that I realized I was the leader on this one. I think your summary of the state of #46 is accurate, and I want to encourage you to give your comments and improvement suggestions to #46, since we all would appreciate your feedback.

[7c6f434c]

#54 has definitely helped to regain focus in the discussions of #46, in particular by highlighting the range of natural options in some places, and thank you for that.

I think that even if the process won't use it for limiting the packages one can touch, the definition of package scopes and reporting in terms of «90% of LiveCD packages build» is useful to have (and feels like the core part of #54, and seems pretty complete). So even if you do not want to build an argument for your versions of the conflicting normative parts, the nonconflicting ones are valuable and hopefully uncontroversial.

[edolstra]

Now that #46 has been merged, should we keep this RFC open? Are there parts that are not addressed by #46?

[7c6f434c]

Now that #46 has been merged, should we keep this RFC open? Are there parts that are not addressed by #46?

The most detailed part here — classifying the packages — could be useful even without the normative part (also maybe as a part of guidelines about the more vague «most of the most popular» in #46) Not sure if there is anyone willing to work on that restructuring in the short term, though.

[ryantm]

@edolstra Yeah, I think we can close this without any formality. If @CrystalGamma requests reopening, of course we can do that no problem.

[7c6f434c]

@edolstra Yeah, I think we can close this without any formality. If @CrystalGamma requests reopening, of course we can do that no problem.

Indeed, we even have this quote from the author:

…

if my work has inspired improvements in RFC #46 that helped it become ready, then my primary purpose for making this submission (providing clarity about how non-mainstream platforms are handled) is fulfilled.