nixos: convert option documentation to Markdown

[emilazy]

Description

NixOS 23.11 has migrated fully to Markdown and no longer accepts DocBook documentation.

I was going to do this migration for the whole Home Manager tree per #4142 (comment) but am holding off for now in light of LnL7/nix-darwin#707 (comment). Still happy to do it if it's desired, though; I think it makes sense to follow nixpkgs' choices and tooling given the inherent tight coupling, unavoidability of processing at least some Markdown (for module system options), and the code sharing benefits (both in terms of the documentation generation itself and adapting modules between the two).

Checklist

• Change is backwards compatible.

• Code formatted with ./format.

• Code tested through nix-shell --pure tests -A run.all or nix develop --ignore-environment .#all using Flakes.

• Test cases updated/added. See example.

• Commit messages are formatted like

  {component}: {description}
  
  {long description}
  

  See CONTRIBUTING for more information and recent commit messages for examples.

• If this PR adds a new module

  • Added myself as module maintainer. See example.

[emilazy]

cc @rycee @ncfavier

[ncfavier]

Actually this won't work properly. mdDoc is a no-op, so NMD doesn't know to render these descriptions as Markdown. If we restore the old behaviour of mdDoc, then mkEnableOption (mdDoc ...) will fail since that's not supported in nixpkgs anymore.

We should use nixos-render-docs at least to render the NixOS module docs (and eventually, HM options too).

[emilazy]

These options are rendered in the NixOS manual, not the Home Manager one, no?

[emilazy]

Ah, I see, they're in both. That seems like another reason maintaining two separate documentation generators/formats is untenable. There's already options in-tree that use mdDoc so presumably those are broken right now too.

[ncfavier]

These options are rendered in the NixOS manual, not the Home Manager one, no?

Nope

[emilazy]

Right, I saw that after replying :)

Well, I'm not sure what to do here. nixos-render-docs does still interpret { _type = "mdDoc"; } for now, but who knows how long that will last. I could inline that in the PR and it'd presumably get it working with both generators in the short term. But really it seems like there'll only be more problems like this over time as NixOS rips out more of the legacy stuff and I'm not sure how the circle could be squared over the long run.

Standing offer on my part to do the work to move things over if there's agreement that that would be desirable, as I still have it all in working memory and know how to hack up nix-doc-munge now, but of course it's still a lot of work to do if it's not going to be merged.

[ncfavier]

I'm 100% behind that, but of course it's up to @rycee.

[rycee]

Honestly I'm feeling quite skeptical about depending on Nixpkgs for this type of functionality. At one point Home Manager used the old documentation framework that was in Nixpkgs, and for the most part it worked OK. The thing that eventually drove me to excising the code from Nixpkgs into nmd was the frequent breaking of the API, which in turn would break Home Manager builds, which in turn would force me to spend nights fixing things.

And as far as I can tell the new documentation system is intended for NixOS/Nixpkgs documentation, not one for general consumption. The prime evidence being that the entire thing is completely unversioned and hosted inside Nixpkgs. In my view, depending on it would just move us back into the unfortunate dependency situation that I originally tried to avoid with nmd.

I would feel more relaxed if the new documentation system was a proper project separate from Nixpkgs with the expressed purpose of being a general purpose documentation system for Nix projects. Then we would in HM be able to pin a specific version and have the possibility to do updates at a somewhat controlled pace, rather then doing things in a panic whenever things hit Nixpkgs unstable.

So my proposal for now is to just re-introduce a proper mdDoc function (why it was broken in Nixpkgs, I don't know). Migrate the descriptions over to use mdDoc, either in a single batch or gradually, just like you've done in this PR @emilazy. Once all are migrated we can make nmd issue a warning for un-annotated descriptions. Finally, after some time we could make CommonMark the default format.

[pennae]

@rycee we do intend to mark a stable api for the rendering infrastructure once the markdown transition is fully completed and remaining kinks are ironed out. until then some amount of breakage is just unavoidable given the nature and scope of the changes that are going on. using an old snapshot of the docs system will not solve any of these problems. we don't want to break anything, and if something does break we can find ways to fix it or at least make some more time to fix it (like _module.args was hidden in the last unstable cycle). at some point we do have to rip off these band-aids though.

and we do agree about extracting these things into another project. the module system and the docs system could both be extracted and versioned. for now that's not really possible because things are very much in flux and dependency management mechanisms like flakes are hard to get into nixpkgs. until then we do try to make changes in such a way that they don't break well-behaving users' uses, and nmd completely sidesteps that (being an old snapshot of an incompletely migrated system). we'd rather work together with other users of these systems than have them fragment the space by vendoring old code and inevitably having it break on them much later.

[emilazy]

Adding on my own thoughts:

When doing the migration for nix-darwin and talking to @pennae I haven't had the impression that it's intended as an "internal NixOS API, outsiders don't touch" at all; they've expressed enthusiasm about reducing incompatibility and fracturing of the ecosystem by getting everyone on one option documentation processor, and openness to adding necessary customization points and establishing stability going forward. Of course, things have been in flux with a major migration going on, but the process has been ongoing for a while now and the breakage has been gradual and well-flagged; nmd has just resulted in missing all of the warnings and accommodation that otherwise would have happened. From the upstream perspective, the big migration was done gradually and is now complete; it's just that using a fork meant that HM didn't notice until the very last moment.

I understand the desire to avoid dealing with upstream changes by using a fork of the documentation generator but I'm not sure it plays out that way in practice. There was the recent #4142 and this module documentation issue, and looking at the nmd git log I think there have been other cases too? It seems like in the long run nmd would have to duplicate most of the complexity of nixos-render-doc to handle the NixOS Markdown documentation format. Would there have been other breakage if nmd wasn't being used during this time? Probably, but I think it would have been easier to manage, and will especially be easier to deal with going forwards now that the big migration is complete. Fundamentally, when you're tracking a rolling-release branch targeting the next release, backwards-incompatible changes are going to happen; even if nixpkgs-unstable stayed green for every commit, they can't be expected to maintain complete compatibility with a modified fork of an old version of part of the system. They can commit to deprecation warnings and timing breaking changes across their own release cycles if we work with them, though.

In this case, if Home Manager had migrated to the new NixOS documentation system earlier, then there would have been warnings about options with DocBook documentation for months now, and if they were dealt with in a timely manner the latest change would neither have been unexpected nor have broken anything. Given that HM is already fundamentally coupled to nixpkgs in terms of the package set, module/options system, etc., that, to me, seems like a better stability guarantee than hoping nothing upstream will ever break anything with nmd; the only way to truly avoid keeping up with upstream changes would be to fork nixpkgs entirely.

There are already issues the current setup is running into that wouldn't be happening with the setup nix-darwin has. Right now I can't generate HM documentation on Darwin at all because of the nested Nix build (waiting on NixOS/nix#8563 to cut a release with my backport of the fix for NixOS/nix#8485 to solve this). Of course, that's a Nix bug, but it's one that wouldn't have been surfaced using nixosOptionsDoc (unless, admittedly, explicitly opting into the new lazy/eager options split designed for NixOS's tens of thousands of options). And in this PR, we're already in a position where we have to make the NixOS/nix-darwin module documentation a polyglot that both nmd and the upstream options documentation generator will accept. That may simply not be possible for long, and I think that mixing two versions of the documentation generator for fundamentally coupled projects will lead to more issues going forward, both in terms of keeping up with upstream and allowing free exchange of code between HM and NixOS/nixpkgs. Even if there is a strong desire to keep the overall manual generation separate, I think moving to nixosOptionsDoc for processing and rendering the options documentation is really important.

[emilazy]

(All that being said, of course, it's ultimately Home Manager's decision to make as to how it handles documentation; these are just my thoughts on what would be least painful and what the best way forward for the ecosystem is. For what it's worth, though, I would be happy to be pinged going forward about any problems that come up when using the upstream documentation generator.)

[rycee]

Thanks for the nice feedback! Like I said my main concern is that having the documentation framework in Nixpkgs will lead to unexpected breakage whenever somebody decides to change something to suit Nixpkgs. I've been burned by this many times, hence my cantankerous attitude.

But if a reasonable effort is made on the Nixpkgs side to keep the interface stable then I have no problem using the new documentation framework. I interpret @pennae's comment above that this is the case, and that is good enough for me. So, if @emilazy is still willing to take the driving seat on doing a wholesale migration then I think we should do it as soon as possible.

[emilazy]

It's perfectly understandable; dealing with nixpkgs-caused breakage is always a pain. (Though at least in the Nix world it tends to break at build- rather than run-time!)

The impression I get is that a good amount of the churn in the NixOS documentation system in the past has been due to the fiddly (and seemingly unmaintained upstream?) DocBook pipeline and that now things are factored out into individual tools it'll be easier to keep stable, but of course only time will tell how that plays out in practice. I believe that the main potentially breaking change intended before trying for stability is to renovate the HTML output, as it currently matches the (rather dated and IIRC non-validating) previous DocBook XSL-generated markup as closely as possible; that could theoretically result in breaking third-party styles but looking at nmd's style sheet it's mostly very generic selectors so it should be mild and easy to fix if so.

I'll try to set some time aside to make a migration PR in the next few days. Probably best to do it in one big go, as the automated conversion tooling can handle the large majority of option documentation and we'd have to pin an old nixpkgs to keep using DocBook input anyway.

[emilazy]

Just a quick update: I've been plugging away at this for a few days and it's going well; I've successfully wired up nixosOptionsDoc, adapted nix-doc-munge for the HM codebase, and manually dealt with basically all the options it can't handle itself. Still a good amount of work left, but I'll hopefully be able to send a PR next week.

I'll close this PR as it doesn't work properly as-is and will be incorporated into the much larger upcoming one.