Introduce libnixflake

[Ericson2314]

Motivation

Flakes is a separate layer, so it should have its own library!

Context

See each commit for details. The most interesting code changes (that aren't just moving things around) happen in the first commit.

Priorities

Add 👍 to pull requests you find important.

[edolstra]

Please keep registry.* and indirect.* in libfetchers, since they're fetchers and not flake-specific (e.g. you can do fetchTree to fetch a repo looked up via the registry). This PR creates a libflake that isn't really about flakes at all but about the indirect fetcher, which IMHO belongs in libfetchers.

I prefer to keep all URL/flakeref-related regexes like flakeIdRegexS in one place (url-parts.hh) rather than scattered all over.

[Ericson2314]

 $ git grep flake src/libfetchers/
src/libfetchers/fetch-settings.hh:          Example `~/code/flake.nix`:
src/libfetchers/fetch-settings.hh:    Setting<std::string> flakeRegistry{this, "https://channels.nixos.org/flake-registry.json", "flake-registry",
src/libfetchers/fetch-settings.hh:          Path or URI of the global flake registry.
src/libfetchers/fetch-settings.hh:          When empty, disables the global flake registry.
src/libfetchers/fetch-settings.hh:        "Whether to use flake registries to resolve flake references.",
src/libfetchers/fetch-settings.hh:    Setting<bool> acceptFlakeConfig{this, false, "accept-flake-config",
src/libfetchers/fetch-settings.hh:        "Whether to accept nix configuration from a flake without prompting.",
src/libfetchers/fetch-settings.hh:          The commit summary to use when committing changed flake lock files. If
src/libfetchers/fetchers.cc:            // usages (e.g. `builtins.fetchTree` calls or flake inputs).
src/libfetchers/indirect.cc:std::regex flakeRegex("[a-zA-Z][a-zA-Z0-9_-]*", std::regex::ECMAScript);
src/libfetchers/indirect.cc:        if (url.scheme != "flake") return {};
src/libfetchers/indirect.cc:                throw BadURL("in flake URL '%s', '%s' is not a commit hash or branch/tag name", url.url, path[1]);
src/libfetchers/indirect.cc:                throw BadURL("in flake URL '%s', '%s' is not a branch/tag name", url.url, path[1]);
src/libfetchers/indirect.cc:                throw BadURL("in flake URL '%s', '%s' is not a commit hash", url.url, path[2]);
src/libfetchers/indirect.cc:        if (!std::regex_match(id, flakeRegex))
src/libfetchers/indirect.cc:            throw BadURL("'%s' is not a valid flake ID", id);
src/libfetchers/indirect.cc:        if (!std::regex_match(id, flakeRegex))
src/libfetchers/indirect.cc:            throw BadURL("'%s' is not a valid flake ID", id);
src/libfetchers/indirect.cc:        url.scheme = "flake";
src/libfetchers/registry.cc:            for (auto & i : json["flakes"]) {
src/libfetchers/registry.cc:            throw Error("flake registry '%s' has unsupported version %d", path, version);
src/libfetchers/registry.cc:        warn("cannot parse flake registry '%s': %s", path, e.what());
src/libfetchers/registry.cc:        warn("cannot read flake registry '%s': %s", path, e.what());
src/libfetchers/registry.cc:    json["flakes"] = std::move(arr);
src/libfetchers/registry.cc:        auto path = fetchSettings.flakeRegistry.get();
src/libfetchers/registry.cc:            auto storePath = downloadFile(store, path, "flake-registry.json", false).storePath;
src/libfetchers/registry.cc:                store2->addPermRoot(storePath, getCacheDir() + "/nix/flake-registry.json");
src/libfetchers/registry.cc:    if (n > 100) throw Error("cycle detected in flake registry for '%s'", input.to_string());
src/libfetchers/registry.cc:        throw Error("cannot find flake '%s' in the flake registries", input.to_string());
src/libfetchers/tarball.cc:            // FIXME: would be nice to support arbitrary flakerefs
src/libfetchers/tarball.cc:            // here, e.g. git flakes.

if we can cut this down I am fine leaving those files in libfetchers, but we need to actually make an attempt to prove to the user that the feature is not intrinsically part of flakes. Right now I would say it is an "easter egg" the that the Flake Registry is not actually about Flakes at all, but "easter egg" simply isn't good enough; it has to be de jure.

For example, if it isn't flakes-specific we shouldn't call it the Flakes Registry at all.

[Ericson2314]

#9085 My crude attempt to make indirect/registry demonstrably Flake-agnostic by renaming things. I will need some help finishing that PR however.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-09-29-nix-team-meeting-minutes-90/33774/1

[roberth]

we shouldn't call it the Flakes Registry

This frees up the name for registries in the traditional sense that contain artifacts of sorts.

Still not happy that we overload the term "registry" so I think it's worth taking the opportunity to consider an entirely different name.

The indirect scheme is closer to something like DNS than to a registry.

How about source index?

EDIT: I'm aware that DNS has a prominent concept of registry, but our domain is software development and packaging, not internet infrastructure, so "registry" is more easily assumed to refer to package or artifact registries. Also DNS forms an abstraction where the registry is largely irrelevant to its users.

[Ericson2314]

@roberth I made #9085 for the approach of renaming it if we will keep it in libfetchers. I think your comment might go better there. I totally agree "registry" is not a good name. (I did fetcher-registry as a simple stupid sed-like de-flaking, but I agree it is not a good name at all.)

(I still think we should have a libnixflake, but it would have different contents given that plan. I just renamed this and marked this draft with the intention of basically rewriting it assuming that goes through. Sorry I didn't do this earlier.)

[Ericson2314]

(The registry stuff is kept in libfetchers now)

[roberth]

Surprisingly short diff. I like it.
Just a few comments, nothing major.

[fricklerhandwerk]

Reminder, this is part of the implementation of RFC 136: https://github.com/NixOS/rfcs/blob/master/rfcs/0136-stabilize-incrementally.md#step-3-attempt-likewise-splitting-a-store-and-language-only-nix

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-06-26-nix-team-meeting-minutes-156/47740/1

[cole-h]

Looks like this accidentally reverted part of #10691 (commit-lock-file-summary is gone). #10988 fixes that.

[Ericson2314]

Sorry! A negligent rebase conflict resolution on my part, no doubt. Thank you @cole-h for fixing my mistake ❤️.

[roberth]

No need to take all the blame; this feature wasn't tested.

[cole-h]

Implication heard loud and clear -- I'll add a test in my PR resurrecting it.

EDIT: test added

[Ericson2314]

Tests are def good, and objectively the right way to prevent this, but I also pride myself on being a slicing-and-dicing-like-a-sushi-chef rebaser :), so I really was dissapointed I slipped up.