emacs: Add nativeComp packaging infrastructure

[tadfisher]

Motivation for this change

Ref: nix-community/emacs-overlay#74

This is the start of some infrastructure to package native-compilation outputs for Emacs with nativeComp enabled.

I've tried to mimic the existing site-lisp pattern as much as possible:

• Packages build to $out/share/emacs/native-lisp.
• Populate comp-eln-load-path from $profileDir/share/emacs/native-lisp in site-start.el.
• Add an EMACSNATIVELOADPATH env variable to mimic EMACSLOADPATH. As upstream does not support a variable like this, add some code to site-start.el to populate comp-eln-load-path from it.
• Add support to emacsWithPackages to link native-lisp along with site-lisp for included elisp packages.

This is a draft because:

• The packaged ELN files are not loaded by comp because they are hashed by both their content and their absolute file path. There is some code in comp.c that ignores the path part of the hash if it's located in lispdirrel, so perhaps there's a way to rename these files correctly to match what comp expects for a "system" load path.
  • Update: Commit emacs-mirror/emacs@3ae309b fixes this by following symlinks before computing the hash, so .eln files are loaded from the Nix store now when using emacsWithPackages.
• Upstream should probably support EMACSNATIVELOADPATH so we don't have to have the workaround in site-start.el. I've filed #44726 to request this.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[bqv]

emacs#44726 is done, it seems

[adisbladis]

@tadfisher I've implemented the changes related to emacs#44726 and rebased this PR on @alyssais's latest correctness fixes re EMACSLOADPATH. I also implemented the same thing for EMACSNATIVELOADPATH.

[adisbladis]

A protip for future readers:
If you want to use Nix as your only emacs package manager I recommend setting (setq comp-deferred-compilation nil) to avoid native compilation kicking in at runtime.

[bqv]

Would either of you be interested in implementing similar for the home-manager emacs infrastructure?

No problem if not, just that I've just realised this doesn't apply to me because I use that instead.

[tadfisher]

@bqv home-manager doesn't have emacs packaging infrastructure. emacsWithPackages from nixpkgs is what actually builds the wrapped Emacs with dependencies.

[bqv]

Whoops, I guess I misundertood what was happening here. It doesn't apply to me because I don't use emacsWithPackages, not otherwise. Sorry!

[alyssais]

A protip for future readers: If you want to use Nix as your only emacs package manager I recommend setting `(setq comp-deferred-compilation nil)` to avoid native compilation kicking in at runtime.

Should we make emacsWithPackages set this by default? I'd assume that most people using it for one package use it for all of their packages, or at least almost all.

[collares]

A protip for future readers:
If you want to use Nix as your only emacs package manager I recommend setting (setq comp-deferred-compilation nil) to avoid native compilation kicking in at runtime.

There's a caveat to this which might be worth mentioning. The files you get in ~/.emacs.d/eln-cache if you don't do this really weren't native-compiled by Nix, so you will not get speedups for such files if you disable deferred compilation (which is a valid choice if you don't want to deal with ~/.emacs.d/eln-cache clutter). The most common cause for this (after #107777 is merged, at least) is if a dependency is missing from the Lisp package's Nix derivation, or if a native package such as mu does not native-compile its elisp files. Such cases are rare, and in my case (which includes mu4e) deferred compilation at first startup takes less than 10 seconds.