Configuration for common programs

[energizah]

I haven't thought this through, but what about supplementing the files= and directories= with programs= or services= which would know that

• bluetooth=true means persist /var/lib/bluetooth and
• networkManager=true means persist /var/lib/NetworkManager/system-connections

and so on?

This has the advantage that "what files does this program need to persist" can be figured out by the community once instead of by each user individually.

[talyz]

Sorry for not replying earlier to this. It sounds like a great idea! I think we could even go further with it and automatically figure out what to persist based on what NixOS options are enabled, or at least default our options to the relevant NixOS option value.

[lovesegfault]

One idea on how to make this work, at least for the NixOS module, would be to design it to look like

{
  environment.persistence = {
    stateDirectory = "/example/state";
    directories = [ "/etc/example"  "/var/lib/example" ];
  };
}

Then we could have a number of ++ lib.optionals config.hardware.bluetooth.enable [ "/var/lib/bluetooth" ] etc. Maybe even behind a enableDefaultDirectories = lib.mkDefault true;

The main loss here is the ability to have multiple state directories.

Another alternative is letting users set a rootState = lib.mkDefault false; attr to their persistence config, and we then assert only one config has that set, and to that one we ++ all the per-service/per-program state dirs.

What do you think @talyz?

[talyz]

I'm leaning toward the latter alternative, but was thinking we could let the user decide this on a submodule / persistent path level, so an example config would look like

{
  environment.persistence."/persistent" = {
    presets =  {
      enable = true;
      bluetooth = false;
    };
    directories = [
      "/var/lib/additional_directory/bluetooth"
    ];
    files = [
      "/etc/my_important_file"
    ];
  };
}

The bluetooth options default value would be set to config.hardware.bluetooth.enable and its value would decide whether we add the /var/lib/bluetooth directory to the directories list.

The duplicate check should really be done by comparing all the directory and file lists across all persistent paths, making sure all items in all lists are unique. This would catch all duplicates, whether they're introduced by presets, the user or both.

[KFearsoff]

I am interested in doing that and am willing to start the process. That said, I am pretty new to the Nix ecosystem, so I am at a bit of a loss where to start. Any pointers towards the first step?

[Majiir]

Candidates for special configuration

Machine ID

This is simply defined with environment.persistence.<dir>.files = [ "/etc/machine-id" ]. Since it's such a commonly recommended configuration, it might be worth a config flag like environment.persistence.<dir>.machineId = true.

Clock file

On systems without an RTC (e.g. a Raspberry Pi), the clock file can be crucial for startup. For example, bind will fail to validate DNSSEC keys if the clock is wrong, and the service will fail to start correctly. Like machine-id, this configuration is simple (environment.persistence.<dir>.files = [ "/var/lib/systemd/timesync/clock" ]) but it could be nice to wrap it in a simple config like environment.persistence.<dir>.clock = true. Other time services like chrony also rely on persistent clock files, so this one parameter could inspect the config to see which are enabled and persist whichever clock files are needed.

SSH host keys

This snippet persists the host keys for services.openssh, which populates the hostKeys parameter even when the defaults are used:

{
  environment.persistence."<dir>".files =
    lib.concatMap (key: [ key.path (key.path + ".pub") ]) config.services.openssh.hostKeys;
}

This is nontrivial and a good candidate for defining once. A config setting like environment.persistence.<dir>.sshHostKeys = true could inspect config to see which SSH services are enabled and persist their host keys.

Questions

Aliasing

The clock file and SSH host keys are examples where the same kind of state could be managed by different services. For example, systemd-timesync or chrony could be responsible for a clock file.

1. Should these be exposed through a single config flag (clock) or a separate flag per service?
2. If they are exposed through a single flag, should there be a single persisted path (e.g. <dir>/clock) or a separate path per service?

Defaults

The whole point of impermanence is to manage opt-in state. However, some state is crucial to system operation in ways that users may not anticipate. Arguably, all three of the examples above are state that users may not think to persist, but which will cause serious issues if not persisted. It may be sensible to persist them by default.

1. Should impermanence ever persist state by default?
2. If so, how should it handle the possibility that multiple persistence directories are configured? (For example, if clock is persisted by default, which persistence directory should clock be stored in?)

[KFearsoff]

This is simply defined with environment.persistence.<dir>.files = [ "/etc/machine-id" ]. Since it's such a commonly recommended configuration, it might be worth a config flag like environment.persistence.<dir>.machineId = true.

I don't think bind mounts work in this case, actually: systemd generates new machine-id before the bind mount is finished. Or at least, I had that problem a few months ago. My solution was to use a symlink instead, it works great. I think we should implement #99 for NixOS as well, because there might be other software that conflicats with bind mounts.

This snippet persists the host keys for services.openssh, which populates the hostKeys parameter even when the defaults are used:

{
  environment.persistence."<dir>".files =
    lib.concatMap (key: [ key.path (key.path + ".pub") ]) config.services.openssh.hostKeys;
}

This is nontrivial and a good candidate for defining once. A config setting like environment.persistence.<dir>.sshHostKeys = true could inspect config to see which SSH services are enabled and persist their host keys.

I am not sure that works, too. I think I had problems trying to bind mount the SSH keys. I haven't tried symlinks; my solution was to redefine hostKeys.*.path from /etc/ssh/... to /persist/etc/ssh/.... We can test it once we have something going.

I can't speak about the clock file, I don't have a RPI. But my system that persist /var/lib/systemd via bind mount works fine. But you've missed one more crucial path that needs persisting, that is /var/lib/nixos. That directory keeps track of UIDs and GIDs of services; since NixOS uses dynamic users for systemd services whenever it can, it is important to persist their UIDs and GIDs to not have corrupted state on disk.

1. Should impermanence ever persist state by default?

I think it should not, but due to a slightly different reason than what you might imagine. I think there are definitely reasonable defaults that Impermanence should suggest. I don't think it should ever enable them (unless we are talking strictly about symlinks), since bind mounts are destructive. If you've started persisting /var/lib/test when you hadn't persisted it before - /var/lib/test will be empty, since it will be bind-mounted from an empty directory /persist/var/lib/test. This should be written in caps, bold and red letters all over the documentation, and it is for this reason that I think we shouldn't enable any templates by default.

2. If so, how should it handle the possibility that multiple persistence directories are configured? (For example, if clock is persisted by default, which persistence directory should clock be stored in?)

You brought up a good point, so don't mind if I'll use it! That problem solves itself if we just don't enable any presets by default. But I think it is a good idea to suggest using several persistence directories. Currently, I see the following categories for the data worth persisting:

• system data (such as things discussed above) - should be persisted, shouldn't be backed up
• service data (such as databases) - should be persisted, should be backed up
• user data (such as photos) - should be persisted, should be backed up

I think that it's a good idea to do presets with these categories in mind. It would be also pretty nice if the docs would suggest that layout and show examples with it.

I'm not currently working on the PR that would implement presets API. I tried, but I failed miserably because I'm just not good enough at Nix. But I can share some knowledge and know-how about things that should probably go into design, so if someone makes a PR - please do ping me, I'll try to help however I can!

[KFearsoff]

I've opened a draft PR that aims to implement presets. While it's still a very early stage, feedback is very welcome #108

[CertainLach]

What about instead of doing this just for this project, it can also be done for nixos configuration as a whole? There is already a bootspec project (NixOS RFC 0125: https://github.com/NixOS/rfcs/blob/master/rfcs/0125-bootspec.md), which standartizes nixos way of listing bootloader entries, what about persistencespec?

Then most of the work might be offloaded to nixos module authors themselves, every file to be stored should be described in nixos module itself, i.e postgresql module should define

# Entries like this can even be produced by systemd module itself, when service
# declares StateDirectory (It can even handle DynamicUser with its /var/lib/private state dir)
persistent.postgresql = {
  directories = ["/var/lib/postgresql"];
  meta.kind = lib.persistence.systemdStateDir;
};

Those entries then can be used in manual for nixos itself, for some backup solutions, and of course, for impermanence.

Impermanence then would extend persistent.<name> attrset with its own options

persistent.postgresql.storageClass = "main";
environment.persistence."/persistent".classes = ["main"];