Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using sops with arion-compose #195

Open
nviets opened this issue Jun 9, 2023 · 9 comments
Open

Using sops with arion-compose #195

nviets opened this issue Jun 9, 2023 · 9 comments

Comments

@nviets
Copy link

nviets commented Jun 9, 2023

I am trying to set up a container that needs a secret stored with sops-nix. My .sops.yaml and secrets/ are set up, but I'm not sure how to configure my arion-compose.nix file. I have something like:

{ lib, config, pkgs, ... }: let
  mb_key = ''
    export MY_KEY=${config.sops.secrets."my_key".path}
  '';
in {
  project.name = "alma";
  services = {
    gui.service = {
      hostname = "gui";
      image = "almalinux:9";
      command = [
        "${pkgs.writeScript "entrypoint" "${mb_key}"}"
      ];
    };
};

But it's failing on:

error: attribute 'sops' missing

       at /home/nviets/repos/alma-docker/arion-compose.nix:4:21:

            3|   wb_key = ''
            4|     export MY_KEY=${config.sops.secrets."workbench_key".path}
             |                     ^
            5|   '';

Thanks in advance!

@Grantimatter
Copy link

Grantimatter commented Apr 26, 2024

I'm working on doing basically the same thing for my setup.
From what I've gathered, you can't use the values of the secrets directly in nix configurations in this way, as this will just insert the path to the /run/secrets/ directory of the secret that sops-nix generates.

sops-nix templates seem like they may work for this. I'm hoping to have a working example soon to share.

*Edit: Templates are not the solution here either. This will also just insert the path to a config generated with sops-nix. sops-nix cannot be used to insert secrets directly into nix configurations. See using-secrets-at-evaluation-time in the sops-nix readme. At this point I am thinking about just using sops-nix to create the config, and either use arion to extend the compose file with docker compose secrets, or forego arion and just use a plain docker-compose.yml in my setup that can reference the secrets files that sops-nix outputs.

The only alternative I'm seeing is scalpel which hasn't had a commit in over 2 years, so it's most likely dead.

If there's any other alternatives that anyone knows about, or if you ended up solving this since your post, I'd love to hear your solution!

In case I forget to come back to this, whatever method I end up using will be in my NixOS configuration for my Homelab Setup.

@KiaraGrouwstra
Copy link
Contributor

that docker compose/swarm secret mechanism exposes the secrets as a file to the container. when you only need to access a secret during a docker build step, another approach is to use docker buildx secrets.

(for using sops-nix, also see #247 on passing inputs to arion.)

@thenbe
Copy link

thenbe commented Jul 17, 2024

The workaround shown in #247 is great, it solved my flake issues. The build passes and the container starts.

When it comes to sops-nix, however, that still doesn't quite work. There are a couple of errors when running arion up:

grafana-1  | could not create symlink /etc/hosts at /nix/store/rg5rf512szdxmnj9qal3wfdnpfsx38qi-setup-etc.pl line 133.
grafana-1  | sops-install-secrets: failed to mount filesystem for secrets: cannot mount: operation not permitted
grafana-1  | Activation script snippet 'setupSecrets' failed (1)

After the container starts, I can verify that /run/secrets directory does not exist.

Here are the extended logs:

$ arion up --remove-orphans
trace: warning: system.stateVersion is not set, defaulting to 24.11. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
trace: Obsolete option `boot.tmpOnTmpfs' is used. It was renamed to `boot.tmp.useTmpfs'.
/nix/store/a2q1w13d55asknig0gfb3rn3vqhb7rxb-docker-compose.yaml
WARN[0000] /home/nbe/projects/sail/.tmp-arion-docker-compose1910732-0.yaml: `version` is obsolete
[+] Running 2/0
 ✔ Container full-nixos-loki-1     Removed                                                                                                                                                                                                                                                                                                                                                                                                                                           0.0s
 ✔ Container full-nixos-grafana-1  Created                                                                                                                                                                                                                                                                                                                                                                                                                                           0.0s
Attaching to grafana-1
grafana-1  |
grafana-1  | <<< NixOS Stage 2 >>>
grafana-1  |
grafana-1  | booting system configuration /nix/store/rhyjydwq95ca9qy7v311n037ll97shh7-nixos-system-unnamed-24.11pre-git
grafana-1  | running activation script...
grafana-1  | setting up /etc...
grafana-1  | could not create symlink /etc/hosts at /nix/store/rg5rf512szdxmnj9qal3wfdnpfsx38qi-setup-etc.pl line 133.
grafana-1  | setting up secrets...
grafana-1  | /nix/store/52a2jyq0y5v93g6276cq5j0jqnid404r-sops-install-secrets-0.0.1/bin/sops-install-secrets: failed to mount filesystem for secrets: cannot mount: operation not permitted
grafana-1  | Activation script snippet 'setupSecrets' failed (1)
grafana-1  | Setting up sops templates...
grafana-1  | Traceback (most recent call last):
grafana-1  |   File "/nix/store/wgzaf45f12da7pq0jri545i9yn248a8i-substitute", line 27, in <module>
grafana-1  |     main()
grafana-1  |   File "/nix/store/wgzaf45f12da7pq0jri545i9yn248a8i-substitute", line 24, in main
grafana-1  |     print(substitute(target, subst))
grafana-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
grafana-1  |   File "/nix/store/wgzaf45f12da7pq0jri545i9yn248a8i-substitute", line 15, in substitute
grafana-1  |     with open(path) as f:
grafana-1  |          ^^^^^^^^^^
grafana-1  | FileNotFoundError: [Errno 2] No such file or directory: '/run/secrets/DATABASE_PASSWORD'
grafana-1  | Activation script snippet 'renderSecrets' failed (1)
grafana-1  | starting systemd...
grafana-1  | systemd 255.6 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)
grafana-1  | Detected virtualization docker.
grafana-1  | Detected architecture x86-64.
grafana-1  |
grafana-1  | Welcome to NixOS 24.11 (Vicuna)!
grafana-1  |
grafana-1  | Queued start job for default target Multi-User System.
grafana-1  | [  OK  ] Created slice Slice /system/modprobe.
grafana-1  | [  OK  ] Started Dispatch Password Requests to Console Directory Watch.
grafana-1  | [  OK  ] Started Forward Password Requests to Wall Directory Watch.
grafana-1  | [  OK  ] Reached target Local Encrypted Volumes.
grafana-1  | [  OK  ] Reached target Login Prompts.
grafana-1  | [  OK  ] Reached target Containers.
grafana-1  | [  OK  ] Reached target Path Units.

@KiaraGrouwstra
Copy link
Contributor

@thenbe what about if you give your arion service service.privileged = true;?

@thenbe
Copy link

thenbe commented Jul 31, 2024

@KiaraGrouwstra Yup that works. Thanks for the hint!

{
  service.privileged = true;
  service.volumes = [ "${toString ./.}/keys.txt:/var/lib/secrets/age" ];
}

It also works if we only grant it the SYS_ADMIN capability, which "grants a smaller subset of capabilities to the container, compared to the --privileged switch" (source). See https://docs.docker.com/reference/cli/docker/container/run/#privileged

-service.privileged = true;
+service.capabilities = { SYS_ADMIN = true; };

@thenbe
Copy link

thenbe commented Jul 31, 2024

I'm still not sure exactly why the container needs to be privileged, the best I have so far is this similar discussion in the nix repo: NixOS/nix#3059


After searching a bit more, I found another solution that sidesteps the privilege requirement altogether.

{
  sops.useTmpfs = true;
}

See the issue and PR for more info. This solution also comes with a caveat.

@KiaraGrouwstra
Copy link
Contributor

@thenbe nice! would you maybe have an example of how to expose the secret thru arion?

@thenbe
Copy link

thenbe commented Aug 2, 2024

Something like this:

{
  project.name = "nixos container";
  services.webserver = { pkgs, lib, ... }: {
    service.volumes = [ "${toString ./.}/keys.txt:/var/lib/secrets/age" ];
    nixos.useSystemd = true;
    service.useHostStore = true;
    nixos.configuration = { config, lib, options, pkgs, ... }: {
      boot.tmp.useTmpfs = true;

      # 1. setup sops-nix
      imports = [
        inputs.sops-nix.nixosModules.sops
      ];
      sops.defaultSopsFile = ../secrets.yaml;
      sops.defaultSopsFormat = "yaml";
      sops.age.keyFile = "/var/lib/secrets/age";
      sops.useTmpfs = true;

      # 2. use sops-nix
      sops.secrets.MYSECRET = { };
      environment.variables = {
        MYSECRET = config.sops.secrets.MYSECRET.path;
      };

    };
  };
}

@KiaraGrouwstra
Copy link
Contributor

KiaraGrouwstra commented Aug 2, 2024

thanks!
i'm also looking into docker-compose secrets as mentioned by @Grantimatter, tho it looks like the relevants attributes aren't whitelisted in arion yet. (edit: see #52)
the above example may well already address the scope of the OP's question tho.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants