-
-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Running NixOS inside Docker #2878
Comments
Create nixos rootfs using https://nixos.org/wiki/NixOS_and_libvirt and then On Mon, Jun 9, 2014 at 8:53 PM, Domen Kožar [email protected]
-----BEGIN PGP PUBLIC KEY BLOCK----- mQENBFEY1PEBCADPOfERF2wo4qeoq9L1m2z4pKfWqNd4B6BsoFUWPNd7BXmY+9JG |
Thanks for making this, @iElectric. I think Docker is very different from other container projects, such as libvirt, because Docker is super-easy to get started with. It has images in a central repo for easy download, like VagrantCloud, and it's got projects, both big and small, which provide nice GUIs. |
So patch for running systemd has been applied to docker master(libcontainer), and you can test it by updating and patching docker #3015. Now when it looks more promising, i will try to get nixos running inside docker. |
This is awesome! I suppose it still needs a privileged container? Where is this patch discussed? |
Awesome! They look innocent enough. Why doesn't shell work? Not that it's On Fri, Jun 27, 2014 at 11:28 AM, Jaka Hudoklin [email protected]
|
Well getty does not seem to work, you can spawn init script in the On Fri, Jun 27, 2014 at 12:17 PM, wmertens [email protected] wrote:
-----BEGIN PGP PUBLIC KEY BLOCK----- mQENBFEY1PEBCADPOfERF2wo4qeoq9L1m2z4pKfWqNd4B6BsoFUWPNd7BXmY+9JG |
Currently this #3105 or moby/moby#6730 issues are blocking nixos running on docker. If one of this is resolved, we are ready :) |
This is test case:
To run:
|
You can beta test here: https://registry.hub.docker.com/u/offlinehacker/nixos/ |
@offlinehacker this works now right? Closing. |
Yeah, but this is not intended usage of docker. Docker is basically process
|
@offlinehacker how did you build https://registry.hub.docker.com/u/offlinehacker/nixos/ ? Can you publish an updated version ? you can also use the following Dockerfile: FROM offlinehacker/nixos
COPY *.nix /etc/nixos/
CMD "nixos-rebuild switch --upgrade" and config { config, pkgs, ... }:
{
imports = [ <nixos/modules/virtualisation/docker-image.nix> ];
environment.systemPackages = with pkgs; [
tree
];
} to customize the container |
@offlinehacker Is it possible to add an updated version? Or give instructions on how to build our own image? |
@saulshanabrook see #4642 for instructions and followup. |
@573 I don't understand what the final outcome is, is there now a docker image that contains a recent version of NixOS? |
Is it possible that NixOS running inside docker container? And what is the "closing" meaning? @domenkozar |
I made an attempt at this with all steps used to reproduce, see https://hub.docker.com/r/bbarker/nixos/ and associated urls (including git repo). Some work is needed to get systemd fully working, but it is partly working. Need the help of some NixOS/systemd experts! Maybe open a new issue? |
There is no official image to run NixOS inside docker, but some people have reported to have success doing so. I'll reopen to track any effort. |
@nh2 as far as I know there is none. |
Note: Not exactly for people interested in running NixOS (systemd etc. etc.) in a docker container but as a reference for using nix-related technology from inside a docker container, sorry for cross-posting. I slightly reenacted my efforts and found @LnL7
|
@offlinehacker would like to use custom ✘ ~/project nix ●✚ docker pull offlinehacker/nixos Using default tag: latest
latest: Pulling from offlinehacker/nixos
118f3436959c: Already exists
Digest: sha256:0a2326bcc26551cd0f1380217a3715011998a8d7559172ee8e210feba43bc4e8
Status: Image is up to date for offlinehacker/nixos:latest
✘ ~/project nix ●✚ docker run -t -i --cap-add=SYS_ADMIN --cap-add=SYS_RESOURCE -e "container=docker" offlinehacker/nixos /bin/init
/nix/store/1vahrabkmxxhh1pq8l6m1ilvs33zp29b-docker-17.09.1-ce/libexec/docker/docker: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "exec: \"/bin/init\": stat /bin/init: no such file or directory". P.S. where is the source code of this image, would like to know how this was created |
Just to show the current status: { system ? builtins.currentSystem
, pkgs ? import <nixpkgs> { inherit system; }
}:
with pkgs;
with pkgs.lib;
let
container = (import <nixpkgs/nixos/lib/eval-config.nix> {
modules = [{
nixpkgs.system = system;
boot.isContainer = true;
networking.firewall.enable = false;
}];
}).config.system.build.toplevel;
in dockerTools.buildImage {
name = "nixos";
config = {
Cmd = [ "${container}/init" ];
};
} Build it, load it, then run: $ docker run nixos:xj0lfkd8hqkwgxlmgp9nifjja13chf7a
<<< NixOS Stage 2 >>>
mount: /: permission denied.
mount: /nix/store: permission denied.
mount: /nix/store: permission denied.
running activation script...
setting up /etc...
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 120.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 120.
hostname: you don't have permission to set the host name
Activation script snippet 'hostname' failed (1)
mount: /dev: permission denied.
mount: /dev/pts: permission denied.
mount: /dev/shm: permission denied.
mount: /proc: permission denied.
mount: /run: permission denied.
mount: /run/keys: permission denied.
mount: /run/wrappers: permission denied.
Activation script snippet 'specialfs' failed (32)
starting systemd... |
@FRidh maybe this example from arion is helpful Though not sure how to make all systemd units to run |
We are using https://github.com/hercules-ci/arion for local development and reusing NixOS services. It's not trivial to wire them up, but it has gotten better in last few months :) |
Robert has built Arion, docker-compose integration with Nix that can run in containers:
|
Idk it seems that people do it in their own ways and it seems solved to me, at least in a sense what you can force docker and nixos to do. I decided to try to find better alternatives to docker, from podman, kata containers, firecracker, cri-o,... and better alternatives to abstract services. |
Considering that the issue mentioned (#1088) mentions, that NixOS should support Docker, the link I have given provides a answer that we do, plus we have this in unstable ( https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/docker-containers.nix ), that allows us to declare containers inside our configuration file. :) If DevOps need to support anything, then the issue should maybe be opened in DevOps repo :) |
@eyJhb the link you gave makes a docker image with Nix, not NixOS. There's no systemd running, no way to declare which services run etc. |
Running NixOS inside docker requires running systemd, which doesn't work in there. There's multiple solutions available these days to
Other proposed things above (firecracker and whatnot) probably should go into a separate issue, but I don't see anything actionable here. I propose closing this issue. |
systemd runs inside docker, you just need to mount cgroups. It even runs with user namespaces, using cgroup hierarchy v2, unfortunately currently only using podman. |
Well, in that case, there should be an issue about building an OCI image containing a whole NixOS system, and some documentation/tooling (?) on how to mount the cgroups. |
I think that issue ... is this issue 😁 |
This comment has been minimized.
This comment has been minimized.
@volth Why was this closed? From what i understood from the last few posts, it should be possible to make this work. |
I think we could have a function that generates a docker image with NixOS configured, taking inspirations from arion. |
@offlinehacker Could you tell us how you got NixOS running inside podman? I tried #2878 (comment) with podman instead of Docker but it just gives the exact same error messages. |
I only managed it on fedora 32, as I already knew you can run systemd podman containers out of the box on it. I need to figure out what nixos has different configured but I remembered something with cgroups and user slices. I tried both cgroups hierarchy v1 and v2, it will not work with v1 hierarchy at all. |
I marked this as stale due to inactivity. → More info |
Up! |
I think the issue here is that activate fails and that means no systemd in the path so
|
I marked this as stale due to inactivity. → More info |
This is still important. |
I followed the whole ticket and I see that the nixos docker image from @offlinehacker is not there anymore. And also one I can see which needs a deeper look into it. For now, is there any solution for this? Yes I know, that docker was not made for whole OS but when I want to use a docker image, I would like to use NixOS instead of alpine or anything else. |
@Chris2011 if you don’t need the entire NixOS (with systemd etc), but just nix (and any packages built with nix) you can check this out: https://github.com/nix-community/docker-nixpkgs |
@adrian-gierakowski thx for the link, will have a look and it seems promising for me :) |
#175474 looks slightly related |
Follow up from #1088. I'd like to run NixOS inside a Docker container. The motivation is to provision Docker images with NixOS as devops are used to Docker API.
Solutions:
The text was updated successfully, but these errors were encountered: