Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Everything that changed in my fork of this repo #56

Merged
merged 17 commits into from
Mar 24, 2022
Merged

Conversation

nzbr
Copy link
Member

@nzbr nzbr commented Feb 15, 2022

The changes that I have implemented in my fork of this repo can't be separated easily anymore, so I have put all of them in one PR.

#53, #45 and #39 are integrated in this pull request

  • The configuration has been split into multiple modules, which are exposed through the flake
    • the system configuration accesses the flake using flake-compat, so that it can be built without a flake-enabled nix
  • Optional Docker Desktop support (can be enabled in configuration.nix)
  • nixos.nix has been removed in favor of using flake-compat for nix-build. The CI pipeline has been adjusted accordingly
  • syschdemd.sh loads /etc/set-environment before starting a command. This makes the WSL-Remote in VSCode work if https://github.com/msteen/nixos-vscode-server is used
  • A module has been added to build an installer tarball.
    • This wraps around the system tarball and should be used with wsl --import instead. Importing the system tarball can fail with an "Unspecified Error". One of the causes for this error (at least in my testing) is any hard-link being present in the tarball. The installer mitigates this issue by including only a statically-compiled version of busybox, that is used to unpack the system tarball when starting the system for the first time

Fixes #30
Fixes #32
Fixes #34
Fixes #35
Fixes #50
Fixes #51
Fixes #57

@grantbevis
Copy link

I've had a read through and this looks really good, I'd be interested in adding an option for enabling full docker within NixOS with virtualisation.docker = enable = true; - This then negates the need for Docker for Windows.

I'd be happy to raise a PR to extend on the above to let someone choose, native docker or Docker for Windows.

@nzbr
Copy link
Member Author

nzbr commented Feb 16, 2022

I have actually used native docker in NixOS-WSL for quite a while. That is until the docker daemon did not start anymore (probably after a nixpkgs update). If you find a way to fix it, that'd be great!

@grantbevis
Copy link

grantbevis commented Feb 16, 2022

I have actually used native docker in NixOS-WSL for quite a while. That is until the docker daemon did not start anymore (probably after a nixpkgs update). If you find a way to fix it, that'd be great!

It's because in 21.11 we switched from using iptables to nftables through a compatibility layer. You can set an extraOptions and do --iptables=false but then I found none of the containers have outbound network connectivity. I've got this working by sticking with 21.05 for now with WSL2.

I was going to try and pin docker, systemd & iptables/nftables to 21.05 nixpkgs in my flake.nix but this feels janky and not the solution.

Edit: NixOS/nixpkgs#81172

@nzbr
Copy link
Member Author

nzbr commented Feb 17, 2022

The interesting thing is, that I have multiple hosts (with NixOS on hardware) on 21.11, which run docker just fine. They are using the exact same revision of nixpkgs as my WSL host

@grantbevis
Copy link

The interesting thing is, that I have multiple hosts (with NixOS on hardware) on 21.11, which run docker just fine. They are using the exact same revision of nixpkgs as my WSL host

Same, I’m using 21.11 in production with zero issues with docker. There’s something funky around it running in WSL.

@elebeaup
Copy link

I'm using 21.11 in WSL and with docker but I had to recompile the WSL kernel by enabling all nftables modules

@grantbevis
Copy link

I'm using 21.11 in WSL and with docker but I had to recompile the WSL kernel by enabling all nftables modules

How did you do this @elebeaup - Do you have some example nix for the modules needed?

@elebeaup
Copy link

  1. Clone the WSL kernel source
$ git clone --depth 1 https://github.com/microsoft/WSL2-Linux-Kernel.git
  1. Enable the nftables modules
$ cat <<EOT >> Microsoft/config-wsl
CONFIG_NFT_CT=y
CONFIG_NFT_COUNTER=y
CONFIG_NFT_CONNLIMIT=y
CONFIG_NFT_LOG=y
CONFIG_NFT_LIMIT=y
CONFIG_NFT_MASQ=y
CONFIG_NFT_REDIR=y
CONFIG_NFT_NAT=y
CONFIG_NFT_TUNNEL=y
CONFIG_NFT_OBJREF=y
CONFIG_NFT_QUEUE=y
CONFIG_NFT_QUOTA=y
CONFIG_NFT_REJECT=y
CONFIG_NFT_REJECT_INET=y
CONFIG_NFT_COMPAT=y
CONFIG_NFT_HASH=y
CONFIG_NF_SOCKET_IPV4=y
CONFIG_NF_TPROXY_IPV4=y
CONFIG_NFT_REJECT_IPV4=y
CONFIG_NF_SOCKET_IPV6=y
CONFIG_NF_TPROXY_IPV6=y
CONFIG_NFT_REJECT_IPV6=y

# CONFIG_KCSAN is not set
EOT
  1. Compile the kernel with nftables support
$ make KCONFIG_CONFIG=Microsoft/config-wsl
  1. Copy the custom kernel image (i.e. arch/x86/boot/bzImage) to the windows filesystem

  2. Configure the .wslconfig file as follow:

[wsl2]
kernel=C:\\temp\\myCustomKernel

@TeofilC
Copy link
Contributor

TeofilC commented Feb 25, 2022

In case it's of interest, here's an alternative way to work around the hard linking issue (it worked when I wrote it; I haven't confirmed that it still works): TeofilC@cccb4e9

@nzbr
Copy link
Member Author

nzbr commented Mar 18, 2022

Thank you @TeofilC! I used the workaround to imporove upon the installer. I don't really want to modify the final system's derivations by replacing hardlinks with symlinks there. It should't be a problem for the installer tarball though, because none of those files remain in the installed system anyway

@TeofilC
Copy link
Contributor

TeofilC commented Mar 20, 2022

Nice! I'm glad you found it helpful

@nzbr nzbr merged commit 50d1429 into nix-community:main Mar 24, 2022
This was referenced Mar 24, 2022
@nzbr nzbr added enhancement New feature or request and removed enhancement New feature or request labels Mar 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment