A minimal Nix development environment on WSL

[vereis]

Creating a minimal Nix development environment for WSL

When doing my day-to-day development work, I primarily work within some Unix-like environment and I revel in the command line. I've briefly explained that I also prefer to use Windows on most of my computers to eke out the most of my hardware (as far as GPU-support is concerned); but also because I use them more-so as general purpose computing devices.

Background—my computers and use-cases

I have a few laptops in different form factors to support doing different things such as media consumption (primarily visual novels, ebooks, movies) or actually computing on the go. All of these devices run Windows since that is the operating system with the fewest sacrifices I need to make in order to get the most out of their intended use-case with respect to software availability / drivers / power management.

My main desktop machine is a workstation–gaming machine hybrid with a powerful CPU (for the former) but also a powerful GPU (for the latter). Whilst the gaming situation has improved dramatically with software like Valve's Proton, the majority of the games that lock me into using Windows are unfortunately multiplayer and secured by DRM such as EasyDRM which is not easily supported by such endeavours. I've also tinkered with PCIE passhtrough which worked well enough but was difficult to maintain and required some convoluted workarounds to issues.

To avoid much more repetition; modern Windows is (in my opinion) a far better operating system for development work than MacOS for the simple reason that using WSL provides a much more standardized environment for Unix-like development; with WSL 2 I literally have access to a Linux kernel and any* distribution I want without needing to make too many sacrifices in the OS interoperability front.

Background—Nix and NixOS

The main issue I've encountered in the past year since writing this post about setting up a development environment on Windows is that I've discovered Nix and NixOS which I now run on all my servers and dedicated Linux machines. Over the last year I've written a high-level introduction to NixOS as well as a guide on how I use Nix-isms to enable me to write code, but I've skimmed over one major issue: you cannot install an official NixOS distribution atop WSL!

There is work being done regarding this but it's as of now still unofficially supported and when I tried it out it didn't work as I expected all of the time. I'm eagerly awaiting NixOS officially supporting WSL which seems like it might happen so 🤞.

Until a better, more official solution comes along I've just been running Nix (the package manager and language) atop a standard Ubuntu WSL installation; but we can do better 💪😉.

A minimally bloated base for Nix

My issue with using the standard Ubuntu WSL installation as a base for Nix was simply that it provided a lot of stuff out-of-the-box which was great when I wanted to actually use Ubuntu to do my development; but since switching all of my workflows to use Nix/OS I wanted to be able to simply either install NixOS on bare metal, or install Nix atop some existing distribution and
just be done. Because Ubuntu/Fedora/Debian/openSUSE/etc provide a bunch of base packages, it proves relatively difficult to actually be able to define one's environment almost entirely using home-manager like one might use NixOS since default built-in packages may/may not exist on different distributions or may even be called different things.

As said; I want to be able to grab my Nix configuration from the web and simply do a home-manager install and have that set everything up for me much like doing the same via NixOS and nixos-rebuild switch sans the hardware setup and thus I wanted to port over my Nix configuration to a very commonly used (and thus well supported) yet super lean and minimal Linux distribution: (Alpine)[https://alpinelinux.org/]!

Alpine is a great base distribution for this kind of thing:

1. It's super lean. It contains basically no cruft whatsoever as it was originally designed for use on servers or on embedded devices. This isn't really a huge concern for me in terms of install-size but it means I'm basically working as "close to the metal" as possible with Nix; with the fewest moving parts.
2. This leanness lends itself well to being basically the de-facto base Docker image of choice. While this doesn't really affect us as non-Docker focused users of Alpine, it does mean that it's very well supported in terms of packages we might want to install with apk and also it's very well documented. Docker images using Alpine also serve as de-facto recipes and documentation for how to get things working 😉
3. Alpine provides a busybox environment and a musl libc whose project philosophies are rather interesting to me. This apparently can cause some issues down the line (though point number 2 shows this probably isn't an issue in the long run) but is cool enough that I wanted to give it a shot. This is my first experience using busybox/musl and it works really well and rather transparently!
4. There is an officially supported Alpine WSL package you can install with winget (or the Microsoft Store) 🏆🏆🏆

Enabling WSL and installing Alpine WSL

Since I've already covered a more detailed WSL installation procedure (though it's a little outdated now), I'll simply give a very high level overview instead of dredging through the minute details and tradeoffs.

The first thing we can do is enable WSL. The only pre-requisites to using WSL are that your Windows version is at least 1903 and that you're using a 64-bit build of Windows. Provided this is true, you can enable WSL by executing the following in an elevated powershell prompt (Win+X):

# Enable Hyper-V (for WSL 2) and WSL features for Windows
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

# Use WSL 2 by default over WSL 1
wsl.exe --set-default-version 2

As you can see, we're also going to default to WSL 2. This is because it provides us a real Linux kernel (and thus all the niceties of using Linux on bare metal essentially; including Snapd and Docker support). It's also much faster for workflows which create/manipulate a lot of files (so long as you're doing this in the Linux environment, if your work requires you to work on the host filesystem then WSL 1 might be a better choice).

Once these commands are executed, you're going to need to reboot. After rebooting, you can install Alpine WSL via the Microsoft Store though you can also install this with Window's new package manager if you have that configured (it's still early access).

For maximal user experience, I also suggest installing Windows Terminal which honestly is the only sane choice on Windows and easily rivals terminal emulators I use on Linux such as urxvt or kitty. Windows Terminal, once installed, will allow you to automatically detect and switch profiles between any installed WSL distribution, the standard CMD prompt, Powershell and more.

Once all of this is done, simply run wsl.exe -d Alpine (or use Windows Terminal to launch it) which should cause it to prompt you to create a new user and you can begin hacking around if you want. We want to bootstrap Nix though so there is some more setup that needs to be done.

Bootstrapping Nix

Alpine is minimal and a little quirky to the uninitiated. This means there are a few pre-requisites we need to install before we're even going to be able to install and use Nix.

We need to actually install sudo ourselves since it's not one of the built-in packages and it's a hard dependency of the Nix installation script. Because our created use doesn't have permissions to install new packages via apk (😅) we need to do the following:

# Change user to root and install `sudo`
su -
apk add --no-cache sudo

# Enable group `wheel` to use `sudo` (this is convention)
echo '%wheel ALL=(ALL) ALL' > /etc/sudoers.d/wheel

You also need to take care and add your created user to this wheel group to enable it to be able to run the sudo command. This is a bit weird on Alpine WSL since your initially created user doesn't actually have a password set that can be entered for the sudo command to actually work. Thus, while still logged in as the root we need to do the following:

# Add user to the `wheel` group
adduser <USERNAME> wheel 

# Set password for your user. This is not set by default in Alpine-WSL
passwd <USERNAME>

# Return to your original user shell
exit                                               

Once this is done, confirm that you can now execute sudo as your base user by running something benign like sudo vi. If this works then we can carry onto the next step: installing Nix.

The Nix install script unfortunately has two* more neccessary pre-requisites (three in total if we count curl, two being sudo and xz) so we need to get those installed and out of the way. Execute the following:

sudo apk add --no-cache curl xz

And once this is one, we can simply follow the standard Nix installation instructions. For ease of following this guide I've copied the commands below but make sure you don't just pipe random things into sh without understanding the consequences. If instead of wanting to take my word for what commands to run (which is wise), follow the install procedure on the official site:

# Fetch and execute `nix` install script
curl -L https://nixos.org/nix/install | sh

# The install script asks you to do the following (this might be different based on the OS you use)
echo ". /home/<USERNAME>/.nix-profile/etc/profile.d/nix.sh" >> ~/.profile

From this point on, your Nix on WSL experience is more or less ready to be used. I want to set up a few other utilities which are coupled more-so to my own workflow that I've outlined here that I'll go over briefly but if you have your own configuration then feel free to try porting that over instead.

Installing Home Manager

The main issue I'm trying to solve is that I want a single nix expression I want to evaluate which will install all my applications and development setup similar to how you might do so on NixOS. You can achieve something very similar by using home-manager which also is recommended even if you're using NixOS anyway. This way, I literally can share configuration regardless of whether or not my environment is a true NixOS install versus a Nix/Ubuntu, Nix/Fedora, Nix/Alpine/WSL install which is pretty cool 💪😎 You can simply execute the following to download and configure home-manager:

nix-channel --add https://github.com/rycee/home-manager/archive/master.tar.gz home-manager
nix-channel --update
nix-shell '<home-manager>' -A install

And after grabbing my dotfiles and symlinking them to the appropriate place I can run a home-manager switch and everything is installed automatically and automatically configured.

# Install `git` so that we can fetch dotfiles from the web
nix-env -i git
git clone https://github.com/vereis/nixos

# Remove default created `home.nix` configuration and replace with fetched one
ln -s $(pwd)/<MY_CONFIG>.nix $HOME/.config/nixpkgs/home.nix

# Install everything as specified in config
home-manager switch

Post-install—Setting login shells installed by Nix

Since I don't use ash and don't have any configuration for it (and personally just very much prefer zsh) I have nixpkgs.zsh specified in my home.nix. It takes a bit of hacking to be able to use chsh -s <SHELL> when the shell you've installed is managed via nix however since Alpine's chsh has no idea you've installed a new shell.

All packages installed by nix and home-manager are symlinked in $HOME/.nix-profile/bin and thus we need to manually add your newly installed shell to /etc/shells in order to be able to use chsh -s ... as expected:

For setting up zsh, I ran the following commands:

cd ~
sudo echo "$(pwd)/.nix-profile/bin/<SHELL>" >> /etc/shells
chsh -s $(pwd)/.nix-profile/bin/<SHELL>

edit: For some reason, setting the shell this way doesn't quite work. Trying to run applications like tmux or sometimes weirdly during normal editing during vim causes the screen to get messed up. I'm not sure why this is the case but it doesn't seem to happen if my terminal emulator is xterm (or anything else Linux-native). All the Windows-native terminal emulators I've tried are buggy 🤔

I've resorted to keeping the default shell on Alpine as /bin/ash and I added /home/chris/.nix-profile/bin/zsh; exit to my ~/.profile to automatically start zsh on login.

Of course, this isn't ideal but it works fine for all of my use cases and is pretty transparent.

edit: I end up playing around with a few setups on a few different machines and WSL instances. I've written a basic script that is installed with home-manager to automatically make this change for me.

Post-install—Nix-ish services on WSL

WSL doesn't have a traditional init system (no SystemD or anything like it) and as such, a bunch of stuff doesn't work quite the same as usual. In more standard installations of WSL distributions, there still exists some service command which can be used for starting services manually; but no built-in way exists to do so automatically.

For me, the main service I want to be able to run is docker since I need it to do any kind of development work. It turns out that under most distributions, one shouldn't really install docker via nix since you won't be able to use nix to configure system services automatically (this is possible in NixOS but again, there is no official NixOS WSL distribution for us to use). In the past I've been forced to use my main distribution to install docker (i.e. via apt-get on Ubuntu WSL) and just accept the fact that one cannot use nix to manage docker outside of NixOS; but with some tweaking it's actually possible and ideal in Alpine!

Alternatively: you can install the Docker engine on Windows itself and configure it to use it's WSL 2 backend but I dislike doing this since I don't want to be managing anything development-centric inside Windows... regardless...

Hacking in Docker support

After digging around to find out what sudo service start docker is doing in Ubuntu WSL, it turns out that the main thing that needs to happen is one needs to execute sudo dockerd manually. Doing so will allow docker to do it's thing and even have docker-compose work exactly how you'd expect. Thus we can actually simply install docker via nix as per any other package as long as we're ok with executing sudo dockerd ourselves.

This can be automated to run on shell startup and automatically daemonized using the nixpkgs.daemonize utility. While this utility doesn't really do anything fancy such as automatic restarting of processes, generally it works well enough. You can daemonize a command (essentially just backgrounding it) by installing nixpkgs.daemonize and then executing daemonize <APP> <ARGS...> as you might expect.

A good first step would simply be to write a shell script that does the following automatically after logging in:

sudo daemonize -u root dockerd

dockerd needs to be run as root though, and thus one needs to always enter the root password when this runs. If this proves too annoying to endure (which it is for me since I already have to type in my ssh passphrase for keyring on first login) you can use the follow dirty hack to circumvent needing a password to run things as root:

# DISCLAIMER:
# This hack allows us to execute any command as any user without needing password authentication.
# This is _disgusting and insecure and is probably pretty problematic._ This is however, a nice simple way
# to get around the lack of any init system in WSL distributions w.r.t starting the docker service on login.
#
# As bad as this is, anyone with access to a WSL installation is able to do this anyway, so us abusing it
# for a better user experience is probably fine all things being considered.
#
# Please evaluate _for yourself_ before doing something like this.
nixpath=$HOME/.nix-profile/bin
wsl.exe -u root -e $nixpath/daemonize -u root $nixpath/dockerd

Please read the disclaimer comment before actually doing this 😅 I won't be held responsible for any pain/suffering/pwnage that might follow; but honestly if someone has access to your machine/terminal anyway, you're already owned. Doing this for the sake of UX is worth it in my opinion—especially since wsl.exe -u ... makes it so easy...

Other (standard, simpler) services

Lorri is an example of another service we want to always have running. Luckily for us (and thankfully for the creators of daemonize), we can simply write a script similarly to our docker-service script above (without the ugly root access hack) to start lorri daemon in the background on startup:

daemonize $HOME/.nix-profile/bin/lorri daemon

These can simply be added to your .bashrc or .zshrc and can these scripts can even be automatically written for you with a custom nix derivation. You can check out my nix configuration for examples of how this is done. If interested, the particular files responsible for doing this will be in modules/wsl/service-lorri.nix and modules/wsl/service-docker.nix.

Conclusion

After trudging through all this, you should now have a very minimal Nix on WSL environment ready for you to start hacking away at. Following this guide, I only have 23 packages installed and managed by Alpine's apk compared to the few hundred on a standard Ubuntu WSL installation and thus pretty much everything is managed by Nix in combination with home-manager.

While services are a little bit of a hurdle, that's not a Nix or Alpine problem perse, but more of a general issue to workaround WSL's limitations which hopefully one day will be fixed.

Hopefully this guide proves helpful to others trying to get something even remotely NixOS like on WSL 🍻 Thanks for reading!