Setting up a development environment on Windows 10

[vereis]

Setting up a development environment on Windows 10

I find myself writing a lot of code, both professionally and as a hobby. When I write code, 99% of the time, I'm targeting a Unix-like environment.

I also find myself doing a lot of other things, such as playing games or streaming some TV. It is my sincere opinion that for these things, Windows is the operating system to beat: monopolistic malpractices or not, 4K Netflix only works on Windows 10, as do the majority of games I play which I use to keep in contact with old friends.

Even when I'm not using my powerful workstation with its beefy GPU to play games, I frequently use various models of the Microsoft Surface lineup because their form factor is really appealing to me—especially for a machine for the go. Linux, without surprise, does not run very well at all on these devices 😅

None of this is Linux's fault, of course. If I could use Linux on everything, I would, but the year of the Linux desktop is certainly not 2019 for me. It's not for lack of trying either, I've been a longtime Linux user, and I've tried many approaches to try and get the best of both worlds. I hope that this post can share what I've learnt along the way 💪

Popular (or not) approaches I've tried

As I've said, I've tried a lot of different ways to combine the leisure of Windows and the development potential (and customisability) or Linux. Some of the ones I actually stuck to for a while are outlined in the seconds below:

Windows–native development

For a long time, primarily during the first half of my University experience, I used Windows as a host OS and developed exclusively on/for Windows. If I were writing Python, or Erlang, or Ocaml, I'd use the Windows-native builds of those programming languages/compilers.

This felt natural to me because I started my developer journey when I was nine and playing around with Game Maker. It also worked relatively well for interpreted languages, as well as for C/C++ so long as I was developing against a Windows API. Everything kind of works, but a few weird things also don't work how you'd expect; maybe because most developers writing in languages such as Python or Erlang aren't using the Windows builds of their toolchains? 🤔 I assume if I wanted to stick to C# or .NET development, say, things would go a lot smoother, but alas.

When I couldn't manage to compile some NIFs for the BEAM (which I needed for a particular project at University), I ended up looking for another solution.

Linux VM on a Windows host

After trying to go all-in with native Windows development and being bitten by a couple of small and weird issues, I decided to do what anyone in my situation might and download a virtual machine to run Linux on.

The first thing I tried was VirtualBox, which ran pretty slow and had awful input latency when doing anything. I think this might have been a hardware issue.

VMware Workstation worked a lot better but still suffered from weird quirks like the virtual sound device not working correctly.

If not for the sound device issue, I probably could have lived with this setup. VMware could be maximised, and all of its Window decorations/toolbars could be hidden. It had some rudimentary 3D acceleration support as well, and things generally ran very smoothly. Unfortunately, if I were to do this, I'd want the sound to work as expected; it was also expensive both monetarily and battery-wise for my non-workstation machines.

The Windows ricing community: Cygwin & bblean

In 2015, I really fell into the Windows ricing community. To eke out the customisation that Linux gave me on Windows, I hacked around with hundreds of custom AutoHotkey scripts for system automation and custom keybindings, custom theming, custom shells to replace Window's default explorer.exe and more.

I discovered a custom shell for Windows called bblean which was based on Blackbox, a Linux window manager, perhaps best known nowadays for being a precursor of the much more popular Openbox and Fluxbox window managers. This scratched my customisation itch like never before and was genuinely perfect—until I switched to Windows 10 for modern DirectX support. Unfortunately, bblean was built targeting Windows 2000, and it was a miracle it worked as well as it did on every OS until Windows 10, where it still kind of works to this day.

There weren't very many people using bblean, and other alternative shells on Windows, however. The community was very tight-knit, and we shared a lot. Through being a part of this community, I also discovered Cygwin, which is essentially a collection of GNU (and other) tools, and a DLL providing POSIX-like functionality that you can install on Windows. Cygwin allows you to both install applications you might expect to find as part of a Linux distribution (such as zsh, vim, git etc.), and also compile POSIX-compliant applications to run on Windows natively (provided you have the DLL Cygwin provides).

The community was very focused on sharing setups and ideas. I'm unfortunately no longer an active participant in the community because I simply don't use Cygwin anymore (for my use case, a tool we'll explore in a moment is a much better fit). Still, I do have a few old screenshots of the lengths I'd go using these tools—something always fun to reminisce about.

The following images are Windows 7 configurations I had in 2015 and beyond. I taught myself CSS and basic JavaScript to be able to customise my Firefox 😁

Retrospectively, these tools taught me a lot and even contributed a lot to my first paying software development experience (which was a small company I interned at during my University degree. We used Cygwin on Windows extensively, and I attribute my outside experience with having used Cygwin as a strong selling point!)

These tools shall remain forever in my heart ❤️

Windows VM on a Linux host with GPU passthrough

After a few years of the bblean/Cygwin setup, I decided that I would try and learn to use Linux properly and full-time: my internship had just ended, and I had a lot of time on my hands before my final year at University, I had used Linux as a server OS and was familiar with the GNU userland because of Cygwin, but I was not 100% familiar with using it as a host OS for day to day computing.

It worked relatively well, all things being equal. Games, in general, don't work (though Valve's Proton is making genuinely amazing leaps to this end), especially online multiplayer games with anti-cheat. There were other small annoyances, such as high-dpi support not quite working properly (Wayland was not anywhere near being ready yet 😏), but it worked fine in general.

To get around the gaming issue, I set up a virtual machine using QEMU + GPU passthrough, which allowed me to run a Windows 10 in a virtual machine with a dedicated GPU. This worked amazingly, and game performance was effectively native—I couldn't even tell I was in a virtual machine!

It did require 2 monitors, however, and a software/hardware KVM to be used so that I could move my input devices from my host machine to the VM... and every time my GPU's driver updated, I'd end up getting bluescreens 😅

So close, yet so far...

Windows Subsystem for Linux

I heard about Windows Subsystem for Linux on Hacker News while I was still using my GPU passthrough solution and decided I'd try it out. If it worked well enough, I'd switch to it, given the grief my driver updates were causing me.

In short: it was similar to Cygwin in that it gives you a Linux-like CLI on Windows, but it worked by intercepting and translating Linux syscalls to NT syscalls in real-time, providing a real Linux userspace! No more Cygwin bundled DLLs and compiling any software I wanted for myself. I was effectively running a reverse Wine, and things worked pretty well!

This was pretty frictionless, but the performance wasn't the best (primarily because of Windows Defender and NTFS filesystem performance, apparently), but it was good enough for most things. Since WSL worked via syscall translation, however, there were some things that would not work: the lack of a real Linux kernel meant no Docker, no rootfs, no app images, but you could typically work around these issues.

Microsoft then decided to up their game by providing an alternative implementation of WSL: instead of translating syscalls, they would do the following:

1. Run a real Linux kernel alongside Window's NT Kernel using Hyper-V
2. Wrap said Linux kernel into a super lightweight VM (kind of like a Docker container)
3. Perform some black magic kernel extensions to enable Linux–Windows interoperability
4. Name it WSL 2 and profit 💰💰💰

I have been using WSL 2 in production since it was first available to try out in a preview branch of Windows. It's been nothing but a pleasure to use, and when using it, I genuinely feel like I have the best of both worlds.

The rest of this post will be about how to build a development environment atop WSL 2; however, before carrying on, there are two main things to note:

1. WSL 2 requires Hyper-V, which means your Windows 10 version will need to support it, and at the time of writing, this makes a machine using WSL 2 incompatible with Virtualbox or VMware software.

2. WSL 2 is currently only available in preview builds of Windows 10 as it's currently in development. WSL 2 is due for general release in Q1 2020, which isn't too far off, at least 🙏

Setting up WSL 2

Right now, WSL 2 is only available on Insider builds of Windows 10. You need to be on build number 18917 or higher to install it. You can follow these instructions to opt into these Insider builds, but it is important to note that you'll need to re-install Windows if you ever want to roll back to the standard distribution channel.

Next, we can enable the pre-requisite services for WSL 2 by executing the following lines in an elevated Powershell terminal (win+x should help bring it up):

Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

You'll be asked to reboot after running these commands, and after you do, you can go to the Microsoft Store and install any WSL distribution you'd like. I usually go for Ubuntu since it's the most supported one at the time of writing, though others are available.

To speed things up, after installing the distribution of your choice, but before executing it, in a Powershell or CMD prompt, execute the following:

wsl.exe --set-default-version 2 

This will cause your installed distribution to initialise itself in WSL 2 mode. You can convert versions freely between WSL 1 mode and WSL 2 mode, but it takes quite a long time, even for a base installation of Ubuntu, so running wsl.exe --set-default-version is preferable in my mind.

Congratulations, you now have access to a pretty complete Linux computing environment inside Windows 10! 🎉 Assuming you're using Ubuntu, you'll be able to apt-get install and run most commands you usually would.

After getting this far, one glaringly obvious issue is that your WSL 2 installation is being hosted in an old-school CMD prompt. This isn't ideal because it doesn't do very much: no true colour support, copy/paste is broken, and it generally isn't very customisable. Thankfully we can work around that in the next section 😉

Choosing a Terminal Emulator

There isn't too much choice for Terminal Emulators on Windows; unfortunately, every single one has some tradeoffs to consider.

For example, the default CMD prompt is actually blazingly fast with regards to input latency. It lacks, however, pretty much everything else I'd want from a terminal outside of decent mouse support (surprisingly). At this point, both my customised Vim configuration and Tmux completely break text rendering because of weird Unicode rendering issues however 😅

Cygwin's default terminal is pretty good. It's called minTTY, and it basically feels like the standard CMD prompt but supercharged. It has mouse support, 24-bit true colour support and more. Unfortunately, it also doesn't support rendering Unicode, so it's a no-go for my use case. If you're okay with Unicode not being supported, you can download wslTTY, which is essentially minTTY with WSL-integration—neat!

ConEmu is another super popular terminal emulator for Windows. It (and its several derivatives) are extremely customisable, approaching the level of terminal emulators you'd find on Linux. There are two main downsides with ConEmu, however: it doesn't easily support WSL by default (it's possible to get a working configuration, but of all the ones I tried, there are always tradeoffs with respect to losing some functionality, be it mouse support or weird text wrapping issues), and it gets very, very slow when running WSL for some reason 🤔 This might be due to the extreme newness of WSL 2 (or the fact what I'm running an insider build of Windows). Because of these issues, I cannot recommend ConEmu at this moment in time.

Then there's Windows Terminal, a super-strong new contender from Microsoft. They really have been doing amazing stuff in the last few years! This terminal emulator supports almost everything I want (lightning-fast, customisation features like the ability to set window padding, ligature support, true colour support), but unfortunately, it doesn't support any kind of mouse integration at the moment, which is a crutch I use a lot in my workflow 😭 Support is planned for the future, and right now the project is super early access. This is one to keep an eye on I'm sure.

Foregoing any of the imperfect choices already talked about (but really, feel free to try them all yourself. It's pretty workflow dependant after all), you can actually run an X Server on Windows and simply use X11 forwarding to run your favourite Linux terminal emulator as though it was running natively. I personally use this approach with the popular terminal emulator called Terminator, but as Windows Terminal matures, I may switch to that.

X11-forwarding between Windows and WSL

A few popular X Servers exist for Windows that work, and most of them are pretty similar. The most popular seem to be VcXsrv, MobaXTerm and X410.

I personally choose to use VcXsrv because it's the one I was already using from my Cygwin days (I believe Cygwin has its own X Server, but I've never tried it). VcXsrv is pretty fast, and it is free.

I have tried X410, and while it worked really seamlessly (it was built with WSL integration in mind, seemingly), my display is 1440p, and running applications would stutter and lag, unlike on VcXsrv. This is likely an issue that'll be fixed soon, but for what it's worth, you also need to buy a license to run X410 from the Microsoft Store.

Both VcXsrv and X410 (and I believe others) will enable you to start them in "multiwindow" mode. This essentially behaves as though Window's explorer.exe is the window manager for your X11 applications, and everything feels super native. Here you can see an image of me running a popular database management GUI, dbeaver, on WSL with X11 forwarding to Windows:

As you can see, it looks and behaves just like any other application!

Before you can run any applications via X11 forwarding, you'll need to add the following line to your shell's configuration:

WSL_HOST=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}')
export WSL_HOST=$WSL_HOST
export DISPLAY=$WSL_HOST:0.0

This sets your WSL instance's X11 display to the server running on Windows. For maximum performance, it's also often recommended to add the following line as well:

export LIBGL_ALWAYS_INDIRECT=1

This forces rendering to happen on Windows' side, rather than in software via WSL itself. Having done this, source your changes, and you should be able to just launch any GUI program and have it just work. Even chromium and vscode work if you really want to run them inside WSL for some reason; performance is decent too!

You can make X410 or VcXsrv automatically start by adding shortcuts to them in the following directory: %appdata%/Roaming/Microsoft/Start\ Menu/Programs/Startup/. I really recommend doing this since there isn't much reason not to avoid running an X Server on Windows. It helps improve the system interoperability story even more than already exists!

Other minor notes

How I use Docker

When I was using the original WSL, I was forced to run Docker on Windows and connect to it from my WSL instance. Occasionally things just broke, and this took time to debug and troubleshoot.

Now that WSL 2 is a thing, Docker for Windows has been updated to have its own WSL 2 backend. This works a lot better than standard WSL + Docker on Windows, but it's still not perfect. Occasionally, not shutting down or restarting my workstation for a few days would cause strange networking issues between my containers. I could get around this by running wsl.exe --shutdown but this, of course, also forced me to stop my work.

WSL 2 does, unlike WSL 1, include a real Linux kernel, however, so I opt for just running the native Linux implementation of Docker inside my WSL 2 distribution, and that just works. At this point, I'm not sure why Docker for Windows is even a thing 🤔

After switching to doing this, everything has been super stable! WSL doesn't support any kind of system services, however, so at least once per boot, you'll need to remember to run sudo service docker start to get everything in working order.

Creating shortcuts to Linux tools

Right now, any GUI application you want to run from WSL needs a command prompt open. This is because WSL shuts itself down if no terminals are open.

You can use (probably with some minor tweaking neccessary) the following Visual Basic script to spawn an invisible terminal as a host to launch your applications:

Set oShell = CreateObject ("Wscript.Shell") 
Dim strArgs
strArgs = "wsl bash -c 'DISPLAY=WSLHOST:0 <APPLICATION>'"
oShell.Run strArgs, 0, false

Simply change <APPLICATION> to your application of choice (i.e. vscode) and save this script as (in this example) vscode.vbs. Running this script will cause Visual Studio code to open as though it were a normal Windows shortcut 💪

Miscellaneous tips

You can get Linux window management capabilities such as easy dragging/resizing with AltDrag

Microsoft Powertools is a collection of utilities such as a pseudo tiling window manager.

You can set up hotkeys easily with AutoHotkey. I personally utilise these hotkeys in conjunction with the shortcut visual basic script I detailed above to launch Linux applications via keyboard shortcuts.

WSL allows you to import and export your distribution by simply running wsl.exe --export <YOUR_DISTRO> <PATH_TO_EXPORT_TO> and wsl.exe --import <NAME_FOR_IMPORTED_DISTRO> <PATH_TO_IMPORT_TO> <PATH_OF_EXPORTED_DISTRO>

Conclusion

I hope that this post helps you set up and configure a WSL 2 based development environment on Windows 10! There were definitely a few bumps along the way, but I think it works pretty well when you consider how new the technology is and how many small moving parts there are!

This technology is super excited for me, and it's only going to get better going forward. I'm just happy I get all the benefits of running Windows while still being able to develop with Unix-like semantics/idioms. 👀 on the future for even more improvements and interoperability! 🤞