Why is the kernel version still v3.4 and not higher?

[maorui2014]

• Windows 10 Enterprise Insider Preview Build 14951

Why is the kernel version still v3.4 and not higher?

[aseering]

Hi @maorui2014 -- what version do you think it should be?

[maorui2014]

I think it should be 3.10+.

[benhillis]

We should probably be reporting whatever Ubuntu 16.10 expects.

[rodrymbo]

@maorui2014: Just to clarify: it is not really a linux kernel; it is the Windows Subsystem for Linux "pretending" to be that linux kernel (and for the most part, succeeding). So anything it reports is at least a bit fictional.

Obviously, as @benhillis says, the WSL should report itself (and behave) as whatever kernel Ubuntu 16.10 expects. In a fantasy world, if you install some other distribution of Linux, the WSL should behave (and report itself) as whatever kernel that distro expects.

[ddfznt]

cat /proc/version

ubuntu:
Linux version 4.4.0-38-generic (buildd@lgw01-58) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2) ) #57-Ubuntu SMP Tue Sep 6 15:42:33 UTC 2016
wsl:
Linux version 3.4.0-Microsoft ([email protected]) (gcc version 4.7 (GCC) ) #1 SMP PREEMPT Wed Dec 31 14:42:53 PST 2014

[aseering]

@rodrymbo -- one challenge with that fantasy world (and, indeed, with Ubuntu itself) is that a single distribution release can support multiple kernels. For example, Ubuntu 14.04 ships official kernel packages based on Linux 3.13, 3.16, 3.19, 4.2, and 4.4 on x86_64 (the list is longer on arm). They have a Wiki page about their kernel-version support strategy here.

@benhillis -- I believe Ubuntu 16.10 uses a Linux 4.8 kernel. Also, for what it's worth, to my knowledge, no version of Ubuntu has ever packaged WSL's currently-selected kernel version of "3.4".

Follow-on question, @benhillis -- do y'all plan to support packages such as linux-tools? Those tools are rebuilt every time Ubuntu ships a kernel bugfix patch; they're tied to a very specific kernel version. I'd love to be able to use perf in particular to profile my code. I find that the current choice of a kernel version that is unambiguously not in Ubuntu's repo helps communicate to users that these sorts of kernel-dependent packages aren't supported.

[rodrymbo]

@aseering -- Well, if it reports itself (and/or behaves) as the distro expects, then it is okay, right? The only problem would come from some behavior the distro doesn't expect (or I suppose if the distro did something the kernel didn't expect).

@maorui2014 -- so what features of a differently-numbered kernel are you missing with what we are currently seeing? Obviously security enhancements and fixes aren't relevant in the same way.

I wasn't able to determine whether the original question was reporting WSL as being somehow deficient at pretending to be Ubuntu 16.10, or about wanting some feature or other of a later version of the kernel, perhaps even later than the kernel(s) available with 16.10.

Either way, at some point it might be good to start thinking of WSL as a whole different kind of kernel, not as just pretending to be a specific linux kernel -- like linux supporting different CPU architectures, only moreso. So Ubuntu could add "WSL 3.4" to the list of (linux) kernels it supports, rather than WSL 3.4 pretending to be linux kernel 4.2 or whatever. I'm not familiar with how or when or if a userspace app might want to query the kernel to learn its version, but as Adam says, if WSL reports 3.4 and there is no extant linux kernel 3.4, that would at least be a clue that it's not in Kansas any more.

[fpqc]

Could report Kernel as 10.buildnumber (although I did read that FreeBSD had big problems (like y2k lol) with their build numbers going up a digit when FreeBSD 10 released.

[carpet92]

@rodrymbo The Linux Kernel Hidden Inside Windows 10 on Black Hat 2016 https://www.youtube.com/watch?v=BgVGxWMu5W4

https://www.blackhat.com/us-16/briefings.html#the-linux-kernel-hidden-inside-windows-10

Initially known as "Project Astoria" and delivered in beta builds of Windows 10
Threshold 2 for Mobile, Microsoft implemented a full blown Linux 3.4 kernel in the
core of the Windows operating system, including full support for VFS, BSD Sockets,
ptrace, and a bonafide ELF loader. After a short cancellation, it's back and improved
in Windows 10 Anniversary Update ("Redstone"), under the guise of Bash Shell
interoperability. This new kernel and related components can run 100% native,
unmodified Linux binaries, meaning that NT can now execute Linux system calls,
schedule thread groups, fork processes, and access the VDSO!

As it's implemented using a full-blown, built-in, loaded-by-default, Ring 0 driver
with kernel privileges, this not a mere wrapper library or user-mode system call
converter like the POSIX subsystem of yore. The very thought of an alternate virtual
file system layer, networking stack, memory and process management logic, and
complicated ELF parser and loader in the kernel should tantalize exploit writers -
why choose from the attack surface of a single kernel, when there's now two?

But it's not just about the attack surface - what effects does this have on security
software? Do these frankenLinux processes show up in Procmon or other security
drivers? Do they have PEBs and TEBs? Is there even an EPROCESS? And can a
Windows machine, and the kernel, now be attacked by Linux/Android malware?
How are Linux system calls implemented and intercepted?

As usual, we'll take a look at the internals of this entirely new paradigm shift in the
Windows OS, and touch the boundaries of the undocumented and unsupported to
discover interesting design flaws and abusable assumptions, which lead to a wealth
of new security challenges on Windows 10 Anniversary Update ("Redstone")
machines.

[rodrymbo]

@Zx-EvM - Hi Dimitry

Are you saying that the video reveals that the "Ring 0 driver" it talks about is made up mostly of code from the linux kernel project, simply signed by Microsoft and set up to run as a Windows system driver? I was under the impression that it is Microsoft code from Microsoft developers. Maybe the video (that I don't have time to watch right now) says differently?

The thing is, code we might run in bash-on-ubuntu-on-windows is not able to tell that it is not an actual linux kernel (except when it runs into a beta-level bug that will get fixed), nor prove conclusively that it is. I get that it quacks like the proverbial duck. Still, unless you are saying that a Microsoft developer reveals in that video that they took most of the code from the linux kernel project, I'll continue to conclude that it is a Microsoft Windows system driver made to look and behave like a linux kernel.

Not that it matters a whole lot; my point was that when we (humans) look at the version number, it should reveal that it is WSL's version of linux, not Ubuntu's or Debian's or whatever.

[therealkenc]

At best WSL could indicate the version that MS' implementation of Linux is targeting. My Android phone is kernel version 3.10.61-7569018. So I can expect Torvald's core 3.10.61 features to exist on my phone, plus whatever features GoogleCorp and Samsung decided to add. WSL today might be 4.4.0-14951-microsoft.

Not that targeting means much, other than that MS implicitly says "we think Xenial mostly works". Nor does the kernel version number itself tell you much, because what a given Linux version can do is determined more by its configuration than its version.

Userspace interaction with the kernel is defined by the syscall signatures, the kernel netlink protocol, ioctls on /dev, and the poor-man's acsii format of /proc and /sys. WSL seeks to emulate that surface. Indeed you can't even be sure /proc/config.gz (CONFIG_IKCONFIG_PROC=y) exists on real Linux, even though it has been a feature since 2.6.0. Linux performance counters have existed since 2.6.31, but only if CONFIG_PERF_EVENTS=y.

Which is why userspace software must probe for existence of kernel features (of or lack thereof) themselves, or, at their sole discretion, just break horribly. Which is why almost nothing, not even linux-tools, bothers to look at /proc/version.

[aseering]

@therealkenc -- good point regarding linux-tools and /proc/version. To be more precise, the linux-tools package depends on the specific version of the kernel package. (And apt does enforce that.) I realize that's different than /proc/version, but if /proc/version and apt are in some sense out of sync, then someday I expect to see someone file a ticket about that... There's no technical need for them to be the same but it's a bit of a surprise if they're not.

[carpet92]

@rodrymbo but if it is as you say GCC should compile to PE (Windows) instead of ELF (Linux). You can see this StackOverflow Question http://stackoverflow.com/questions/38786014/how-to-compile-executable-for-windows-with-gcc-with-linux-subsystem Linux Subsystem works as a Linux-computer. In my previous message is a summary of the video:

Key:

Microsoft implemented a full blown Linux 3.4 kernel in the
core of the Windows operating system, including full support for VFS, BSD Sockets,
ptrace, and a bonafide ELF loader

[therealkenc]

@aseering Understood. WSL doesn't, and can't, depend on the Ubuntu kernel-image-generic aptitude package. So it isn't that surprising /proc/version doesn't match. They can't match. There's an implied loose coupling with the initial Canonical image downloaded from Windows Store, but that's all.

For what it's worth, linux-tools isn't tied to "a very specific kernel version" as tightly as you might be implying up-thread. They are tied tighter than say glibc is tied to a given kernel. But not objectively more-so than say how the mesa userspace (libGL.so) is tied to whatever Intel/AMD/NVIDIA feels like doing in any given Linux kernel release.

[therealkenc]

@Zx-EvM "full blown" in that sentence isn't meant to be taken literally. Better:

Microsoft has implemented a very impressive emulation of the Linux kernel system surface, 
approximating somewhere between Linux 2.6.0 and 4.4.0, depending. This includes pretty 
passable support for VFS, much of the BSD socket layer, ptrace, and a bonafide ELF loader

[fpqc]

@Zx-EvM It's more of a kernel extension/add-on than a full kernel. At least for the moment, all hardware access still has to go through the NT Kernel (although there is no usermode-kernelmode context change going on, since the translation to NT happens in the lxss kernelmode). Alex has done some more research more recently and found a new kernelmode driver interface on the lxcore (extend the module out by some more modules), but it's not clear if this is going to be documented for third parties to write drivers or if Microsoft is going to use the interface to write generic shim drivers in between the real Windows driver stack and the lxcore driver.

If they could write a generic interface for, say, graphics driver ioctls being transformed into a generic Windows kernelmode interface for graphics, this would mean MS would probably have to distribute their own libgl for that driver shim. On the other hand, MS might leave it up to the graphics card companies, in which case they might document it for them. We'll see. It could be even easier, depending on how Windows and Windows driver-makers implement OpenGL.

It's hard to say because neither side of the interface in Windows is open source or (freely?) documented, and I'm not a reverse engineer.

[rodrymbo]

@Zx-EvM

Here's another statement from a MSDN blog, which includes a diagram:

By placing unmodified Linux binaries in Pico processes we enable Linux system calls to be directed into the Windows kernel. The lxss.sys and lxcore.sys drivers translate the Linux system calls into NT APIs and emulate the Linux kernel.

I suspect the statements you are quoting are from journalists or researchers or enthusiasts who aren't constrained by being Microsoft employees to stating things clearly and accurately. It's easier to summarize as "it's a linux kernel" into an article than to say that "lxss.sys and lxcore.sys drivers translate the Linux system calls to emulate the Linux kernel". And as I said, it doesn't matter a whole lot, since software running in WSL can't tell the difference (when everything is working right, since it is beta-level still).

So back to the original question: what features or characteristics of 4.8 or whatever does the OP miss in the current release, besides it reporting it is 3.4? Is something not working as expected, or is it too slow, or is there a security risk that is not addressed in Microsoft's implementation of 3.4, or what?

[fpqc]

@rodrymbo Alex Ionescu knows what he is talking about, but maybe his talk's abstract took liberties in the expectation that any lack of clarity would be remediated by watching the actual talk. I've watched the video of that talk and he is absolutely crystal clear.

[ghostoy]

This seems to be fixed by reporting as 4.4 since build 14986.

[benhillis]

Correct. In creators update we report 4.4.

[rainabba]

So how do I get perf (usually provided by linux-tools) installed?! :)

Seriously, I'm trying to profile a node.js app and find myself wondering if I need to build a custom kernel (which I now know can't be done) because I cannot figure out how to install perf and my attempts to do so indicate that a package doesn't exist for "this kernel", but as this conversation shows, things care about the kernel version :/

@therealkenc Back to this thread yet again. Finally got perf compiled/installed manually, and this I hit another wall which brought me right back here (not just a coincidence I'm sure).

CONFIG_PERF_EVENTS apparently isn't implemented in the pseudo-kernel behind WSL. This makes profiling a node.js app on WSL quite next to impossible.

Here's the issue I filed with that project (knowing it may be entirely dependant on WSL): davidmarkclements/0x#68

[therealkenc]

@rainabba - Yep CONFIG_PERF_EVENTS is #329 #1577, maybe others. I don't know if anyone ever did a User Voice.

[WSLUser]

Since we're speaking about linux kernels here, can the WSL simulated kernel keep up with all the stable releases the Linux kernel team creates? For instance: Linux 4.14 has been released on 12 Nov 2017.

Linux 4.14 has been released on 12 Nov 2017.
Summary: This release includes support for bigger memory limits in x86 hardware (128PiB of virtual address space, 4PiB of physical address space); support for AMD Secure Memory Encryption; a new unwinder that provides better kernel traces and a smaller kernel size; a cgroups "thread mode" that allows resource distribution across the threads of a group of processes; support for the zstd compression algorithm has been added to Btrfs and Squashfs; support for zero-copy of data from user memory to sockets; better asynchronous buffered I/O support; support for Heterogeneous Memory Management that will be needed in future GPUs; better cpufreq behaviour in some corner cases; Longer-lived TLB entries by using the PCID CPU feature; asynchronous non-blocking buffered reads; and many new drivers and other improvements.
See this link for more details.

I think supporting the features of the linux kernel should be something that's looked at and considering a stable release is issued every 2 to 3 months, we would simply see the changes upon the next update for stable Windows (this would include at least 2 versions worth of feature enhancements). I'm not sure how much work would be involved and obviously some features would be prioritized over others but supporting kernel changes would probably attract more people over to Windows to use WSL instead (especially when UserAsks such as FUSE and GUI support is finally a reality). I personally see a day in which we don't see any differences whatsoever between running a WSL environment and a real Linux environment but understandably that day is long in coming (though perhaps the rise of quantum computing once publicly released can help speed things up).