runtime: remove the mlock hack in Go 1.15

[DanielShaulov]

The resolution to #35777 was to mlock the top of stack on "affected" kernels. But then it was discovered that major distros use kernels that appear by the version number to be affected but are actually patched (#37436). It was also discovered that docker under systemd has a very low limit on mlocked pages (same issue). The resolution for that was to stop reporting failures of mlock and delay the warning until a crash is caught.

There is a TODO comment in the code to remove all of that hack for Go 1.15, since unpatched kernels are unlikely to be encountered, but Go 1.15 is due to be released in a short time and the code is still there.

I think it is important to remove the workaround since Ubuntu 20.04 LTS uses a patched 5.4.0 kernel. This means that any user on Ubuntu 20.04 will still unnecessarily mlock pages, and if he runs in a docker container, that warning will be displayed for every crash, disregarding the fact that his kernel is not really buggy. So those users might be sent on a wild goose chase trying to understand and read all this info, and it will have nothing to do with their bug, probably for the entirety of Ubuntu 20.04 life cycle.

Relevant code is in src/runtime/os_linux_x86.go

[ALTree]

There is a TODO comment in the code to remove all of that hack for Go 1.15 [...] but Go 1.15 is due to be released in a short time and the code is still there.

Can you link where this TODO is? The only relevant comment I found was on CL 223121, in the runtime/panic.go file, but it says:

// TODO(austin): Remove this after Go 1.15 when we remove the
// mlockGsignal workaround

and "after Go 1.15" means for Go 1.16.

[randall77]

@aclements

[nemith]

Since the kernel detection is done based off of major/minor numbers we actually have to apply a patch to internal builds of the compiler to include our own kernel builds which are not affected but fall in the affected range. I would love for this to be removed or at least optionally bypassed

[networkimprov]

I don't think this needs further info from the author to be actionable...

@gopherbot remove WaitingForInfo

[aclements]

@nemith , can you explain why you need to patch the compiler? As of Go 1.14.1, the only effect of an mlock failure is an additional message if there's an uncaught panic, so I don't understand why that requires patching.

[ianlancetaylor]

If there is a way to detect the patched kernel, perhaps by reading /proc/version, I think we could implement that safely enough in 1.15.

[networkimprov]

Re kernel version sources, see this and comments immediately preceding it: #37436 (comment)

[aclements]

There's one deciding factor here: is the kernel patch widespread enough that the workaround is no longer necessary? I'm sure all the distros have it, but people aren't necessarily on top of upgrading their kernels. Unfortunately, I have no idea how to figure out the answer to this, and I think we should be fairly conservative about this given the nature of the kernel bug.

[lootch]

On 7/13/20, Daniel Shaulov ***@***.***> wrote: The resolution to #35777 was to `mlock` the top of stack on "affected" kernels. But then it was discovered that major distros use kernels that appear by the version number to be affected but are actually patched (#37436).

It may be a bitch to do and contrary to Go best practices, but an "alert" (nag?) mode that notifies users that the kernel they run under needs a patch, or even better, gives them the ability to check if their kernel needs attention may be the only politically correct answer here. Something that can easily be removed once the user has attended to it. Or, perhaps better, something they can add as part of the installation and drop as soon as possible. Lucio.

[networkimprov]

A compromise solution: omit the mlock workaround for a few major distros when the version is known to include the kernel fix.

More distros could be added in later 1.15.1+ releases by folks who can test them.

[ianlancetaylor]

@DanielShaulov Would it address your problem if we provided an environment variable to disable the use of mlock, as in GODEBUG=mlock=0?

[gopherbot]

Change https://golang.org/cl/243658 mentions this issue: runtime: let GODEBUG=mlock=0 disable mlock calls

[ianlancetaylor]

CL 243658 adds GODEBUG=mlock=0 to disable the use of mlock. If that fixes the problem, we can commit that for 1.15 (and perhaps 1.14.7) and then repurpose this issue for removing this code for 1.16.

[DanielShaulov]

Hi, I am very sorry for not responding, didn't get notifications until I got pinged.

@ALTree Here is the relevant TODO, it says "in Go 1.15" and not "after Go 1.15":

go/src/runtime/os_linux_x86.go

Lines 31 to 33 in 11f92e9

	// TODO(austin): Remove this in Go 1.15, at which point it
	// will be unlikely to encounter any of the affected kernels
	// in the wild.

But I see that the other places used "after Go 1.15", so this one might have been a mistake?

@networkimprov At the very least - all the 5.4.0 kernels that Ubuntu ever published were patched (I checked all the 5.4.0-* tags in their git). So I think just checking for that will cover a pretty large set of false positive.
Redhat family can be checked to see if they have an LTS release stuck with a kernel version in the "bad" range, but if not then their problem is much smaller than Ubuntu's.

@ianlancetaylor See above about the confusion about the version it was supposed to be fixed. I am OK with re-purposing the issue for 1.16, I was just worried that this was just forgotten and be left like this for the whole duration of Ubuntu 20.04.
But I still think adding another check for the major distros as networkimprov suggested is a good idea.
I don't think the DEBUG=mlock=0 thing would really solve the issue. Developers will still spend time looking into those issue, reading the wiki, running the test program, just to find out that their kernel is already patched, and their issue lies elsewhere.
But the resolution for this can wait for Go 1.16 I guess.

[ianlancetaylor]

@DanielShaulov Thanks. Since the GODEBUG environment variable won't help, I'll drop that CL.

I'm not sure how to proceed with checking major distros. What we need is some mapping from the /proc/version string to kernels that are known to be fixed.

[DanielShaulov]

Well - first we need to get our distro name, for that I think /etc/os-release is the most reliable way. We just need the NAME= field. (And if we fail that, no big deal, we just fall back to current behavior of just checking official kernel release).

Then, looking at #37436 (comment) we can probably just parse the end of the kernel release (the part after - which is currently discarded in parseRelease) and save it as tail,
And then in osArchInit, we can just switch on the distro name an have our per-distro logic.

For Ubuntu, it is a dead simple if major == 5 && minor >= 4 { return }, and doesn't even need the tail, since all 5.4.x kernels in Ubuntu are patched. (other distros might need the tail)

[ianlancetaylor]

https://golang.org/cl/244059 is code that skips the mlock on Ubuntu 5.4 systems, based on reading /proc/version.

[gopherbot]

Change https://golang.org/cl/244059 mentions this issue: runtime: don't mlock on Ubuntu 5.4 systems

[networkimprov]

@ianlancetaylor you could leave this issue open for discussion of other distros during the 1.16 cycle.

[ianlancetaylor]

In the 1.16 cycle we're going to remove the mlock calls (#35979).

[networkimprov]

But you may add other distros to the don't-mlock list in 1.15.x.

[ianlancetaylor]

@networkimprov True, but I don't see that as a reason to keep this issue open. There is no action that we would take to close the issue.

[gopherbot]

Change https://golang.org/cl/246200 mentions this issue: runtime: revert signal stack mlocking