runtime/race: ThreadSanitizer failed to allocate 0x0000005c9000 (6066176) bytes at 0x200dc940a0000 (error code: 87)

[neclepsio]

What version of Go are you using (go version)?

$ go version
go version go1.16.4 windows/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\---\AppData\Local\go-build
set GOENV=C:\Users\---\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=c:\---\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=c:\---
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\---\Go\go1.16.4
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\---\Go\go1.16.4\pkg\tool\windows_amd64
set GOVERSION=go1.16.4
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\---\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\---\AppData\Local\Temp\go-build71564379=/tmp/go-build -gno-record-gcc-switches

What did you do?

Use the source in "Usage" section of https://github.com/go-gl/glfw, then run:

$ go build -race; ./test

What did you expect to see?

The program correctly running, as if -race was not provided.

What did you see instead?

ERROR: ThreadSanitizer failed to allocate 0x0000005c9000 (6066176) bytes at 0x200dc940a0000 (error code: 87)

The error appears on each run. The count, address and error code are always the same. Using different program, the count and address are different.

[randall77]

The race detector uses ~10x the memory that a run without -race uses. Are you actually running out of memory? How much memory does a regular run use?

It is possible that the race detector can't allocate memory because the address space it wants is being used by something else.
I'm not sure exactly what would cause that, but if your code mmaps or dlopens lots of stuff, that might be a contributing factor.

[neclepsio]

The task manager show less then 10Mb of use for the simple example. Looking to GLFW source code, mmap is used for some small bitmaps and there are some dlopen, just to OpenGL and related libraries, but it seems to me that they are not used in Windows (LoadLibraryA is used instead, I don't know if it is the same). Moreover, error code 87 is "the parameter is incorrect" under Windows.

[randall77]

Hm, I don't know then. You have officially overreached my Windows knowledge :(

[heschi]

cc @dvyukov @bufflig

[jazzy-crane]

I too am seeing this on my project. I need to confirm, but I think I wasn't seeing the issue with 1.16.3, but then hit it with 1.16.4

[targodan]

Also seeing this with fkie-cad/yapscan on windows only.
That being said, I do do a bunch of c-stuff and reading remote process memory.

go version go1.16.4 windows/amd64

[jazzy-crane]

I too am seeing this on my project. I need to confirm, but I think I wasn't seeing the issue with 1.16.3, but then hit it with 1.16.4

Sorry, scratch that. I do see with 1.16.3 too. Go version bump coincided with other toolchain upgrades

[networkimprov]

cc @zx2c4

[neclepsio]

The problem is still present in 1.16.5.

[neclepsio]

Also affects 1.17beta1.

[neclepsio]

Still present in 1.17rc1

[networkimprov]

cc @dvyukov

[dvyukov]

I don't have a windows machine, so I can't debug it.
Why can VirtualAlloc return 87? I thought maybe it's because size 0x5c9000 is not a multiple of allocation granularity (64k), but the msdn page says that in such cases the size is simply rounded up.

[targodan]

Is the MEM_LARGE_PAGES flag used? In that case the size needs to be a multiple of GetLargePageMinimum.

3. Include the MEM_LARGE_PAGES value when calling the VirtualAlloc function. The size and alignment must be a multiple of the large-page minimum.

Source: https://docs.microsoft.com/en-us/windows/win32/memory/large-page-support

[neclepsio]

I don't know if it's relevant, but according to this message in haskell forum: "this means that address given to VirtualAlloc is either not reserved yet, or that size is too big (ie the block-to-be-committed isn't inside one VirtualAlloc MEM_RESERVE block)". Another source point to ASLR, but this should not be the case since the problem is always with the same address. This IBM bug report states: Windows API VirtualAlloc is requesed to allocate memory ofsize 64KB with flag MEM_LARGE_PAGES. This is a non-standard allocation and the API fails with an error "The parameter is incorrect".

[dvyukov]

MEM_LARGE_PAGES does not seem to be used in compiler-rt/* dir:
https://github.com/llvm/llvm-project/search?q=MEM_LARGE_PAGES

[neclepsio]

This function matches the error message. There is an exception for Windows AddressSanitizer not applied to Go but it seems not related.

[targodan]

It also seems like the allocation is done with a fixed address?

Even if that is not the core of the issue in this case (since it seems to always happen), wouldn't that potentially lead to pseudorandom failures if ASLR decides to load a DLL there?

[dvyukov]

It also seems like the allocation is done with a fixed address?

Yes, with a fixed address.
Well, first, tsan needs to allocate memory at a fixed address. So it's not that it's done for no reason, nor that it's possible to just remove the address.
Second, on linux tsan will avoid regions where kernel can load anything and it reserves the remaining regions, so that even user mmap's won't happen at these addresses. I don't remember what happen on windows.

[targodan]

I see. Thanks for clarifying. :)

[AlexRouSg]

Thought I'd give some data points:

It runs fine if I cross compiled from linux with (GCC) 9.3-posix 20200320
It does not run if I used TDM GCC 9.2 or 10.3
It also runs fine with (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0

Looks like it might be specific to TDM GCC

EDIT:

Running with gdb, it crashes at:
racecall(&__tsan_map_shadow, start, size, 0, 0)

https://github.com/golang/go/blob/go1.16.6/src/runtime/race.go#L393
https://github.com/golang/go/blob/go1.16.6/src/runtime/race_amd64.s#L413

[neclepsio]

This also happens with MSYS2 GCC.

$ gcc -v

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=c:/Ignazio/Lavoro/IgnPack/Go/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-10.3.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 
--host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 
--with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib 
--enable-bootstrap --enable-checking=release --with-arch=x86-64 --with-tune=generic 
--enable-languages=c,lto,c++,fortran,ada,objc,obj-c++,jit --enable-shared --enable-static --enable-libatomic 
--enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes 
--enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-lto --enable-libgomp 
--disable-multilib --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers 
--with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 
--with-isl=/mingw64 --with-pkgversion='Rev5, Built by MSYS2 project'
 --with-bugurl=https://github.com/msys2/MINGW-packages/issues --with-gnu-as --with-gnu-ld 
--with-boot-ldflags='-pipe -Wl,--dynamicbase,--high-entropy-va,--nxcompat,--default-image-base-high 
-Wl,--disable-dynamicbase -static-libstdc++ -static-libgcc' 'LDFLAGS_FOR_TARGET=-pipe 
-Wl,--dynamicbase,--high-entropy-va,--nxcompat,--default-image-base-high' 
--enable-linker-plugin-flags='LDFLAGS=-static-libstdc++\ -static-libgcc\ -pipe\ 
-Wl,--dynamicbase,--high-entropy-va,--nxcompat,--default-image-base-high\ -Wl,--stack,12582912'
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.3.0 (Rev5, Built by MSYS2 project)

[MuweiHe]

Having the same issue when trying running make test on azure pipelines. Win 2019 vm, GCC 11.2, Go 1.16.3. Has anyone tried with earlier/newer versions of Go and does that work? And the address each run is different in my case..

[jazzy-crane]

Having the same issue when trying running make test on azure pipelines. Win 2019 vm, GCC 11.2, Go 1.16.3. Has anyone tried with earlier/newer versions of Go and does that work? And the address each run is different in my case..

I didn't have this problem with 1.16.3 , but was running musl GCC 9.2.1 mingw64 toolchain . When I upgraded Go to 1.16.4 I upgraded the toolchain to GCC 10.X at the same time. That's when I started having problems. So I think the GCC version is the significant thing, not the Go version. Unfortunately I haven't been able to try with >1.16.3 and a 9.2.1 toolchain

[dcantah]

Also can confirm 1.19 gets things in order again.. Believe it was this set of patches from @thanm 0c7fcf6 eaf2125

[Southclaws]

Updated to 1.19 and new error...

Build Error: go test -c -o d:\Work\odin\odin\api\src\services\deal\__debug_bin.exe -gcflags all=-N -l -v -race .
# runtime/cgo
In file included from c:\program files\x86_64-w64-mingw32-native\lib\gcc\x86_64-w64-mingw32\11.2.1\include-fixed\limits.h:34,
                 from c:\program files\x86_64-w64-mingw32-native\include\stdlib.h:11,
                 from _cgo_export.c:3:
c:\program files\x86_64-w64-mingw32-native\include\syslimits.h:12:25: error: no include path in which to search for limits.h
   12 | #include_next <limits.h>
      |                         ^ (exit status 2)

I just want to debug 1 test 😭

Edit: if I run the command manually, apparently it's not even valid:

❯ go test -c -o d:\Work\odin\odin\api\src\services\deal\__debug_bin.exe -gcflags all=-N -l -v -race .
go: unknown flag -l cannot be used with -c

[thanm]

c:\program files\x86_64-w64-mingw32-native\include\syslimits.h:12:25: error: no include path in which to search for limits.h
12 | #include_next <limits.h>
| ^ (exit status 2)

I think this is getting a bit farther afield from the original issue ("ThreadSanitizer failed to allocate"). Seems like maybe something is out of whack with your gcc installation.

❯ go test -c -o d:\Work\odin\odin\api\src\services\deal__debug_bin.exe -gcflags all=-N -l -v -race .
go: unknown flag -l cannot be used with -c

That looks as though the "-l" is being interpreted by the Go command and not passed to the compiler. I would try fixing up your quoting, e.g. "-gcflags all=-N -l" etc.

[Southclaws]

Seems like maybe something is out of whack with your gcc installation.

I forgot this even used gcc haha I'm not sure which version I have installed, I always assumed Go on Windows used the native Microsoft compiler for C code.

I would try fixing up your quoting

none of these are commands I've written, these are just what comes out when I click "debug test" in vscode

I'll experiment a bit more next week, this was working fine for literal years and suddenly one day it all breaks and I can't debug any more, so annoying.

[benitogf]

Hello , I got this on 1.19

Setup go stable version spec 1.19.x
Found in cache @ C:\hostedtoolcache\windows\go\1.19.1\x64
Added go to the path
Successfully setup go version 1.19.x
go version go1.19.1 windows/amd64

go env
  set GO111MODULE=
  set GOARCH=amd64
  set GOBIN=
  set GOCACHE=C:\Users\runneradmin\AppData\Local\go-build
  set GOENV=C:\Users\runneradmin\AppData\Roaming\go\env
  set GOEXE=.exe
  set GOEXPERIMENT=
  set GOFLAGS=
  set GOHOSTARCH=amd64
  set GOHOSTOS=windows
  set GOINSECURE=
  set GOMODCACHE=C:\Users\runneradmin\go\pkg\mod
  set GONOPROXY=
  set GONOSUMDB=
  set GOOS=windows
  set GOPATH=C:\Users\runneradmin\go
  set GOPRIVATE=
  set GOPROXY=https://proxy.golang.org,direct
  set GOROOT=C:\hostedtoolcache\windows\go\1.19.1\x64
  set GOSUMDB=sum.golang.org
  set GOTMPDIR=
  set GOTOOLDIR=C:\hostedtoolcache\windows\go\1.19.1\x64\pkg\tool\windows_amd64
  set GOVCS=
  set GOVERSION=go1.19.1
  set GCCGO=gccgo
  set GOAMD64=v1
  set AR=ar
  set CC=gcc
  set CXX=g++
  set CGO_ENABLED=1
  set GOMOD=NUL
  set GOWORK=
  set CGO_CFLAGS=-g -O2
  set CGO_CPPFLAGS=
  set CGO_CXXFLAGS=-g -O2
  set CGO_FFLAGS=-g -O2
  set CGO_LDFLAGS=-g -O2
  set PKG_CONFIG=pkg-config
  set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\RUNNER~1\AppData\Local\Temp\go-build372[8](https://github.com/benitogf/level/actions/runs/3118570570/jobs/5057948602#step:3:9)52[9](https://github.com/benitogf/level/actions/runs/3118570570/jobs/5057948602#step:3:10)3[11](https://github.com/benitogf/level/actions/runs/3118570570/jobs/5057948602#step:3:12)=/tmp/go-build -gno-record-gcc-switches

==1188==ERROR: ThreadSanitizer failed to allocate 0x000100000000 (4294967296) bytes at 0x040301000000 (error code: 1455)
exit status 66

[dvyukov]

@benitogf you are out of memory:

1455 | The paging file is too small for this operation to complete. | ERROR_COMMITMENT_LIMIT

[benitogf]

got it, thanks @dvyukov 😅

[alexbrainman]

@benitogf you are out of memory:

@dvyukov would it be helpful to display Windows error message as well as error code in ThreadSanitizer error message? I suspect some users that get to see ThreadSanitizer error messages might not know how to lookup these Windows errors by their code.

Alex

[dvyukov]

Probably yes.