gclient sync ran on WebRTC repository fails with [Err: panic: failed to execve "python": cannot allocate memory

[mstyura]

Issue

Unable to properly fetch WebRTC repository under WSL, because python invocation fails with memory allocation error.
The steps to reproduce mostly follows this WebRTC manual https://webrtc.org/native-code/development/

Environment details:

>ver

Microsoft Windows [Version 10.0.17093.1000]

> lsb_release -a:
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

➜  webrtc cat /proc/meminfo
MemTotal:       33491872 kB
MemFree:        21947188 kB
Buffers:           34032 kB
Cached:           188576 kB
SwapCached:            0 kB
Active:           167556 kB
Inactive:         157876 kB
Active(anon):     103104 kB
Inactive(anon):    17440 kB
Active(file):      64452 kB
Inactive(file):   140436 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      41938300 kB
SwapFree:       41925628 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        102824 kB
Mapped:            71404 kB
Shmem:             17720 kB
Slab:              13868 kB
SReclaimable:       6744 kB
SUnreclaim:         7124 kB
KernelStack:        2848 kB
PageTables:         2524 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      515524 kB
Committed_AS:    3450064 kB
VmallocTotal:     122880 kB
VmallocUsed:       21296 kB
VmallocChunk:      66044 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       12280 kB
DirectMap4M:      897024 kB

Steps to reproduce

0. Choose desired working (in my case it looks like this)

➜  googlesource.com pwd
/home/mstyura/Projects/googlesource.com

# in my case Projects directory is symlink to folder on NTFS drive
➜  ~ ls -l
total 0
lrwxrwxrwx 1 mstyura mstyura 15 Feb 17 21:24 Projects -> /mnt/e/Projects

# 
➜  ~ mount -l
rootfs on / type lxfs (rw,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
none on /dev type tmpfs (rw,noatime,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,noatime,gid=5,mode=620)
none on /run type tmpfs (rw,nosuid,noexec,noatime,mode=755)
none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,noatime)
none on /run/shm type tmpfs (rw,nosuid,nodev,noatime)
none on /run/user type tmpfs (rw,nosuid,nodev,noexec,noatime,mode=755)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noatime)
C: on /mnt/c type drvfs (rw,noatime,uid=1000,gid=1000)
D: on /mnt/d type drvfs (rw,noatime,uid=1000,gid=1000)
E: on /mnt/e type drvfs (rw,noatime,uid=1000,gid=1000)
F: on /mnt/f type drvfs (rw,noatime,uid=1000,gid=1000)

1. Download depot tools by

 git clone https://chromium.googlesource.com/chromium/tools/depot_tools

2. Add depot tools to path

export PATH=$(pwd)/depot_tools:$PATH

3. Create directory for WebRTC repository

mkdir webrtc && cd webrtc

4. Run downloading of webrtc repository (on my environment it took about ~1hour)

fetch webrtc

Expected behavior

fetch webrtc

should succeed, but it failed when repository hooks were run

Actual behavior

Since fetch webrtc failed when it was running hooks, the repository is already there and to reproduce the issue again and again I just need to run gclient sync inside downloaded repository

➜  webrtc pwd
/home/mstyura/Projects/googlesource.com/webrtc
➜  webrtc ls
src
➜  webrtc cd src
➜  src git:(7c72c10) gclient sync
#long output skipped
5> Failed to fetch file gs://chromium-webrtc-resources/d19fb2879d091324c699987ab280da82af200933 for src/resources/paris_qcif.yuv, skipping. [Err: panic: failed to execve "/home/mstyura/.vpython-root/9eab6c/bin/python": cannot allocate memory
#long output skipped

strace -ff gclient sync produced thousands of files, some of them has failures found by ag "cannot allocate memory"
Attaching this files which contains memory allocation error: https://www.dropbox.com/s/iv6idf660s7mhrx/webrtc.zip?dl=0

[therealkenc]

Confirmed the OOM here on 17101. I did this on LxFs with Defender real time protection disabled (because sanity) so those aren't variables. Also tried fetch chromium (all 8.5 GiB and a couple hours later worth) and it has the same outcome, naturally. For me webrtc failed with an OOM trying to spawn Python at:

subprocess.CalledProcessError: Command '['download_from_google_storage', '--no_resume', '--no_auth', '--bucket', 'chromium-binutils', '-s', '/home/ken/Devel/webrtc/src/third_party/binutils/Linux_x64/binutils.tar.bz2.sha1'

The exact failure point isn’t likely important (with Chrome it failed on download_nacl_toolchains). This is "new" AFAIK, either because of a change in NT/WSL or a change in Google's build process. Chromium (which is webrtc plus plus more) has built on WSL for a long time. It is a little curious because in theory memory management has got better not worse in WSL over time.

The scary quotes on "new" is because, tellingly, while I have built Chromium under WSL successfully in the past, I don't particularly recommended it, and have not done so recently. LxFs performance (let alone DrvFs performance) is such that you’ll find a happier experience in a VM. If you are hell bent on compiling webrtc on WSL, you could work-around by doing the fetch/gclient sync in a VM and then copy the thing over. Which is in reality how I was doing things here, for example.

[Bhlowe]

Also experiencing. Have 32GB physical RAM.
Initial search shows it may be related issue #92
Upgraded to Windows Insider Preview 170120 without luck.

[u382514]

Same issue here. Any news on this?

[tschelik]

Same issue. Running Ubuntu 16.04 on a machine with 16GB of physical RAM and having Ubuntu hit the limit at 4,64GB.
This is sooo not cool.

[coding-pandaren]

windows_version:Microsoft Windows [Version 10.0.16299.334]
Have 32GB physical RAM.
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

typing command gclient sync --reset --with_branch_heads the terminal print
Running hooks: 50% (27/54) luci-go_linux
________ running '/usr/bin/python src/third_party/depot_tools/download_from_google_storage.py --no_resume --no_auth --bucket chromium-luci -d src/tools/luci-go/linux64' in '/mnt/f/android/chromium'
panic: failed to execve "/home/daniel/.vpython-root/758a9e/bin/python": cannot allocate memory

free command show subsytem memory 3.2G~can i make the memory larger?
daniel@DESKTOP-7K3VVG7:/mnt/f/android/chromium/src$ free -m
total used free shared buff/cache available
Mem: 32645 14634 17780 17 230 17873
Swap: 29738 307 29431

[artit-io]

Same issue Windows Pro 1803

Microsoft Windows [Version 10.0.17134.407]

[RReverser]

Yeah same on 1809.

[artit-io]

I would be satisfied with a vcpkg too so I would not have to build it manually.

I don't know the process of making one though.

[RReverser]

For now, while it's rather inconvenient, I managed to make builds work by manually copying each failed download_from_google_storage command into Windows shell and running it there but without --platform linux* part.

Luckily, depot_tools are cross-platform by design, and there were very few such commands (I suppose just on large files), so it worked without further issues and now V8 compiles inside of WSL too.

[liaochongliang]

Hello everyone, I have also encountered this problem.
Unable to compile chromeium source code, exactly that running gclient runhooks failed.
Is this a problem with chromeium or a problem with wsl?

[mstyura]

@liaochongliang it is a problem within WSL (at least was at the time when I've opened this issue).

[ijustlovemath]

I'm seeing this issue as well, and apparently it's been seen in the wild in Russian versions of WSL: https://toster.ru/q/589833

[markuspeloquin]

Just an update, this should be fixed in Windows 10 Insider Preview Build 18342 (as mentioned in #3800). This will be included in the Windows 10 19H1 update, the codename for the 2019 April May update.

I've reproduced the bug myself by calling pthread_create and having that thread call one of the exec functions. The exec call fails and sets errno = ENOMEM for bizarre reasons. This is what is being fixed.

Also, another workaround besides #2959 (comment) is to just re-run the failing command in a loop. Because of the randomness of which Python thread calls exec, it occasionally works:

while ! download_from_google_storage [...] ; do true; done

[therealkenc]

Thanks! I don't have it in me to do a Chrome build to confirm. Not sure what to do with this one. Call it fixedininsiders until proven otherwise? The population of users doing Chrome and/or WebRTC builds on WSL is, let's call it, "small". But AFAICT this is #3800 (comment).

[bb33bb]

same issue with all newest patch

[therealkenc]

panic: failed to execve "/home/daniel/.vpython-root/758a9e/bin/python": cannot allocate memory

Whether the error above was #3800 or otherwise, it will not manifest in WSL2.

[miron]

Not fixed for me, in docker container:
grep MemTotal /proc/meminfo
MemTotal: 32813988 kB

I have 15x that memory free, build is newest with 21359 insider, of course wsl2

[therealkenc]

Not fixed for me

If you are experiencing panic: failed to execve "/home/daniel/.vpython-root/758a9e/bin/python": cannot allocate memory on WSL2 (username notwithstanding), that can be filed until a new issue with the CLI repro steps.

[miron]

That fixed it for me:
#5240 (comment)