System.Net.Requests.Tests failure "Error while reaping child. errno = 10" on release mono Linux x64 Debug failed #55618

eerhardt · 2021-07-14T02:17:10Z

https://github.com/dotnet/runtime/pull/55531/checks?check_run_id=3062575970

https://helix.dot.net/api/2019-06-17/jobs/119fb1a9-e776-4c48-9112-8bbe06cb5da2/workitems/System.Net.Requests.Tests/console

/root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.Net.Requests.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Requests.Tests (found 335 of 353 test cases)
  Starting:    System.Net.Requests.Tests (parallel test collections = on, max threads = 2)
Process terminated due to "Error while reaping child. errno = 10"

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7f7caceff070 - Unknown
	0x7f7cacea025e - Unknown
	0x7f7cacefe91a - Unknown
	0x7f7cad684a90 - Unknown
	0x7f7cad18b7d5 - Unknown
	0x7f7cad174895 - Unknown
	0x7f7cacce8cec - Unknown
	0x7f7cacceb8d4 - Unknown
	0x403480e6 - Unknown

=================================================================
	Telemetry Dumper:
=================================================================
Pkilling 0x140173299336960x from 0x140173299967744x
Pkilling 0x140173265745664x from 0x140173299967744x
Pkilling 0x140173276849920x from 0x140173299967744x
Pkilling 0x140173413672704x from 0x140173299967744x
Pkilling 0x140173279328000x from 0x140173299967744x
Pkilling 0x140173297235712x from 0x140173299967744x
Pkilling 0x140173268027136x from 0x140173299967744x
Pkilling 0x140173456484160x from 0x140173299967744x
Pkilling 0x140173279061760x from 0x140173299967744x
Pkilling 0x140173274748672x from 0x140173299967744x
Pkilling 0x140173415773952x from 0x140173299967744x
Could not exec mono-hang-watchdog, expected on path '/__w/1/s/artifacts/obj/mono/Linux.x64.Release/out/etc/../bin/mono-hang-watchdog' (errno 2)
Entering thread summarizer pause from 0x140173299967744x
Finished thread summarizer pause from 0x140173299967744x.
Failed to create breadcrumb file (null)/crash_hash_0x3ed9dea04

Waiting for dumping threads to resume

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 23]
[New LWP 24]
[New LWP 25]
[New LWP 26]
[New LWP 29]
[New LWP 30]
[New LWP 31]
[New LWP 32]
[New LWP 33]
[New LWP 34]
[New LWP 35]
[New LWP 40]
[New LWP 44]
[New LWP 45]
[New LWP 46]
[New LWP 47]
Missing separate debuginfo for /root/helix/work/correlation/dotnet
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/21/5377aac80b8844e91be30e731e0d61fd251357.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /root/helix/work/correlation/host/fxr/6.0.0/libhostfxr.so
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/8f/b71c0443de36a295bb26306ac9004d1039c342.debug
Missing separate debuginfo for /root/helix/work/correlation/shared/Microsoft.NETCore.App/6.0.0/libhostpolicy.so
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/ff/ad32b7b2e53f6344302b296149f9686c92e646.debug
0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  Id   Target Id                                        Frame 
* 1    Thread 0x7f7cad14a740 (LWP 22) "dotnet"          0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2    Thread 0x7f7cac7ff700 (LWP 23) "SGen worker"     0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3    Thread 0x7f7caaa77700 (LWP 24) "dotnet"          0x00007f7cad24586f in poll () from /lib64/libc.so.6
  4    Thread 0x7f7caa876700 (LWP 25) "Finalizer"       0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
  5    Thread 0x7f7ca3c06700 (LWP 26) "dotnet"          0x00007f7cad21cccf in wait4 () from /lib64/libc.so.6
  6    Thread 0x7f7ca396b700 (LWP 29) ".NET Long Runni" 0x00007f7cad683750 in __lll_lock_wait () from /lib64/libpthread.so.0
  7    Thread 0x7f7ca3b6c700 (LWP 30) ".NET ThreadPool" 0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8    Thread 0x7f7ca2857700 (LWP 31) ".NET ThreadPool" 0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
  9    Thread 0x7f7ca2816700 (LWP 32) ".NET ThreadPool" 0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10   Thread 0x7f7ca25fa700 (LWP 33) ".NET Long Runni" 0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
  11   Thread 0x7f7ca23f9700 (LWP 34) ".NET Long Runni" 0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  12   Thread 0x7f7ca1d90700 (LWP 35) ".NET Sockets"    0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
  13   Thread 0x7f7ca1b63700 (LWP 40) ".NET Timers"     0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  14   Thread 0x7f7c77fff700 (LWP 44) ".NET Long Runni" 0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
  15   Thread 0x7f7c777fe700 (LWP 45) ".NET Long Runni" 0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
  16   Thread 0x7f7c76ffd700 (LWP 46) ".NET Long Runni" 0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  17   Thread 0x7f7c767fc700 (LWP 47) ".NET Long Runni" 0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

Thread 17 (Thread 0x7f7c767fc700 (LWP 47)):
#0  0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7ca147f123 in QuicWorkerThread () from /lib64/libmsquic.so
#2  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f7c76ffd700 (LWP 46)):
#0  0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7ca147f123 in QuicWorkerThread () from /lib64/libmsquic.so
#2  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f7c777fe700 (LWP 45)):
#0  0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f7ca14c72d1 in CxPlatDataPathWorkerThread () from /lib64/libmsquic.so
#2  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f7c77fff700 (LWP 44)):
#0  0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f7ca14c72d1 in CxPlatDataPathWorkerThread () from /lib64/libmsquic.so
#2  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f7ca1b63700 (LWP 40)):
#0  0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7caa4316d0 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f7c7c013730, timeoutMilliseconds=14888) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_threading.c:192
#2  0x0000000040287417 in ?? ()
#3  0x000000000042466c in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f7ca1d90700 (LWP 35)):
#0  0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f7caa42c64e in WaitForSocketEventsInner (port=10, buffer=0x7f7c8013bb00, count=0x7f7ca1d8fb10) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_networking.c:2717
#2  0x00007f7caa42c56f in SystemNative_WaitForSocketEvents (port=10, buffer=0x7f7c8013bb00, count=0x7f7ca1d8fb10) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_networking.c:3017
#3  0x0000000040319969 in ?? ()
#4  0x0000000000000001 in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f7ca23f9700 (LWP 34)):
#0  0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7caa4316d0 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f7c84002730, timeoutMilliseconds=30000) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_threading.c:192
#2  0x0000000040287417 in ?? ()
#3  0x00000000004242c8 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7f7ca25fa700 (LWP 33)):
#0  0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007f7cad682b28 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f7cacd9ecd8 in mono_os_sem_wait (sem=<optimized out>, flags=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
#3  mono_os_sem_timedwait (sem=0x7f7cad071ca0 <suspend_semaphore>, timeout_ms=4294967295, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:237
#4  0x00007f7cacd9e9e6 in mono_threads_wait_pending_operations () at /__w/1/s/src/mono/mono/utils/mono-threads.c:322
#5  0x00007f7cacd707cc in unified_suspend_stop_world (flags=MONO_THREAD_INFO_FLAGS_NO_GC, thread_stopped_callback=0x7f7cacd70c60 <sgen_client_stop_world_thread_stopped_callback>) at /__w/1/s/src/mono/mono/metadata/sgen-stw.c:343
#6  0x00007f7cacd704c0 in sgen_client_stop_world (generation=0, serial_collection=0) at /__w/1/s/src/mono/mono/metadata/sgen-stw.c:153
#7  0x00007f7cacdb828d in sgen_stop_world (generation=0, serial_collection=0) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:3997
#8  0x00007f7cacdb4c9f in sgen_perform_collection_inner (requested_size=<optimized out>, generation_to_collect=<optimized out>, reason=<optimized out>, forced_serial=<optimized out>, stw=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2643
#9  sgen_perform_collection (requested_size=4096, generation_to_collect=0, reason=0x7f7cacfb62a7 "Nursery full", forced_serial=0, stw=1) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2768
#10 0x00007f7cacdb4c03 in sgen_ensure_free_space (size=4096, generation=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2622
#11 0x00007f7cacda8dc7 in sgen_alloc_obj_nolock (vtable=0x7f7c84067a40, size=40) at /__w/1/s/src/mono/mono/sgen/sgen-alloc.c:279
#12 0x00007f7cacda92a9 in sgen_alloc_obj (vtable=0x7f7c84067a40, size=40) at /__w/1/s/src/mono/mono/sgen/sgen-alloc.c:454
#13 0x00007f7cacd72436 in mono_gc_alloc_obj (vtable=0x7f7cad071ca0 <suspend_semaphore>, size=393) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:911
#14 0x000000004016d528 in ?? ()
#15 0x00007f7cac970bc0 in ?? ()
#16 0x00007f7c800011d0 in ?? ()
#17 0x00007f7c80000b60 in ?? ()
#18 0x00007f7c84067a40 in ?? ()
#19 0x0000000000000028 in ?? ()
#20 0x00007f7c80002490 in ?? ()
#21 0x00007f7cac970be8 in ?? ()
#22 0x00007f7ca25f7990 in ?? ()
#23 0x00007f7caca6b5c0 in ?? ()
#24 0x000000004014ad44 in ?? ()
#25 0xffffffffffffffff in ?? ()
#26 0x00007f7ca25f7b00 in ?? ()
#27 0x00007f7cacbc3870 in ?? ()
#28 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f7ca2816700 (LWP 32)):
#0  0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7cacd95564 in mono_os_cond_timedwait (cond=0x7f7ca28158c0, mutex=0x7f7c940413c0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#2  0x00007f7cacd9c8f4 in mono_coop_cond_timedwait (cond=0x7f7ca28158c0, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#3  mono_lifo_semaphore_timed_wait (semaphore=0x7f7c940413c0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#4  0x000000004028e9d7 in ?? ()
#5  0x0000000000000002 in ?? ()
#6  0x0000000000000046 in ?? ()
#7  0x00007f7cac973618 in ?? ()
#8  0x00007f7cac973618 in ?? ()
#9  0x0000000000004e20 in ?? ()
#10 0x00007f7c8c002490 in ?? ()
#11 0x0000000000000046 in ?? ()
#12 0x00007f7ca2815930 in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f7ca2857700 (LWP 31)):
#0  0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007f7cad682b28 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f7cacd9e838 in mono_os_sem_wait (sem=0x7f7c88000bc0, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
#3  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
#4  0x00007f7cacda4a2a in mono_threads_exit_gc_safe_region_unbalanced_internal (cookie=0x7f7c88000b60, stackdata=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:390
#5  0x00007f7cacda4bbf in mono_threads_exit_gc_safe_region_unbalanced (cookie=0x7f7c88000bc0, stackpointer=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:410
#6  0x0000000040287443 in ?? ()
#7  0x0000000000424618 in ?? ()
#8  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f7ca3b6c700 (LWP 30)):
#0  0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7caa4316d0 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f7c900e4ed0, timeoutMilliseconds=12000) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_threading.c:192
#2  0x0000000040287417 in ?? ()
#3  0x000000000042421c in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f7ca396b700 (LWP 29)):
#0  0x00007f7cad683750 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f7cad67be53 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f7cacdb5629 in mono_os_mutex_lock (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:105
#3  mono_coop_mutex_lock (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:57
#4  sgen_gc_lock () at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:3939
#5  0x00007f7cacda929e in sgen_alloc_obj (vtable=0x56546582e098, size=128) at /__w/1/s/src/mono/mono/sgen/sgen-alloc.c:453
#6  0x00007f7cacd72436 in mono_gc_alloc_obj (vtable=0x7f7cad083368 <sgen_gc_mutex>, size=128) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:911
#7  0x000000004016d528 in ?? ()
#8  0x00007f7cac801a40 in ?? ()
#9  0x00007f7c94005230 in ?? ()
#10 0x00007f7c94004bc0 in ?? ()
#11 0x000056546582e098 in ?? ()
#12 0x0000000000000080 in ?? ()
#13 0x00007f7c94024320 in ?? ()
#14 0x00007f7cac801ac0 in ?? ()
#15 0x00007f7ca396a7a0 in ?? ()
#16 0x00007f7caca75360 in ?? ()
#17 0x000000004014ad44 in ?? ()
#18 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f7ca3c06700 (LWP 26)):
#0  0x00007f7cad21cccf in wait4 () from /lib64/libc.so.6
#1  0x00007f7caceff2ca in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:981
#2  mono_dump_native_crash_info (signal=0x7f7cacfb4c6b "SIGABRT", mctx=0x7f7ca3c05158, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:1025
#3  0x00007f7cacea025e in mono_handle_native_crash (signal=0x7f7cacfb4c6b "SIGABRT", mctx=0x7f7ca3c05158, info=0x7f7ca3c05430) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3382
#4  0x00007f7cacefe91a in sigabrt_signal_handler (_dummy=6, _info=0x7f7ca3c05430, context=0x7f7ca3c05300) at /__w/1/s/src/mono/mono/mini/mini-posix.c:227
#5  <signal handler called>
#6  0x00007f7cad18b7d5 in raise () from /lib64/libc.so.6
#7  0x00007f7cad174895 in abort () from /lib64/libc.so.6
#8  0x00007f7cacce8cec in ves_icall_System_Environment_FailFast (message=..., exception=..., errorSource=..., error=<optimized out>) at /__w/1/s/src/mono/mono/metadata/icall.c:6649
#9  0x00007f7cacceb8d4 in ves_icall_System_Environment_FailFast_raw (a0=0x7f7ca3c05bb0, a1=0x7f7ca3c05bb8, a2=<optimized out>) at /__w/1/s/src/mono/mono/metadata/icall-def-netcore.h:99
#10 0x00000000403480e6 in ?? ()
#11 0x0000000000000001 in ?? ()
#12 0x000000000000002a in ?? ()
#13 0x00000000ffffffff in ?? ()
#14 0x00007f7cacabc6f0 in ?? ()
#15 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f7caa876700 (LWP 25)):
#0  0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007f7cad682b28 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f7cacd62b66 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#3  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#4  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:876
#5  0x00007f7cacd3a3ca in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1201
#6  0x00007f7cacd3a259 in start_wrapper (data=0x56546516ad40) at /__w/1/s/src/mono/mono/metadata/threads.c:1263
#7  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f7caaa77700 (LWP 24)):
#0  0x00007f7cad24586f in poll () from /lib64/libc.so.6
#1  0x00007f7cacf9632a in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:453
#2  ds_ipc_poll (poll_handles_data=0x7f7ca4002570, poll_handles_data_len=1, timeout_ms=4294967295, callback=0x7f7cacf95820 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1065
#3  0x00007f7cacf939a5 in ds_ipc_stream_factory_get_next_available_stream (callback=0x7f7cacf95820 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#4  0x00007f7cacf92209 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#5  0x00007f7cacf95801 in ep_rt_thread_mono_start_func (data=0x565465149880) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1248
#6  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f7cac7ff700 (LWP 23)):
#0  0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7cacdec1c3 in mono_os_cond_wait (cond=<optimized out>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#2  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#3  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#4  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f7cad14a740 (LWP 22)):
#0  0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7caa43158c in SystemNative_LowLevelMonitor_Wait (monitor=0x5654658709b0) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_threading.c:152
#2  0x000000004022e87c in ?? ()
#3  0x00007f7cac967cf8 in ?? ()
#4  0x00007f7ca2ae81d0 in ?? ()
#5  0xffffffffffffffff in ?? ()
#6  0x0000000000000001 in ?? ()
#7  0x00007f7ca2ae8200 in ?? ()
#8  0x00007f7ca2ae81d0 in ?? ()
#9  0xffffffffffffffff in ?? ()
#10 0x0000565465146850 in ?? ()
#11 0x00007ffc9b2dcad0 in ?? ()
#12 0x00007ffc9b2dc9e0 in ?? ()
#13 0x00005654650a1dc0 in ?? ()
#14 0x000000004022e6ac in ?? ()
#15 0xffffffffffffffff in ?? ()
#16 0x000000004022e66c in ?? ()
#17 0x00007f7ca2ae8200 in ?? ()
#18 0x000000004022dd58 in ?? ()
#19 0x00007f7cacfa0d25 in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/6.0.0/libcoreclr.so
#20 0x0000000000000000 in ?? ()
[Inferior 1 (process 22) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7f7cad18b7d5):0x7f7cad18b7c5  d2 4c 89 ce bf 02 00 00 00 b8 0e 00 00 00 0f 05  .L..............
0x7f7cad18b7d5  48 8b 84 24 08 01 00 00 64 48 2b 04 25 28 00 00  H..$....dH+.%(..
0x7f7cad18b7e5  00 75 24 44 89 c0 48 81 c4 18 01 00 00 c3 0f 1f  .u$D..H.........
0x7f7cad18b7f5  44 00 00 48 8b 15 79 66 18 00 f7 d8 41 b8 ff ff  D..H..yf....A...

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Environment:FailFast <0x00095>
	  at System.Environment:FailFast <0x0002f>
	  at System.Diagnostics.ProcessWaitState:TryReapChild <0x0021b>
	  at System.Diagnostics.ProcessWaitState:CheckChildren <0x0030b>
	  at System.Diagnostics.Process:OnSigChild <0x0006f>
	  at System.Diagnostics.Process:OnSigChild <0x00067>
=================================================================
./RunTests.sh: line 162:    22 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Wed Jul 14 01:09:29 UTC 2021 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
ulimit -c value: unlimited
[ 4229.260421] docker0: port 1(veth2106719) entered blocking state
[ 4229.260422] docker0: port 1(veth2106719) entered forwarding state
[ 4245.736835] docker0: port 1(veth2106719) entered disabled state
[ 4245.736868] vethb7155d4: renamed from eth0
[ 4245.847115] docker0: port 1(veth2106719) entered disabled state
[ 4245.847986] device veth2106719 left promiscuous mode
[ 4245.847988] docker0: port 1(veth2106719) entered disabled state
[ 4252.578039] docker0: port 1(veth34113d2) entered blocking state
[ 4252.578042] docker0: port 1(veth34113d2) entered disabled state
[ 4252.578084] device veth34113d2 entered promiscuous mode
[ 4252.578138] IPv6: ADDRCONF(NETDEV_UP): veth34113d2: link is not ready
[ 4252.904234] eth0: renamed from veth3c9b01b
[ 4252.951089] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4252.953234] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4252.953244] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 4252.953263] IPv6: ADDRCONF(NETDEV_CHANGE): veth34113d2: link becomes ready
[ 4252.953280] docker0: port 1(veth34113d2) entered blocking state
[ 4252.953282] docker0: port 1(veth34113d2) entered forwarding state
[ 4274.783972] docker0: port 1(veth34113d2) entered disabled state
[ 4274.784046] veth3c9b01b: renamed from eth0
[ 4274.877240] docker0: port 1(veth34113d2) entered disabled state
[ 4274.878115] device veth34113d2 left promiscuous mode
[ 4274.878117] docker0: port 1(veth34113d2) entered disabled state
[ 4286.223521] docker0: port 1(veth0ea6c09) entered blocking state
[ 4286.223523] docker0: port 1(veth0ea6c09) entered disabled state
[ 4286.223564] device veth0ea6c09 entered promiscuous mode
[ 4286.223621] IPv6: ADDRCONF(NETDEV_UP): veth0ea6c09: link is not ready
[ 4286.576326] eth0: renamed from veth30f8c3c
[ 4286.623592] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4286.625509] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4286.625520] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 4286.625535] IPv6: ADDRCONF(NETDEV_CHANGE): veth0ea6c09: link becomes ready
[ 4286.625551] docker0: port 1(veth0ea6c09) entered blocking state
[ 4286.625553] docker0: port 1(veth0ea6c09) entered forwarding state
[ 4304.387500] docker0: port 1(veth0ea6c09) entered disabled state
[ 4304.387555] veth30f8c3c: renamed from eth0
[ 4304.500520] docker0: port 1(veth0ea6c09) entered disabled state
[ 4304.501432] device veth0ea6c09 left promiscuous mode
[ 4304.501434] docker0: port 1(veth0ea6c09) entered disabled state
[ 4340.697125] docker0: port 1(veth8484c1c) entered blocking state
[ 4340.697127] docker0: port 1(veth8484c1c) entered disabled state
[ 4340.697166] device veth8484c1c entered promiscuous mode
[ 4340.697217] IPv6: ADDRCONF(NETDEV_UP): veth8484c1c: link is not ready
[ 4341.148288] eth0: renamed from veth3ef6170
[ 4341.202127] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4341.203994] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4341.204028] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 4341.204045] IPv6: ADDRCONF(NETDEV_CHANGE): veth8484c1c: link becomes ready
[ 4341.204062] docker0: port 1(veth8484c1c) entered blocking state
[ 4341.204063] docker0: port 1(veth8484c1c) entered forwarding state

The text was updated successfully, but these errors were encountered:

ghost · 2021-07-14T02:17:13Z

Tagging subscribers to this area:
See info in area-owners.md if you want to be subscribed.

Issue Details

https://github.com/dotnet/runtime/pull/55531/checks?check_run_id=3062575970

https://helix.dot.net/api/2019-06-17/jobs/119fb1a9-e776-4c48-9112-8bbe06cb5da2/workitems/System.Net.Requests.Tests/console

/root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.Net.Requests.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Requests.Tests (found 335 of 353 test cases)
  Starting:    System.Net.Requests.Tests (parallel test collections = on, max threads = 2)
Process terminated due to "Error while reaping child. errno = 10"

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7f7caceff070 - Unknown
	0x7f7cacea025e - Unknown
	0x7f7cacefe91a - Unknown
	0x7f7cad684a90 - Unknown
	0x7f7cad18b7d5 - Unknown
	0x7f7cad174895 - Unknown
	0x7f7cacce8cec - Unknown
	0x7f7cacceb8d4 - Unknown
	0x403480e6 - Unknown

=================================================================
	Telemetry Dumper:
=================================================================
Pkilling 0x140173299336960x from 0x140173299967744x
Pkilling 0x140173265745664x from 0x140173299967744x
Pkilling 0x140173276849920x from 0x140173299967744x
Pkilling 0x140173413672704x from 0x140173299967744x
Pkilling 0x140173279328000x from 0x140173299967744x
Pkilling 0x140173297235712x from 0x140173299967744x
Pkilling 0x140173268027136x from 0x140173299967744x
Pkilling 0x140173456484160x from 0x140173299967744x
Pkilling 0x140173279061760x from 0x140173299967744x
Pkilling 0x140173274748672x from 0x140173299967744x
Pkilling 0x140173415773952x from 0x140173299967744x
Could not exec mono-hang-watchdog, expected on path '/__w/1/s/artifacts/obj/mono/Linux.x64.Release/out/etc/../bin/mono-hang-watchdog' (errno 2)
Entering thread summarizer pause from 0x140173299967744x
Finished thread summarizer pause from 0x140173299967744x.
Failed to create breadcrumb file (null)/crash_hash_0x3ed9dea04

Waiting for dumping threads to resume

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 23]
[New LWP 24]
[New LWP 25]
[New LWP 26]
[New LWP 29]
[New LWP 30]
[New LWP 31]
[New LWP 32]
[New LWP 33]
[New LWP 34]
[New LWP 35]
[New LWP 40]
[New LWP 44]
[New LWP 45]
[New LWP 46]
[New LWP 47]
Missing separate debuginfo for /root/helix/work/correlation/dotnet
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/21/5377aac80b8844e91be30e731e0d61fd251357.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /root/helix/work/correlation/host/fxr/6.0.0/libhostfxr.so
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/8f/b71c0443de36a295bb26306ac9004d1039c342.debug
Missing separate debuginfo for /root/helix/work/correlation/shared/Microsoft.NETCore.App/6.0.0/libhostpolicy.so
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/ff/ad32b7b2e53f6344302b296149f9686c92e646.debug
0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  Id   Target Id                                        Frame 
* 1    Thread 0x7f7cad14a740 (LWP 22) "dotnet"          0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2    Thread 0x7f7cac7ff700 (LWP 23) "SGen worker"     0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3    Thread 0x7f7caaa77700 (LWP 24) "dotnet"          0x00007f7cad24586f in poll () from /lib64/libc.so.6
  4    Thread 0x7f7caa876700 (LWP 25) "Finalizer"       0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
  5    Thread 0x7f7ca3c06700 (LWP 26) "dotnet"          0x00007f7cad21cccf in wait4 () from /lib64/libc.so.6
  6    Thread 0x7f7ca396b700 (LWP 29) ".NET Long Runni" 0x00007f7cad683750 in __lll_lock_wait () from /lib64/libpthread.so.0
  7    Thread 0x7f7ca3b6c700 (LWP 30) ".NET ThreadPool" 0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8    Thread 0x7f7ca2857700 (LWP 31) ".NET ThreadPool" 0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
  9    Thread 0x7f7ca2816700 (LWP 32) ".NET ThreadPool" 0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10   Thread 0x7f7ca25fa700 (LWP 33) ".NET Long Runni" 0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
  11   Thread 0x7f7ca23f9700 (LWP 34) ".NET Long Runni" 0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  12   Thread 0x7f7ca1d90700 (LWP 35) ".NET Sockets"    0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
  13   Thread 0x7f7ca1b63700 (LWP 40) ".NET Timers"     0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  14   Thread 0x7f7c77fff700 (LWP 44) ".NET Long Runni" 0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
  15   Thread 0x7f7c777fe700 (LWP 45) ".NET Long Runni" 0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
  16   Thread 0x7f7c76ffd700 (LWP 46) ".NET Long Runni" 0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  17   Thread 0x7f7c767fc700 (LWP 47) ".NET Long Runni" 0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

Thread 17 (Thread 0x7f7c767fc700 (LWP 47)):
#0  0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7ca147f123 in QuicWorkerThread () from /lib64/libmsquic.so
#2  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f7c76ffd700 (LWP 46)):
#0  0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7ca147f123 in QuicWorkerThread () from /lib64/libmsquic.so
#2  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f7c777fe700 (LWP 45)):
#0  0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f7ca14c72d1 in CxPlatDataPathWorkerThread () from /lib64/libmsquic.so
#2  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f7c77fff700 (LWP 44)):
#0  0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f7ca14c72d1 in CxPlatDataPathWorkerThread () from /lib64/libmsquic.so
#2  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f7ca1b63700 (LWP 40)):
#0  0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7caa4316d0 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f7c7c013730, timeoutMilliseconds=14888) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_threading.c:192
#2  0x0000000040287417 in ?? ()
#3  0x000000000042466c in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f7ca1d90700 (LWP 35)):
#0  0x00007f7cad250a1e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f7caa42c64e in WaitForSocketEventsInner (port=10, buffer=0x7f7c8013bb00, count=0x7f7ca1d8fb10) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_networking.c:2717
#2  0x00007f7caa42c56f in SystemNative_WaitForSocketEvents (port=10, buffer=0x7f7c8013bb00, count=0x7f7ca1d8fb10) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_networking.c:3017
#3  0x0000000040319969 in ?? ()
#4  0x0000000000000001 in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f7ca23f9700 (LWP 34)):
#0  0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7caa4316d0 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f7c84002730, timeoutMilliseconds=30000) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_threading.c:192
#2  0x0000000040287417 in ?? ()
#3  0x00000000004242c8 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7f7ca25fa700 (LWP 33)):
#0  0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007f7cad682b28 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f7cacd9ecd8 in mono_os_sem_wait (sem=<optimized out>, flags=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
#3  mono_os_sem_timedwait (sem=0x7f7cad071ca0 <suspend_semaphore>, timeout_ms=4294967295, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:237
#4  0x00007f7cacd9e9e6 in mono_threads_wait_pending_operations () at /__w/1/s/src/mono/mono/utils/mono-threads.c:322
#5  0x00007f7cacd707cc in unified_suspend_stop_world (flags=MONO_THREAD_INFO_FLAGS_NO_GC, thread_stopped_callback=0x7f7cacd70c60 <sgen_client_stop_world_thread_stopped_callback>) at /__w/1/s/src/mono/mono/metadata/sgen-stw.c:343
#6  0x00007f7cacd704c0 in sgen_client_stop_world (generation=0, serial_collection=0) at /__w/1/s/src/mono/mono/metadata/sgen-stw.c:153
#7  0x00007f7cacdb828d in sgen_stop_world (generation=0, serial_collection=0) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:3997
#8  0x00007f7cacdb4c9f in sgen_perform_collection_inner (requested_size=<optimized out>, generation_to_collect=<optimized out>, reason=<optimized out>, forced_serial=<optimized out>, stw=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2643
#9  sgen_perform_collection (requested_size=4096, generation_to_collect=0, reason=0x7f7cacfb62a7 "Nursery full", forced_serial=0, stw=1) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2768
#10 0x00007f7cacdb4c03 in sgen_ensure_free_space (size=4096, generation=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2622
#11 0x00007f7cacda8dc7 in sgen_alloc_obj_nolock (vtable=0x7f7c84067a40, size=40) at /__w/1/s/src/mono/mono/sgen/sgen-alloc.c:279
#12 0x00007f7cacda92a9 in sgen_alloc_obj (vtable=0x7f7c84067a40, size=40) at /__w/1/s/src/mono/mono/sgen/sgen-alloc.c:454
#13 0x00007f7cacd72436 in mono_gc_alloc_obj (vtable=0x7f7cad071ca0 <suspend_semaphore>, size=393) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:911
#14 0x000000004016d528 in ?? ()
#15 0x00007f7cac970bc0 in ?? ()
#16 0x00007f7c800011d0 in ?? ()
#17 0x00007f7c80000b60 in ?? ()
#18 0x00007f7c84067a40 in ?? ()
#19 0x0000000000000028 in ?? ()
#20 0x00007f7c80002490 in ?? ()
#21 0x00007f7cac970be8 in ?? ()
#22 0x00007f7ca25f7990 in ?? ()
#23 0x00007f7caca6b5c0 in ?? ()
#24 0x000000004014ad44 in ?? ()
#25 0xffffffffffffffff in ?? ()
#26 0x00007f7ca25f7b00 in ?? ()
#27 0x00007f7cacbc3870 in ?? ()
#28 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f7ca2816700 (LWP 32)):
#0  0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7cacd95564 in mono_os_cond_timedwait (cond=0x7f7ca28158c0, mutex=0x7f7c940413c0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#2  0x00007f7cacd9c8f4 in mono_coop_cond_timedwait (cond=0x7f7ca28158c0, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#3  mono_lifo_semaphore_timed_wait (semaphore=0x7f7c940413c0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#4  0x000000004028e9d7 in ?? ()
#5  0x0000000000000002 in ?? ()
#6  0x0000000000000046 in ?? ()
#7  0x00007f7cac973618 in ?? ()
#8  0x00007f7cac973618 in ?? ()
#9  0x0000000000004e20 in ?? ()
#10 0x00007f7c8c002490 in ?? ()
#11 0x0000000000000046 in ?? ()
#12 0x00007f7ca2815930 in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f7ca2857700 (LWP 31)):
#0  0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007f7cad682b28 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f7cacd9e838 in mono_os_sem_wait (sem=0x7f7c88000bc0, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
#3  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
#4  0x00007f7cacda4a2a in mono_threads_exit_gc_safe_region_unbalanced_internal (cookie=0x7f7c88000b60, stackdata=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:390
#5  0x00007f7cacda4bbf in mono_threads_exit_gc_safe_region_unbalanced (cookie=0x7f7c88000bc0, stackpointer=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:410
#6  0x0000000040287443 in ?? ()
#7  0x0000000000424618 in ?? ()
#8  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f7ca3b6c700 (LWP 30)):
#0  0x00007f7cad6801b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7caa4316d0 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f7c900e4ed0, timeoutMilliseconds=12000) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_threading.c:192
#2  0x0000000040287417 in ?? ()
#3  0x000000000042421c in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f7ca396b700 (LWP 29)):
#0  0x00007f7cad683750 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f7cad67be53 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f7cacdb5629 in mono_os_mutex_lock (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:105
#3  mono_coop_mutex_lock (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:57
#4  sgen_gc_lock () at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:3939
#5  0x00007f7cacda929e in sgen_alloc_obj (vtable=0x56546582e098, size=128) at /__w/1/s/src/mono/mono/sgen/sgen-alloc.c:453
#6  0x00007f7cacd72436 in mono_gc_alloc_obj (vtable=0x7f7cad083368 <sgen_gc_mutex>, size=128) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:911
#7  0x000000004016d528 in ?? ()
#8  0x00007f7cac801a40 in ?? ()
#9  0x00007f7c94005230 in ?? ()
#10 0x00007f7c94004bc0 in ?? ()
#11 0x000056546582e098 in ?? ()
#12 0x0000000000000080 in ?? ()
#13 0x00007f7c94024320 in ?? ()
#14 0x00007f7cac801ac0 in ?? ()
#15 0x00007f7ca396a7a0 in ?? ()
#16 0x00007f7caca75360 in ?? ()
#17 0x000000004014ad44 in ?? ()
#18 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f7ca3c06700 (LWP 26)):
#0  0x00007f7cad21cccf in wait4 () from /lib64/libc.so.6
#1  0x00007f7caceff2ca in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:981
#2  mono_dump_native_crash_info (signal=0x7f7cacfb4c6b "SIGABRT", mctx=0x7f7ca3c05158, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:1025
#3  0x00007f7cacea025e in mono_handle_native_crash (signal=0x7f7cacfb4c6b "SIGABRT", mctx=0x7f7ca3c05158, info=0x7f7ca3c05430) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3382
#4  0x00007f7cacefe91a in sigabrt_signal_handler (_dummy=6, _info=0x7f7ca3c05430, context=0x7f7ca3c05300) at /__w/1/s/src/mono/mono/mini/mini-posix.c:227
#5  <signal handler called>
#6  0x00007f7cad18b7d5 in raise () from /lib64/libc.so.6
#7  0x00007f7cad174895 in abort () from /lib64/libc.so.6
#8  0x00007f7cacce8cec in ves_icall_System_Environment_FailFast (message=..., exception=..., errorSource=..., error=<optimized out>) at /__w/1/s/src/mono/mono/metadata/icall.c:6649
#9  0x00007f7cacceb8d4 in ves_icall_System_Environment_FailFast_raw (a0=0x7f7ca3c05bb0, a1=0x7f7ca3c05bb8, a2=<optimized out>) at /__w/1/s/src/mono/mono/metadata/icall-def-netcore.h:99
#10 0x00000000403480e6 in ?? ()
#11 0x0000000000000001 in ?? ()
#12 0x000000000000002a in ?? ()
#13 0x00000000ffffffff in ?? ()
#14 0x00007f7cacabc6f0 in ?? ()
#15 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f7caa876700 (LWP 25)):
#0  0x00007f7cad682a24 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007f7cad682b28 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f7cacd62b66 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#3  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#4  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:876
#5  0x00007f7cacd3a3ca in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1201
#6  0x00007f7cacd3a259 in start_wrapper (data=0x56546516ad40) at /__w/1/s/src/mono/mono/metadata/threads.c:1263
#7  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f7caaa77700 (LWP 24)):
#0  0x00007f7cad24586f in poll () from /lib64/libc.so.6
#1  0x00007f7cacf9632a in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:453
#2  ds_ipc_poll (poll_handles_data=0x7f7ca4002570, poll_handles_data_len=1, timeout_ms=4294967295, callback=0x7f7cacf95820 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1065
#3  0x00007f7cacf939a5 in ds_ipc_stream_factory_get_next_available_stream (callback=0x7f7cacf95820 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#4  0x00007f7cacf92209 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#5  0x00007f7cacf95801 in ep_rt_thread_mono_start_func (data=0x565465149880) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1248
#6  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f7cac7ff700 (LWP 23)):
#0  0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7cacdec1c3 in mono_os_cond_wait (cond=<optimized out>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#2  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#3  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#4  0x00007f7cad679432 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7cad2506d3 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f7cad14a740 (LWP 22)):
#0  0x00007f7cad67fe92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7caa43158c in SystemNative_LowLevelMonitor_Wait (monitor=0x5654658709b0) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_threading.c:152
#2  0x000000004022e87c in ?? ()
#3  0x00007f7cac967cf8 in ?? ()
#4  0x00007f7ca2ae81d0 in ?? ()
#5  0xffffffffffffffff in ?? ()
#6  0x0000000000000001 in ?? ()
#7  0x00007f7ca2ae8200 in ?? ()
#8  0x00007f7ca2ae81d0 in ?? ()
#9  0xffffffffffffffff in ?? ()
#10 0x0000565465146850 in ?? ()
#11 0x00007ffc9b2dcad0 in ?? ()
#12 0x00007ffc9b2dc9e0 in ?? ()
#13 0x00005654650a1dc0 in ?? ()
#14 0x000000004022e6ac in ?? ()
#15 0xffffffffffffffff in ?? ()
#16 0x000000004022e66c in ?? ()
#17 0x00007f7ca2ae8200 in ?? ()
#18 0x000000004022dd58 in ?? ()
#19 0x00007f7cacfa0d25 in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/6.0.0/libcoreclr.so
#20 0x0000000000000000 in ?? ()
[Inferior 1 (process 22) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7f7cad18b7d5):0x7f7cad18b7c5  d2 4c 89 ce bf 02 00 00 00 b8 0e 00 00 00 0f 05  .L..............
0x7f7cad18b7d5  48 8b 84 24 08 01 00 00 64 48 2b 04 25 28 00 00  H..$....dH+.%(..
0x7f7cad18b7e5  00 75 24 44 89 c0 48 81 c4 18 01 00 00 c3 0f 1f  .u$D..H.........
0x7f7cad18b7f5  44 00 00 48 8b 15 79 66 18 00 f7 d8 41 b8 ff ff  D..H..yf....A...

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Environment:FailFast <0x00095>
	  at System.Environment:FailFast <0x0002f>
	  at System.Diagnostics.ProcessWaitState:TryReapChild <0x0021b>
	  at System.Diagnostics.ProcessWaitState:CheckChildren <0x0030b>
	  at System.Diagnostics.Process:OnSigChild <0x0006f>
	  at System.Diagnostics.Process:OnSigChild <0x00067>
=================================================================
./RunTests.sh: line 162:    22 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Wed Jul 14 01:09:29 UTC 2021 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
ulimit -c value: unlimited
[ 4229.260421] docker0: port 1(veth2106719) entered blocking state
[ 4229.260422] docker0: port 1(veth2106719) entered forwarding state
[ 4245.736835] docker0: port 1(veth2106719) entered disabled state
[ 4245.736868] vethb7155d4: renamed from eth0
[ 4245.847115] docker0: port 1(veth2106719) entered disabled state
[ 4245.847986] device veth2106719 left promiscuous mode
[ 4245.847988] docker0: port 1(veth2106719) entered disabled state
[ 4252.578039] docker0: port 1(veth34113d2) entered blocking state
[ 4252.578042] docker0: port 1(veth34113d2) entered disabled state
[ 4252.578084] device veth34113d2 entered promiscuous mode
[ 4252.578138] IPv6: ADDRCONF(NETDEV_UP): veth34113d2: link is not ready
[ 4252.904234] eth0: renamed from veth3c9b01b
[ 4252.951089] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4252.953234] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4252.953244] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 4252.953263] IPv6: ADDRCONF(NETDEV_CHANGE): veth34113d2: link becomes ready
[ 4252.953280] docker0: port 1(veth34113d2) entered blocking state
[ 4252.953282] docker0: port 1(veth34113d2) entered forwarding state
[ 4274.783972] docker0: port 1(veth34113d2) entered disabled state
[ 4274.784046] veth3c9b01b: renamed from eth0
[ 4274.877240] docker0: port 1(veth34113d2) entered disabled state
[ 4274.878115] device veth34113d2 left promiscuous mode
[ 4274.878117] docker0: port 1(veth34113d2) entered disabled state
[ 4286.223521] docker0: port 1(veth0ea6c09) entered blocking state
[ 4286.223523] docker0: port 1(veth0ea6c09) entered disabled state
[ 4286.223564] device veth0ea6c09 entered promiscuous mode
[ 4286.223621] IPv6: ADDRCONF(NETDEV_UP): veth0ea6c09: link is not ready
[ 4286.576326] eth0: renamed from veth30f8c3c
[ 4286.623592] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4286.625509] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4286.625520] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 4286.625535] IPv6: ADDRCONF(NETDEV_CHANGE): veth0ea6c09: link becomes ready
[ 4286.625551] docker0: port 1(veth0ea6c09) entered blocking state
[ 4286.625553] docker0: port 1(veth0ea6c09) entered forwarding state
[ 4304.387500] docker0: port 1(veth0ea6c09) entered disabled state
[ 4304.387555] veth30f8c3c: renamed from eth0
[ 4304.500520] docker0: port 1(veth0ea6c09) entered disabled state
[ 4304.501432] device veth0ea6c09 left promiscuous mode
[ 4304.501434] docker0: port 1(veth0ea6c09) entered disabled state
[ 4340.697125] docker0: port 1(veth8484c1c) entered blocking state
[ 4340.697127] docker0: port 1(veth8484c1c) entered disabled state
[ 4340.697166] device veth8484c1c entered promiscuous mode
[ 4340.697217] IPv6: ADDRCONF(NETDEV_UP): veth8484c1c: link is not ready
[ 4341.148288] eth0: renamed from veth3ef6170
[ 4341.202127] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4341.203994] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4341.204028] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 4341.204045] IPv6: ADDRCONF(NETDEV_CHANGE): veth8484c1c: link becomes ready
[ 4341.204062] docker0: port 1(veth8484c1c) entered blocking state
[ 4341.204063] docker0: port 1(veth8484c1c) entered forwarding state

Author:	eerhardt
Assignees:	-
Labels:	`area-VM-meta-mono`
Milestone:	-

stephentoub · 2021-07-14T02:21:15Z

cc: @tmds

tmds · 2021-07-14T06:48:09Z

"Error while reaping child. errno = 10"

This error occurs when someone else reaps child processes that were started by Process.Start

There was a similar case here: #33297. strace can help debug the issue: #33297 (comment).

SamMonoRT · 2021-07-15T22:07:59Z

cc @steveisok @lambdageek - any info on number of hits on this lane ?

lambdageek · 2021-07-16T15:53:16Z

cc @steveisok @lambdageek - any info on number of hits on this lane ?

I'm not sure how to do that. @steveisok could you check?

I went through src/mono/mono looking for wait, waitpid, waitid, SIGCHLD etc. I'm not sure this is mono-specific. We do have some waits, but they are in code that kicks in after we're already in crash reporting. We also explicitly unblock SIGCHLD - but we don't change the signal handler for it.

There's also no calls to wait in msquic, as far as I can tell. (It shows up in the stack trace above).

eerhardt · 2021-07-16T16:03:18Z

I'm not sure this is mono-specific.

I'm not either anymore. I know I added it to the mono area when I opened this issue because it failed on a mono leg and I saw it was coming from native code. But I didn't really know when I logged it which area it should be in.

SamMonoRT · 2021-07-26T13:38:24Z

@steveisok - is there a query to check for reliability of this test suite ?

akoeplinger · 2021-08-06T12:07:42Z

I used this query:

Jobs
| where Queued >= ago(21d) and (QueueName == "ubuntu.1604.amd64.open.rt" or QueueName == "ubuntu.1804.amd64.open.rt")
| join WorkItems on JobId
| where FriendlyName == "System.Net.Requests.Tests" and Status == "BadExit"
| order by Queued desc

and it returned just three results and all of them were other issues. If I extend the query to e.g. 30 days there are a lot more results up until July 17th, but the Console log is no longer available so we can't check if it's this issue. My guess is whatever caused this was fixed around that timeframe.

SamMonoRT · 2021-08-06T12:11:32Z

Thanks @akoeplinger - Based on that, makes sense to close the issue.

eerhardt added the area-VM-meta-mono label Jul 14, 2021

dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Jul 14, 2021

eerhardt mentioned this issue Jul 14, 2021

Compression.ZipFile support for Unix Permissions #55531

Merged

SamMonoRT removed the untriaged New issue has not been triaged by the area owner label Jul 26, 2021

SamMonoRT added this to the 6.0.0 milestone Jul 26, 2021

SamMonoRT closed this as completed Aug 6, 2021

ghost locked as resolved and limited conversation to collaborators Sep 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

System.Net.Requests.Tests failure "Error while reaping child. errno = 10" on release mono Linux x64 Debug failed #55618

System.Net.Requests.Tests failure "Error while reaping child. errno = 10" on release mono Linux x64 Debug failed #55618

eerhardt commented Jul 14, 2021

ghost commented Jul 14, 2021

stephentoub commented Jul 14, 2021

tmds commented Jul 14, 2021

SamMonoRT commented Jul 15, 2021

lambdageek commented Jul 16, 2021

eerhardt commented Jul 16, 2021

SamMonoRT commented Jul 26, 2021

akoeplinger commented Aug 6, 2021

SamMonoRT commented Aug 6, 2021

System.Net.Requests.Tests failure "Error while reaping child. errno = 10" on release mono Linux x64 Debug failed #55618

System.Net.Requests.Tests failure "Error while reaping child. errno = 10" on release mono Linux x64 Debug failed #55618

Comments

eerhardt commented Jul 14, 2021

ghost commented Jul 14, 2021

stephentoub commented Jul 14, 2021

tmds commented Jul 14, 2021

SamMonoRT commented Jul 15, 2021

lambdageek commented Jul 16, 2021

eerhardt commented Jul 16, 2021

SamMonoRT commented Jul 26, 2021

akoeplinger commented Aug 6, 2021

SamMonoRT commented Aug 6, 2021