Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.0][mono][arm64] futex-internal.h: No such file or directory - futex_wait_cancelable native crash in Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests #87444

Closed
carlossanlop opened this issue Jun 13, 2023 · 7 comments
Labels
arch-arm64 area-VM-meta-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-linux Linux OS (any supported distro) runtime-mono specific to the Mono runtime
Milestone

Comments

@carlossanlop
Copy link
Member

carlossanlop commented Jun 13, 2023

Error Blob

Unsure if this is a known issue or not. I didn't find an open with this method failing in the callstack. Feel free to close as dup if you find one.

{
  "ErrorMessage": "futex-internal.h: No such file or directory",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0xffaf543edff0 (LWP 25) "dotnet" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0xffaf533ff1c0 (LWP 26) "SGen worker" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0xffaf516921c0 (LWP 27) "dotnet" 0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0xffaf514911c0 (LWP 28) "Finalizer" 0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0xffaf491de1c0 (LWP 29) "dotnet" 0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0xffaf490c71c0 (LWP 32) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0xffaf48ec61c0 (LWP 33) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0xffaf48b3b1c0 (LWP 34) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  9    Thread 0xffaf48ada1c0 (LWP 35) ".NET ThreadPool" mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
  10   Thread 0xffaf488861c0 (LWP 36) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  11   Thread 0xffaf486851c0 (LWP 37) ".NET Long Runni" 0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  12   Thread 0xffaf276891c0 (LWP 39) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142

Thread 12 (Thread 0xffaf276891c0 (LWP 39)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf27688340, mutex=0xffaf380c8520, cond=0xffaf27688390) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, abstime=0xffaf27688340) at pthread_cond_wait.c:667
#3  0x0000ffaf53c45e98 in mono_os_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#4  0x0000ffaf53c4b494 in mono_coop_cond_timedwait (cond=0xffaf27688390, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#5  mono_lifo_semaphore_timed_wait (semaphore=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#6  0x0000ffaf48aea098 in ?? ()
#7  0x0000ffaf53ba6f28 in ves_icall_System_Threading_Thread_YieldInternal () at /__w/1/s/src/mono/mono/metadata/icall.c:7185
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 11 (Thread 0xffaf486851c0 (LWP 37)):
#0  0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ffaf53d8706c in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xffaf4867ead0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ffaf53d455ac in mono_handle_native_crash (signal=0xffaf53b47770 "SIGSEGV", mctx=0xffaf4867ead0, info=0xffaf4867ee30) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ffaf53cafa80 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xffaf4867ee30, context=0xffaf4867eeb0, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3774
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ffaf2789be98 in ?? ()
#8  0x0000ffaf5347b3a8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 10 (Thread 0xffaf488861c0 (LWP 36)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf2800e960, cond=0xffaf2800e990) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf2800e990, mutex=0xffaf2800e960) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf2800e960) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000000000000002 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (Thread 0xffaf48ada1c0 (LWP 35)):
#0  mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
#1  0x0000ffaf53ca673c in mono_codegen (cfg=0xffaf357a7990) at /__w/1/s/src/mono/mono/mini/mini.c:2212
#2  0x0000ffaf53ca8da0 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3935
#3  0x0000ffaf53caa0b8 in mono_jit_compile_method_inner (method=0xffaf34f93480, opt=172, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini.c:4129
#4  0x0000ffaf53caeba0 in mono_jit_compile_method_with_opt (method=0xffaf34f93480, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2708
#5  jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2763
#6  jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2779
#7  0x0000ffaf53cae018 in mono_jit_compile_method (method=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2798
#8  0x0000ffaf53d47a18 in common_call_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", m=0xffaf34f93480, vt=0xffaf3540ef10, vtable_slot=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#9  0x0000ffaf53d48788 in mono_vcall_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", slot=1, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:840
#10 0x0000ffaf53920504 in ?? ()
#11 0x0000ffaf5368b8d0 in ?? ()
#12 0x0000ffaf5368cec8 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further

Thread 8 (Thread 0xffaf48b3b1c0 (LWP 34)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48b3a2a8, mutex=0xffaf30020ed0, cond=0xffaf30020f00) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf30020f00, mutex=0xffaf30020ed0, abstime=0xffaf48b3a2a8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf30020ed0, timeoutMilliseconds=500) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xffaf48ec61c0 (LWP 33)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48ec50f8, mutex=0xffaf3c00cdf0, cond=0xffaf3c00ce20) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf3c00ce20, mutex=0xffaf3c00cdf0, abstime=0xffaf48ec50f8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf3c00cdf0, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000ffaf48ec5910 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0xffaf490c71c0 (LWP 32)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf3800d0b0, cond=0xffaf3800d0e0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf3800d0e0, mutex=0xffaf3800d0b0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf3800d0b0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342f150 in ?? ()
#6  0x0000ffaf534302b0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 5 (Thread 0xffaf491de1c0 (LWP 29)):
#0  0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x0000ffaf50fa86ac in SignalHandlerLoop (arg=0xab11907743d0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3dd9f) at pthread_create.c:463
#3  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 4 (Thread 0xffaf514911c0 (LWP 28)):
#0  0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x0000ffaf543b0b60 in __new_sem_wait_slow (sem=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x0000ffaf53c1c130 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:891
#6  0x0000ffaf53bf6e8c in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1202
#7  0x0000ffaf53bf6d38 in start_wrapper (data=0xab118feeff20) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#8  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e01f) at pthread_create.c:463
#9  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 3 (Thread 0xffaf516921c0 (LWP 27)):
#0  0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000ffaf53e125a0 in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#2  ds_ipc_poll (poll_handles_data=0xffaf4c0032d0, poll_handles_data_len=1, timeout_ms=4294967295, callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1096
#3  0x0000ffaf53e0fb4c in ds_ipc_stream_factory_get_next_available_stream (callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#4  0x0000ffaf53e0e3c4 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#5  0x0000ffaf53e11930 in ep_rt_thread_mono_start_func (data=0xab118fed3d40) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1356
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e16f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 2 (Thread 0xffaf533ff1c0 (LWP 26)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf53e47768 <lock>, cond=0xffaf53e47798 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf53e47798 <work_cond>, mutex=0xffaf53e47768 <lock>) at pthread_cond_wait.c:655
#3  0x0000ffaf53c97b5c in mono_os_cond_wait (cond=0xffaf53e477c0 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e10f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffaf543edff0 (LWP 25)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xab119041d6c0, cond=0xab119041d6f0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xab119041d6f0, mutex=0xab119041d6c0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xab119041d6c0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342b0e0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Report

Build Definition Test Pull Request
442524 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #93642
442020 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution #93373
2293756 dotnet-runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution #34502
441666 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution #93373
441292 dotnet/runtime LibraryImportGenerator.Unit.Tests.WorkItemExecution #91800
440443 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution #93582
440204 dotnet/runtime System.IO.Pipelines.Tests.WorkItemExecution #77359
439954 dotnet/runtime System.Runtime.Tests.WorkItemExecution #92881
439461 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #93545
438315 dotnet/runtime System.Runtime.Tests.WorkItemExecution #92881
436469 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution #93408
436470 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution #93390
435859 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #93280
435814 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #93380
435488 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution #93182
435309 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution #93182
435247 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #93369
434561 dotnet/runtime JSImportGenerator.Unit.Tests.WorkItemExecution #93282
433593 dotnet/runtime System.Numerics.Tensors.Tests.WorkItemExecution #93296
433363 dotnet/runtime System.Numerics.Tensors.Tests.WorkItemExecution #93296
433289 dotnet/runtime LibraryImportGenerator.Unit.Tests.WorkItemExecution #93292
433207 dotnet/runtime LibraryImportGenerator.Unit.Tests.WorkItemExecution #93289
432794 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
432457 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #93259
432361 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution
432035 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution #93182
431388 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #93191
431038 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution #93182
431005 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #93180
430413 dotnet/runtime Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution
429831 dotnet/runtime System.ComponentModel.EventBasedAsync.Tests.WorkItemExecution #93112
428844 dotnet/runtime System.Text.RegularExpressions.Tests.RegexMatchTests.Match_ExcessPrefix #93078
428746 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution #92980
428612 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution #93006
428315 dotnet/runtime JSImportGenerator.Unit.Tests.WorkItemExecution #93025
428288 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #93053
428216 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #93049
427970 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92974
427728 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #93027
427388 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #93010
2283304 dotnet-runtime Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution #33850
426948 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #79432
426756 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution #92980
426746 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution
426427 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #92967
2282611 dotnet-runtime Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution #33850
425523 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92864
424117 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92863
423438 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution
423068 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution #92829
422545 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92057
422012 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92579
420717 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution #92714
420222 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #91633
420178 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92579
420129 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #89587
419941 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92674
419275 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #86787
2276547 dotnet-runtime Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution #33850
418883 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #89587
418864 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92578
418525 dotnet/runtime PayloadGroup0.WorkItemExecution #92503
418328 dotnet/runtime IntrinsicsInSystemPrivateCoreLib.Tests.WorkItemExecution #92503
417623 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution #92337
417573 dotnet/runtime Microsoft.Extensions.Hosting.Tests.ConsoleLifetimeExitTests.EnsureEnvironmentExitCode #92465
417063 dotnet/runtime LibraryImportGenerator.Unit.Tests.WorkItemExecution #92503
416958 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92534
415444 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92410
415389 dotnet/runtime JSImportGenerator.Unit.Tests.WorkItemExecution
415330 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92457
415290 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #92454
415255 dotnet/runtime LibraryImportGenerator.Unit.Tests.WorkItemExecution #92451
414574 dotnet/runtime JSImportGenerator.Unit.Tests.WorkItemExecution #90074
414177 dotnet/runtime PayloadGroup0.WorkItemExecution
413933 dotnet/runtime LibraryImportGenerator.Unit.Tests.WorkItemExecution #92287
413702 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92385
413187 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution #92354
412471 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution #92287
412440 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution #90436
412090 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution #90436
411570 dotnet/runtime PayloadGroup0.WorkItemExecution
410140 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #90436

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
5 17 82

Known issue validation

Build: 🔎
Result validation: ⚠️ Provided build not found. Provide a valid build in the "Build: 🔎" line.
Validation performed at: 9/1/2023 6:28:52 PM UTC

@carlossanlop carlossanlop added arch-arm64 blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-VM-meta-mono runtime-mono specific to the Mono runtime area-Extensions-Logging Known Build Error Use this to report build issues in the .NET Helix tab labels Jun 13, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 13, 2023
@ghost
Copy link

ghost commented Jun 13, 2023

Tagging subscribers to this area: @dotnet/area-extensions-logging
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

Unsure if this is a known issue or not. I didn't find an open with this method failing in the callstack. Feel free to close as dup if you find one.

{
  "ErrorMessage": "in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": true
}

Reproduction Steps

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0xffaf543edff0 (LWP 25) "dotnet" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0xffaf533ff1c0 (LWP 26) "SGen worker" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0xffaf516921c0 (LWP 27) "dotnet" 0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0xffaf514911c0 (LWP 28) "Finalizer" 0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0xffaf491de1c0 (LWP 29) "dotnet" 0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0xffaf490c71c0 (LWP 32) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0xffaf48ec61c0 (LWP 33) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0xffaf48b3b1c0 (LWP 34) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  9    Thread 0xffaf48ada1c0 (LWP 35) ".NET ThreadPool" mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
  10   Thread 0xffaf488861c0 (LWP 36) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  11   Thread 0xffaf486851c0 (LWP 37) ".NET Long Runni" 0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  12   Thread 0xffaf276891c0 (LWP 39) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142

Thread 12 (Thread 0xffaf276891c0 (LWP 39)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf27688340, mutex=0xffaf380c8520, cond=0xffaf27688390) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, abstime=0xffaf27688340) at pthread_cond_wait.c:667
#3  0x0000ffaf53c45e98 in mono_os_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#4  0x0000ffaf53c4b494 in mono_coop_cond_timedwait (cond=0xffaf27688390, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#5  mono_lifo_semaphore_timed_wait (semaphore=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#6  0x0000ffaf48aea098 in ?? ()
#7  0x0000ffaf53ba6f28 in ves_icall_System_Threading_Thread_YieldInternal () at /__w/1/s/src/mono/mono/metadata/icall.c:7185
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 11 (Thread 0xffaf486851c0 (LWP 37)):
#0  0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ffaf53d8706c in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xffaf4867ead0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ffaf53d455ac in mono_handle_native_crash (signal=0xffaf53b47770 "SIGSEGV", mctx=0xffaf4867ead0, info=0xffaf4867ee30) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ffaf53cafa80 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xffaf4867ee30, context=0xffaf4867eeb0, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3774
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ffaf2789be98 in ?? ()
#8  0x0000ffaf5347b3a8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 10 (Thread 0xffaf488861c0 (LWP 36)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf2800e960, cond=0xffaf2800e990) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf2800e990, mutex=0xffaf2800e960) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf2800e960) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000000000000002 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (Thread 0xffaf48ada1c0 (LWP 35)):
#0  mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
#1  0x0000ffaf53ca673c in mono_codegen (cfg=0xffaf357a7990) at /__w/1/s/src/mono/mono/mini/mini.c:2212
#2  0x0000ffaf53ca8da0 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3935
#3  0x0000ffaf53caa0b8 in mono_jit_compile_method_inner (method=0xffaf34f93480, opt=172, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini.c:4129
#4  0x0000ffaf53caeba0 in mono_jit_compile_method_with_opt (method=0xffaf34f93480, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2708
#5  jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2763
#6  jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2779
#7  0x0000ffaf53cae018 in mono_jit_compile_method (method=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2798
#8  0x0000ffaf53d47a18 in common_call_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", m=0xffaf34f93480, vt=0xffaf3540ef10, vtable_slot=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#9  0x0000ffaf53d48788 in mono_vcall_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", slot=1, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:840
#10 0x0000ffaf53920504 in ?? ()
#11 0x0000ffaf5368b8d0 in ?? ()
#12 0x0000ffaf5368cec8 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further

Thread 8 (Thread 0xffaf48b3b1c0 (LWP 34)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48b3a2a8, mutex=0xffaf30020ed0, cond=0xffaf30020f00) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf30020f00, mutex=0xffaf30020ed0, abstime=0xffaf48b3a2a8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf30020ed0, timeoutMilliseconds=500) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xffaf48ec61c0 (LWP 33)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48ec50f8, mutex=0xffaf3c00cdf0, cond=0xffaf3c00ce20) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf3c00ce20, mutex=0xffaf3c00cdf0, abstime=0xffaf48ec50f8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf3c00cdf0, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000ffaf48ec5910 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0xffaf490c71c0 (LWP 32)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf3800d0b0, cond=0xffaf3800d0e0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf3800d0e0, mutex=0xffaf3800d0b0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf3800d0b0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342f150 in ?? ()
#6  0x0000ffaf534302b0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 5 (Thread 0xffaf491de1c0 (LWP 29)):
#0  0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x0000ffaf50fa86ac in SignalHandlerLoop (arg=0xab11907743d0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3dd9f) at pthread_create.c:463
#3  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 4 (Thread 0xffaf514911c0 (LWP 28)):
#0  0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x0000ffaf543b0b60 in __new_sem_wait_slow (sem=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x0000ffaf53c1c130 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:891
#6  0x0000ffaf53bf6e8c in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1202
#7  0x0000ffaf53bf6d38 in start_wrapper (data=0xab118feeff20) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#8  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e01f) at pthread_create.c:463
#9  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 3 (Thread 0xffaf516921c0 (LWP 27)):
#0  0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000ffaf53e125a0 in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#2  ds_ipc_poll (poll_handles_data=0xffaf4c0032d0, poll_handles_data_len=1, timeout_ms=4294967295, callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1096
#3  0x0000ffaf53e0fb4c in ds_ipc_stream_factory_get_next_available_stream (callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#4  0x0000ffaf53e0e3c4 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#5  0x0000ffaf53e11930 in ep_rt_thread_mono_start_func (data=0xab118fed3d40) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1356
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e16f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 2 (Thread 0xffaf533ff1c0 (LWP 26)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf53e47768 <lock>, cond=0xffaf53e47798 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf53e47798 <work_cond>, mutex=0xffaf53e47768 <lock>) at pthread_cond_wait.c:655
#3  0x0000ffaf53c97b5c in mono_os_cond_wait (cond=0xffaf53e477c0 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e10f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffaf543edff0 (LWP 25)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xab119041d6c0, cond=0xab119041d6f0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xab119041d6f0, mutex=0xab119041d6c0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xab119041d6c0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342b0e0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
Author: carlossanlop
Assignees: -
Labels:

arch-arm64, blocking-clean-ci, area-VM-meta-mono, runtime-mono, area-Extensions-Logging, Known Build Error

Milestone: -

@carlossanlop carlossanlop added the os-linux Linux OS (any supported distro) label Jun 13, 2023
@marek-safar marek-safar removed the untriaged New issue has not been triaged by the area owner label Jul 26, 2023
@marek-safar marek-safar added this to the 7.0.x milestone Jul 26, 2023
@carlossanlop
Copy link
Member Author

There are variants that KnownBuild is not collecting and tagging, so I think this issue is much more impactful that we are seeing. Here is an example:

cc @lewing @steveisok

Output (expand)
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Linq.Queryable.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Linq.Queryable.Tests (found 785 test cases)
  Starting:    System.Linq.Queryable.Tests (parallel test collections = on, max threads = 2)

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7f17cef12771 - Unknown
	0x7f17ceeb9abe - Unknown
	0x7f17cee21b31 - Unknown
	0x7f17cf74d140 - Unknown
	0x7f17cef15cc9 - Unknown
	0x7f17cef14296 - Unknown
	0x7f17cf041b97 - Unknown
	0x7f17cf04a74b - Unknown
	0x7f17cf001d53 - Unknown
	0x7f17cf00c36b - Unknown
	0x7f17cef2695f - Unknown
	0x7f17cef25115 - Unknown
	0x7f17cef1699a - Unknown
	0x7f17cef14296 - Unknown
	0x7f17cf041b97 - Unknown
	0x7f17cf059db4 - Unknown
	0x7f17cf741ea7 - Unknown
	0x7f17cf330a2f - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 23]
[New LWP 24]
[New LWP 25]
[New LWP 26]
[New LWP 29]
[New LWP 30]
[New LWP 31]
[New LWP 32]
[New LWP 33]
[New LWP 34]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so.dbg]
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libSystem.Native.so.dbg]
futex_wait_cancelable (private=0, expected=0, futex_word=0x558565f264d4) at ../sysdeps/nptl/futex-internal.h:186
186	../sysdeps/nptl/futex-internal.h: No such file or directory.
  Id   Target Id                                        Frame 
* 1    Thread 0x7f17cf230740 (LWP 22) "dotnet"          futex_wait_cancelable (private=0, expected=0, futex_word=0x558565f264d4) at ../sysdeps/nptl/futex-internal.h:186
  2    Thread 0x7f17ce3ff700 (LWP 23) "SGen worker"     futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17cf113d68 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
  3    Thread 0x7f17cc601700 (LWP 24) ".NET EventPipe"  0x00007f17cf32496f in __GI___poll (fds=0x7f17c8002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7f17cc400700 (LWP 25) "Finalizer"       futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf104f50 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
  5    Thread 0x7f17c558d700 (LWP 26) ".NET SigHandler" __libc_read (nbytes=1, buf=0x7f17c558cee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
  6    Thread 0x7f17c54f0700 (LWP 29) ".NET Long Runni" futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17b4248df0) at ../sysdeps/nptl/futex-internal.h:186
  7    Thread 0x7f17c51ee700 (LWP 30) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c51ed668, clockid=-987834928, expected=0, futex_word=0x7f17b80a7250) at ../sysdeps/nptl/futex-internal.h:323
  8    Thread 0x7f17c5268700 (LWP 31) ".NET TP Gate"    futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17ac000bc0) at ../sysdeps/nptl/futex-internal.h:323
  9    Thread 0x7f17c490b700 (LWP 32) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c490a6e8, clockid=-997153184, expected=0, futex_word=0x7f17c490a768) at ../sysdeps/nptl/futex-internal.h:323
  10   Thread 0x7f17c4609700 (LWP 33) ".NET Long Runni" 0x00007f17cf2fc747 in __GI___wait4 (pid=35, stat_loc=0x7f17c4605080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
  11   Thread 0x7f17c4307700 (LWP 34) ".NET Long Runni" futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf0fef10 <suspend_semaphore>) at ../sysdeps/nptl/futex-internal.h:323

Thread 11 (Thread 0x7f17c4307700 (LWP 34) ".NET Long Runni"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf0fef10 <suspend_semaphore>) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7f17cf0fef10 <suspend_semaphore>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007f17cf74b278 in __new_sem_wait_slow (sem=0x7f17cf0fef10 <suspend_semaphore>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007f17cefcacc8 in mono_os_sem_timedwait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefca965 in mono_threads_wait_pending_operations () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cf08d546 in unified_suspend_stop_world () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cf08d24d in sgen_client_stop_world () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cf0ae2cb in sgen_stop_world () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf0ab08e in sgen_perform_collection () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf0aaff3 in sgen_ensure_free_space () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf0a647d in sgen_alloc_obj_nolock () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cf0a6999 in sgen_alloc_obj () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007f17cf08f0d6 in mono_gc_alloc_obj () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007f17cef1c01a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#14 0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#15 0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#16 0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#17 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f17c4609700 (LWP 33) ".NET Long Runni"):
#0  0x00007f17cf2fc747 in __GI___wait4 (pid=35, stat_loc=0x7f17c4605080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1  0x00007f17cef128a2 in mono_dump_native_crash_info () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007f17ceeb9abe in mono_handle_native_crash () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007f17cee21b31 in mono_sigsegv_signal_handler_debug () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  <signal handler called>
#5  0x00007f17cef15cc9 in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf04a74b in mono_runtime_try_invoke_byrefs () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf001d53 in ves_icall_InternalInvoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf00c36b in ves_icall_InternalInvoke_raw () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cef2695f in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007f17cef25115 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#14 0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#15 0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#16 0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#17 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f17c490b700 (LWP 32) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c490a6e8, clockid=-997153184, expected=0, futex_word=0x7f17c490a768) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7f17c490a6e8, clockid=-997153184, mutex=0x7f17b40970b0, cond=0x7f17c490a740) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7f17c490a740, mutex=0x7f17b40970b0, abstime=0x7f17c490a6e8) at pthread_cond_wait.c:656
#3  0x00007f17cefc39f3 in mono_os_cond_timedwait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefc8899 in mono_lifo_semaphore_timed_wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef268d7 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef25115 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f17c5268700 (LWP 31) ".NET TP Gate"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17ac000bc0) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7f17ac000bc0, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007f17cf74b278 in __new_sem_wait_slow (sem=0x7f17ac000bc0, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007f17cefca7c8 in mono_thread_info_wait_for_resume () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefd02c1 in mono_threads_exit_gc_safe_region_unbalanced_internal () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250de in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f17c51ee700 (LWP 30) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c51ed668, clockid=-987834928, expected=0, futex_word=0x7f17b80a7250) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7f17c51ed668, clockid=-987834928, mutex=0x7f17b80a7200, cond=0x7f17b80a7228) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7f17b80a7228, mutex=0x7f17b80a7200, abstime=0x7f17c51ed668) at pthread_cond_wait.c:656
#3  0x00007f17ce82ca97 in SystemNative_LowLevelMonitor_TimedWait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007f17cef268d7 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250cf in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f17c54f0700 (LWP 29) ".NET Long Runni"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17b4248df0) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f17b4248da0, cond=0x7f17b4248dc8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7f17b4248dc8, mutex=0x7f17b4248da0) at pthread_cond_wait.c:638
#3  0x00007f17ce82c937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007f17cef2688e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250cf in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f17c558d700 (LWP 26) ".NET SigHandler"):
#0  __libc_read (nbytes=1, buf=0x7f17c558cee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __libc_read (fd=7, buf=0x7f17c558cee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007f17ce82c086 in SignalHandlerLoop () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#3  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#4  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f17cc400700 (LWP 25) "Finalizer"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf104f50 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7f17cf104f50 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007f17cf74b278 in __new_sem_wait_slow (sem=0x7f17cf104f50 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007f17cf081008 in finalizer_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cf059d09 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f17cc601700 (LWP 24) ".NET EventPipe"):
#0  0x00007f17cf32496f in __GI___poll (fds=0x7f17c8002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f17cefa455e in ds_ipc_poll () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007f17cefa1288 in ds_ipc_stream_factory_get_next_available_stream () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007f17cef9f7af in server_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefa39c1 in ep_rt_thread_mono_start_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f17ce3ff700 (LWP 23) "SGen worker"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17cf113d68 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f17cf113d18 <lock>, cond=0x7f17cf113d40 <work_cond>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7f17cf113d40 <work_cond>, mutex=0x7f17cf113d18 <lock>) at pthread_cond_wait.c:638
#3  0x00007f17cf0e8ac3 in thread_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f17cf230740 (LWP 22) "dotnet"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x558565f264d4) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558565f26480, cond=0x558565f264a8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x558565f264a8, mutex=0x558565f26480) at pthread_cond_wait.c:638
#3  0x00007f17ce82c937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007f17cef2688e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250cf in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf048808 in mono_runtime_exec_main_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cee7a51f in mono_jit_exec () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cee7cd8b in mono_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007f17ceef0ad7 in monovm_execute_assembly () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007f17cf14c778 in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#14 0x00007f17cf17adea in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#15 0x00007f17cf17b0df in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#16 0x00007f17cf17c082 in corehost_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#17 0x00007f17cf1f6afc in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#18 0x00007f17cf1f1c73 in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#19 0x00007f17cf1ee0ec in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#20 0x00007f17cf1ed1cb in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#21 0x00007f17cf1e6115 in hostfxr_main_startupinfo () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#22 0x0000558562e2e312 in ?? ()
#23 0x0000558562e2e835 in ?? ()
#24 0x00007f17cf258d0a in __libc_start_main (main=0x558562e2e750, argc=18, argv=0x7ffcc15b9eb8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcc15b9ea8) at ../csu/libc-start.c:308
#25 0x0000558562e21979 in ?? ()
[Inferior 1 (process 22) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7f17cef15cc9):0x7f17cef15cb9  1d 00 49 89 db ff 24 c1 49 8b 44 24 08 48 8b 00  ..I...$.I.D$.H..
0x7f17cef15cc9  48 8b 70 18 48 8d 3d de 48 e6 ff 31 c0 e8 05 9f  H.p.H.=.H..1....
0x7f17cef15cd9  09 00 48 8d 3d 48 90 e4 ff be 17 0f 00 00 e8 c4  ..H.=H..........
0x7f17cef15ce9  a3 09 00 0f 1f 40 00 4c 89 e6 4c 89 db 48 83 c3  [email protected]..

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x00096>
	  at System.Reflection.MethodBaseInvoker:InterpretedInvoke_Method <0x0003c>
	  at System.Reflection.MethodBaseInvoker:InvokeWithNoArgs <0x0006e>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x000fe>
	  at System.Reflection.MethodBase:Invoke <0x0002e>
	  at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00030>
	  at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x00242>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__1 <0x00060>
	  at <AggregateAsync>d__4:MoveNext <0x0007a>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x00074>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__0 <0x00082>
	  at <RunAsync>d__9:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <InvokeTestMethodAsync>d__48:MoveNext <0x00122>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x00074>
	  at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x000b8>
	  at <<RunAsync>b__47_0>d:MoveNext <0x003ba>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x00060>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at Xunit.Sdk.TestInvoker`1:RunAsync <0x0003a>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x000a8>
	  at <InvokeTestAsync>d__4:MoveNext <0x00178>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x00074>
	  at <>c__DisplayClass43_0:<RunAsync>b__0 <0x0002c>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <RunAsync>d__43:MoveNext <0x002aa>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCaseRunner:RunTestAsync <0x00104>
	  at <RunAsync>d__19:MoveNext <0x001ee>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCase:RunAsync <0x0006a>
	  at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x00076>
	  at <RunTestCasesAsync>d__32:MoveNext <0x000fe>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x00060>
	  at <RunAsync>d__31:MoveNext <0x00114>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x000a2>
	  at <RunTestMethodsAsync>d__38:MoveNext <0x003a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x00060>
	  at <RunAsync>d__37:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x000aa>
	  at <RunTestClassesAsync>d__28:MoveNext <0x001a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x00060>
	  at <RunAsync>d__27:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x00076>
	  at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x00092>
	  at System.Threading.Tasks.Task`1:InnerInvoke <0x00052>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntry <0x000b6>
	  at <>c:<.cctor>b__8_0 <0x0001c>
	  at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x00042>
	  at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x00056>
	  at System.Action`1:invoke_void_T <0x00080>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Object:lambda_method2 <0x0004a>
	  at Xunit.Sdk.ExecutionContextHelper:Run <0x00034>
	  at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x0016c>
	  at <>c:<QueueUserWorkItem>b__5_0 <0x00038>
	  at System.Threading.Tasks.Task:InnerInvoke <0x00092>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x00086>
	  at <>c:<.cctor>b__10_0 <0x00020>
	  at System.Object:runtime_invoke_direct_void__this__ <0x0022c>
	  at <unknown> <0x00000>
=================================================================
./RunTests.sh: line 168:    22 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Linq.Queryable.Tests.runtimeconfig.json --depsfile System.Linq.Queryable.Tests.deps.json xunit.console.dll System.Linq.Queryable.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri 25 Aug 2023 11:45:42 PM UTC ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

@carlossanlop
Copy link
Member Author

Here's another recent hit:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Linq.Queryable.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Linq.Queryable.Tests (found 785 test cases)
  Starting:    System.Linq.Queryable.Tests (parallel test collections = on, max threads = 2)

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7fa43adec811 - Unknown
	0x7fa43ad93b2e - Unknown
	0x7fa43acfbb31 - Unknown
	0x7fa43b627140 - Unknown
	0x7fa43adefd69 - Unknown
	0x7fa43adee336 - Unknown
	0x7fa43af1bc27 - Unknown
	0x7fa43af247db - Unknown
	0x7fa43aedbdf3 - Unknown
	0x7fa43aee640b - Unknown
	0x7fa43ae009ff - Unknown
	0x7fa43adff1b5 - Unknown
	0x7fa43adf0a3a - Unknown
	0x7fa43adee336 - Unknown
	0x7fa43af1bc27 - Unknown
	0x7fa43af33e44 - Unknown
	0x7fa43b61bea7 - Unknown
	0x7fa43b20aa2f - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 23]
[New LWP 24]
[New LWP 25]
[New LWP 26]
[New LWP 29]
[New LWP 30]
[New LWP 31]
[New LWP 32]
[New LWP 33]
[New LWP 34]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so.dbg]
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libSystem.Native.so.dbg]
futex_wait_cancelable (private=0, expected=0, futex_word=0x558319c09954) at ../sysdeps/nptl/futex-internal.h:186
186	../sysdeps/nptl/futex-internal.h: No such file or directory.
  Id   Target Id                                        Frame 
* 1    Thread 0x7fa43b10a740 (LWP 22) "dotnet"          futex_wait_cancelable (private=0, expected=0, futex_word=0x558319c09954) at ../sysdeps/nptl/futex-internal.h:186
  2    Thread 0x7fa43a3ff700 (LWP 23) "SGen worker"     futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa43afeddf8 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
  3    Thread 0x7fa438601700 (LWP 24) ".NET EventPipe"  0x00007fa43b1fe96f in __GI___poll (fds=0x7fa434002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7fa4382ff700 (LWP 25) "Finalizer"       futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa43afdefe0 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
  5    Thread 0x7fa431467700 (LWP 26) ".NET SigHandler" __libc_read (nbytes=1, buf=0x7fa431466ee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
  6    Thread 0x7fa4310c8700 (LWP 29) ".NET Long Runni" futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa42402e050) at ../sysdeps/nptl/futex-internal.h:186
  7    Thread 0x7fa4313ca700 (LWP 30) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4313c9668, clockid=826054096, expected=0, futex_word=0x7fa420028730) at ../sysdeps/nptl/futex-internal.h:323
  8    Thread 0x7fa431142700 (LWP 31) ".NET TP Gate"    futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4311417b8, clockid=823400224, expected=0, futex_word=0x7fa41800ade0) at ../sysdeps/nptl/futex-internal.h:323
  9    Thread 0x7fa4307e7700 (LWP 32) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4307e66e8, clockid=813590112, expected=0, futex_word=0x7fa4307e6768) at ../sysdeps/nptl/futex-internal.h:323
  10   Thread 0x7fa4304e5700 (LWP 33) ".NET Long Runni" 0x00007fa43ae02a20 in interp_cprop () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
  11   Thread 0x7fa417fff700 (LWP 34) ".NET Long Runni" 0x00007fa43b1d6747 in __GI___wait4 (pid=35, stat_loc=0x7fa417ffb080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27

Thread 11 (Thread 0x7fa417fff700 (LWP 34) ".NET Long Runni"):
#0  0x00007fa43b1d6747 in __GI___wait4 (pid=35, stat_loc=0x7fa417ffb080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1  0x00007fa43adec942 in mono_dump_native_crash_info () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007fa43ad93b2e in mono_handle_native_crash () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007fa43acfbb31 in mono_sigsegv_signal_handler_debug () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  <signal handler called>
#5  0x00007fa43adefd69 in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af247db in mono_runtime_try_invoke_byrefs () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43aedbdf3 in ves_icall_InternalInvoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43aee640b in ves_icall_InternalInvoke_raw () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007fa43ae009ff in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007fa43adff1b5 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#14 0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#15 0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#16 0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#17 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7fa4304e5700 (LWP 33) ".NET Long Runni"):
#0  0x00007fa43ae02a20 in interp_cprop () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#1  0x00007fa43ae16f5c in generate () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007fa43ae166df in mono_interp_transform_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007fa43ae2b483 in tier_up_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43ae2b1a2 in mono_interp_tier_up_frame_enter () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adfc8c3 in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#10 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fa4307e7700 (LWP 32) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4307e66e8, clockid=813590112, expected=0, futex_word=0x7fa4307e6768) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7fa4307e66e8, clockid=813590112, mutex=0x7fa42407f440, cond=0x7fa4307e6740) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7fa4307e6740, mutex=0x7fa42407f440, abstime=0x7fa4307e66e8) at pthread_cond_wait.c:656
#3  0x00007fa43ae9da93 in mono_os_cond_timedwait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43aea2939 in mono_lifo_semaphore_timed_wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43ae00977 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adff1b5 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fa431142700 (LWP 31) ".NET TP Gate"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4311417b8, clockid=823400224, expected=0, futex_word=0x7fa41800ade0) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7fa4311417b8, clockid=823400224, mutex=0x7fa41800ad90, cond=0x7fa41800adb8) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7fa41800adb8, mutex=0x7fa41800ad90, abstime=0x7fa4311417b8) at pthread_cond_wait.c:656
#3  0x00007fa4380f8a97 in SystemNative_LowLevelMonitor_TimedWait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007fa43ae00977 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adff16f in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fa4313ca700 (LWP 30) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4313c9668, clockid=826054096, expected=0, futex_word=0x7fa420028730) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7fa4313c9668, clockid=826054096, mutex=0x7fa4200286e0, cond=0x7fa420028708) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7fa420028708, mutex=0x7fa4200286e0, abstime=0x7fa4313c9668) at pthread_cond_wait.c:656
#3  0x00007fa4380f8a97 in SystemNative_LowLevelMonitor_TimedWait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007fa43ae00977 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adff16f in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fa4310c8700 (LWP 29) ".NET Long Runni"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa42402e050) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa42402e000, cond=0x7fa42402e028) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7fa42402e028, mutex=0x7fa42402e000) at pthread_cond_wait.c:638
#3  0x00007fa4380f8937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007fa43ae0092e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adff16f in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fa431467700 (LWP 26) ".NET SigHandler"):
#0  __libc_read (nbytes=1, buf=0x7fa431466ee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __libc_read (fd=7, buf=0x7fa431466ee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007fa4380f8086 in SignalHandlerLoop () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#3  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#4  0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa4382ff700 (LWP 25) "Finalizer"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa43afdefe0 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7fa43afdefe0 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007fa43b625278 in __new_sem_wait_slow (sem=0x7fa43afdefe0 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007fa43af5b098 in finalizer_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43af33d99 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa438601700 (LWP 24) ".NET EventPipe"):
#0  0x00007fa43b1fe96f in __GI___poll (fds=0x7fa434002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fa43ae7e5fe in ds_ipc_poll () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007fa43ae7b328 in ds_ipc_stream_factory_get_next_available_stream () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007fa43ae7984f in server_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43ae7da61 in ep_rt_thread_mono_start_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa43a3ff700 (LWP 23) "SGen worker"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa43afeddf8 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa43afedda8 <lock>, cond=0x7fa43afeddd0 <work_cond>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7fa43afeddd0 <work_cond>, mutex=0x7fa43afedda8 <lock>) at pthread_cond_wait.c:638
#3  0x00007fa43afc2b53 in thread_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa43b10a740 (LWP 22) "dotnet"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x558319c09954) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558319c09900, cond=0x558319c09928) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x558319c09928, mutex=0x558319c09900) at pthread_cond_wait.c:638
#3  0x00007fa4380f8937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007fa43ae0092e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adff16f in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af22898 in mono_runtime_exec_main_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43ad5457f in mono_jit_exec () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007fa43ad56deb in mono_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007fa43adcab77 in monovm_execute_assembly () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007fa43b026778 in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#14 0x00007fa43b054dea in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#15 0x00007fa43b0550df in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#16 0x00007fa43b056082 in corehost_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#17 0x00007fa43b0d0afc in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#18 0x00007fa43b0cbc73 in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#19 0x00007fa43b0c80ec in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#20 0x00007fa43b0c71cb in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#21 0x00007fa43b0c0115 in hostfxr_main_startupinfo () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#22 0x0000558317ede312 in ?? ()
#23 0x0000558317ede835 in ?? ()
#24 0x00007fa43b132d0a in __libc_start_main (main=0x558317ede750, argc=18, argv=0x7ffc8c56dc78, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc8c56dc68) at ../csu/libc-start.c:308
#25 0x0000558317ed1979 in ?? ()
[Inferior 1 (process 22) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fa43adefd69):0x7fa43adefd59  1d 00 49 89 db ff 24 c1 49 8b 44 24 08 48 8b 00  ..I...$.I.D$.H..
0x7fa43adefd69  48 8b 70 18 48 8d 3d 3e 48 e6 ff 31 c0 e8 05 9f  H.p.H.=>H..1....
0x7fa43adefd79  09 00 48 8d 3d 67 8f e4 ff be 17 0f 00 00 e8 c4  ..H.=g..........
0x7fa43adefd89  a3 09 00 0f 1f 40 00 4c 89 e6 4c 89 db 48 83 c3  [email protected]..

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x00096>
	  at System.Reflection.MethodBaseInvoker:InterpretedInvoke_Method <0x0003c>
	  at System.Reflection.MethodBaseInvoker:InvokeWithNoArgs <0x0006e>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x000fe>
	  at System.Reflection.MethodBase:Invoke <0x0002e>
	  at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00030>
	  at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x00242>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__1 <0x00060>
	  at <AggregateAsync>d__4:MoveNext <0x0007a>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x00074>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__0 <0x00082>
	  at <RunAsync>d__9:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <InvokeTestMethodAsync>d__48:MoveNext <0x00122>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x00074>
	  at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x000b8>
	  at <<RunAsync>b__47_0>d:MoveNext <0x003ba>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x00060>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at Xunit.Sdk.TestInvoker`1:RunAsync <0x0003a>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x000a8>
	  at <InvokeTestAsync>d__4:MoveNext <0x00178>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x00074>
	  at <>c__DisplayClass43_0:<RunAsync>b__0 <0x0002c>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <RunAsync>d__43:MoveNext <0x002aa>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCaseRunner:RunTestAsync <0x00104>
	  at <RunAsync>d__19:MoveNext <0x001ee>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCase:RunAsync <0x0006a>
	  at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x00076>
	  at <RunTestCasesAsync>d__32:MoveNext <0x000fe>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x00060>
	  at <RunAsync>d__31:MoveNext <0x00114>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x000a2>
	  at <RunTestMethodsAsync>d__38:MoveNext <0x003a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x00060>
	  at <RunAsync>d__37:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x000aa>
	  at <RunTestClassesAsync>d__28:MoveNext <0x001a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x00060>
	  at <RunAsync>d__27:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x00076>
	  at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x00092>
	  at System.Threading.Tasks.Task`1:InnerInvoke <0x00052>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntry <0x000b6>
	  at <>c:<.cctor>b__8_0 <0x0001c>
	  at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x00042>
	  at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x00056>
	  at System.Action`1:invoke_void_T <0x00080>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Object:lambda_method2 <0x0004a>
	  at Xunit.Sdk.ExecutionContextHelper:Run <0x00034>
	  at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x0016c>
	  at <>c:<QueueUserWorkItem>b__5_0 <0x00038>
	  at System.Threading.Tasks.Task:InnerInvoke <0x00092>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x00086>
	  at <>c:<.cctor>b__10_0 <0x00020>
	  at System.Object:runtime_invoke_direct_void__this__ <0x0022c>
	  at <unknown> <0x00000>
=================================================================
/root/helix/work/workitem/e
./RunTests.sh: line 168:    22 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Linq.Queryable.Tests.runtimeconfig.json --depsfile System.Linq.Queryable.Tests.deps.json xunit.console.dll System.Linq.Queryable.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
----- end Fri 01 Sep 2023 12:29:36 AM UTC ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
ulimit -c value: unlimited
[ 2375.998335]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[ 2375.998337] RIP: 0033:0x7f6cadb3311b
[ 2375.998338] Code: 0f 05 48 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8 f3 fb ff ff 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 31 fc ff ff 8b 44
[ 2375.998339] RSP: 002b:00007f6ca2bcfe30 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
[ 2375.998340] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f6cadb3311b
[ 2375.998341] RDX: 0000000000000000 RSI: 0000000000000007 RDI: 0000000000000012
[ 2375.998342] RBP: 00007f6ca2bcfe60 R08: 0000000000000000 R09: 0000000000000006
[ 2375.998342] R10: 00007f6ca2bcfd90 R11: 0000000000000293 R12: 0000000000000000
[ 2375.998343] R13: 0000000000000000 R14: 00007f6c80001c60 R15: 0000000000000012
[ 2375.998345] ---[ end trace f9ac0af0f2808493 ]---
[ 2378.475234] can: request_module (can-proto-0) failed.
[ 2383.576928] docker0: port 1(vethe5ca674) entered disabled state
[ 2383.577594] veth17fbed9: renamed from eth0
[ 2383.678445] docker0: port 1(vethe5ca674) entered disabled state
[ 2383.684547] device vethe5ca674 left promiscuous mode
[ 2383.684562] docker0: port 1(vethe5ca674) entered disabled state
[ 2396.822816] docker0: port 1(veth13b9403) entered blocking state
[ 2396.822818] docker0: port 1(veth13b9403) entered disabled state
[ 2396.822974] device veth13b9403 entered promiscuous mode
[ 2397.131010] eth0: renamed from vethaaad365
[ 2397.187877] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 2397.187902] IPv6: ADDRCONF(NETDEV_CHANGE): veth13b9403: link becomes ready
[ 2397.187935] docker0: port 1(veth13b9403) entered blocking state
[ 2397.187937] docker0: port 1(veth13b9403) entered forwarding state
[ 2405.259833] docker0: port 1(veth13b9403) entered disabled state
[ 2405.259898] vethaaad365: renamed from eth0
[ 2405.368180] docker0: port 1(veth13b9403) entered disabled state
[ 2405.372802] device veth13b9403 left promiscuous mode
[ 2405.372815] docker0: port 1(veth13b9403) entered disabled state
[ 2416.166844] docker0: port 1(vethb84acd7) entered blocking state
[ 2416.166846] docker0: port 1(vethb84acd7) entered disabled state
[ 2416.166958] device vethb84acd7 entered promiscuous mode
[ 2416.483555] eth0: renamed from vethaf30f8c
[ 2416.541940] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 2416.541964] IPv6: ADDRCONF(NETDEV_CHANGE): vethb84acd7: link becomes ready
[ 2416.541997] docker0: port 1(vethb84acd7) entered blocking state
[ 2416.541998] docker0: port 1(vethb84acd7) entered forwarding state
[ 2418.189035] docker0: port 1(vethb84acd7) entered disabled state
[ 2418.189097] vethaf30f8c: renamed from eth0
[ 2418.295760] docker0: port 1(vethb84acd7) entered disabled state
[ 2418.302104] device vethb84acd7 left promiscuous mode
[ 2418.302115] docker0: port 1(vethb84acd7) entered disabled state
[ 2426.257601] docker0: port 1(veth3cb3d94) entered blocking state
[ 2426.257603] docker0: port 1(veth3cb3d94) entered disabled state
[ 2426.257677] device veth3cb3d94 entered promiscuous mode
[ 2426.552209] eth0: renamed from veth4fbbe8b
[ 2426.613649] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 2426.613672] IPv6: ADDRCONF(NETDEV_CHANGE): veth3cb3d94: link becomes ready
[ 2426.613703] docker0: port 1(veth3cb3d94) entered blocking state
[ 2426.613705] docker0: port 1(veth3cb3d94) entered forwarding state
Waiting a few seconds for any dump to be written..
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 0
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dump..
... found no dump in /root/helix/work/workitem/e
+ export _commandExitCode=134
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 8436822 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiI3N2Y3M2NkOC00YzQxLTRhNTEtYTMxOS1hMjA4MzFlYjllNGIiLCJzaWQiOiIwZjRlZmJjYy0wYmM5LTQ4NDItODJkZS1hNDQ1MGI2NzJhZDkiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzM5MjQ4NCIsImpvYnJlZiI6IjI0NzhhZDEyLTk2Y2ItNDQ2OS1iNTFlLTQxNmViNjQyN2FiMDo5MzEwMTg1ZC05ODhmLTU5MzAtNGEwOS02OTE4YzFiYjM2ZmIiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMzkyNDg0Iiwib3JjaGlkIjoiMjQ3OGFkMTItOTZjYi00NDY5LWI1MWUtNDE2ZWI2NDI3YWIwLmJ1aWxkLmxpYnJhcmllc190ZXN0X3J1bl9yZWxlYXNlX21vbm9faW50ZXJwcmV0ZXJfbGludXhfeDY0X2RlYnVnLl9fZGVmYXVsdCIsInJlcG9JZHMiOiIiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286NmZjYzkyZTUtNzNhNy00Zjg4LThkMTMtZDkwNDViNDVmYjI3IiwibmJmIjoxNjkzNTI2NjczLCJleHAiOjE2OTM1MzY4NzN9.sjxF_cyOrQMYjxUnK59W_yzEa1-RQcHRClnJxborVUk4CqfdJVqT8Goio5w8ZXPuOuCAqgaGdfuHrW1sWPk37OSr9a164QAhbHFdVwQ2DhQXxwHUDTUkV5xOBkmzB38nHtD1b36Zdr4PG1e_DjjNi1wScWuUi2gSyUNqKIx2lWgWmeSDgXcs4VazGHOnIFOiNiqiTmNq86jul-1ZnpXsHLObbJHl-0eH0spPVxwKAIiyvKZhxdRjxQxx1bBuPLTklJ9BigepHNFgEP9Wz0pGuSvJizLVKLesBiZ9c39sLDGgKF1g_aNr7dnd7xcMlp1GQ7VL7G7KzCacRyjzkuCprA
2023-09-01T00:29:46.993Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-09-01T00:29:46.993Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/e' for test results files
2023-09-01T00:29:46.994Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/uploads' for test results files
2023-09-01T00:29:46.994Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-09-01T00:29:46.994Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2023-09-01T00:29:46.994Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1565 bytes
+ exit 134
+ export _commandExitCode=134
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 134

[END EXECUTION]
Exit Code:134

@carlossanlop
Copy link
Member Author

This is very impactful and affecting both 7.0 and 8.0. @lewing do you know who can take a look? This is mono.

@lewing
Copy link
Member

lewing commented Oct 18, 2023

@lambdageek can you triage this?

@lambdageek
Copy link
Member

@carlossanlop Please don't make "futex-internal.h: No such file or directory" tracking issues. That is a generic error message when gdb tries to look up source code for libc after a crash on Linux (mono invokes gdb from its crash handler if it is installed on the system). It will catch literally every single native Mono crash - any two completely unrelated issues.

@carlossanlop
Copy link
Member Author

Ok, in that case, let's close this and start over.

@ghost ghost locked as resolved and limited conversation to collaborators Nov 17, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-arm64 area-VM-meta-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-linux Linux OS (any supported distro) runtime-mono specific to the Mono runtime
Projects
None yet
Development

No branches or pull requests

5 participants