Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Text.Json.Tests crashing with segfault in CI #57198

Closed
ViktorHofer opened this issue Aug 11, 2021 · 21 comments
Closed

System.Text.Json.Tests crashing with segfault in CI #57198

ViktorHofer opened this issue Aug 11, 2021 · 21 comments
Labels
arch-arm64 area-ExceptionHandling-coreclr bug os-linux-musl Linux distributions using musl library.
Milestone
7.0.0

Comments

@ViktorHofer
Copy link
Member

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1287656&view=ms.vss-test-web.build-test-results-tab&runId=38015200&resultId=183344&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Configuration: net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e

/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 2741 of 2798 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 4)
./RunTests.sh: line 162:    93 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Wed Aug 11 10:28:18 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

Console log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57193-merge-32896558dfa04faba1/System.Text.Json.Tests/1/console.0d8519e9.log?sv=2019-07-07&se=2021-08-31T10%3A24%3A36Z&sr=c&sp=rl&sig=sW6tkW0yTybYwRggqu%2FJypbei5TuvxQeYGIra%2BV%2BBa8%3D

Dump: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57193-merge-32896558dfa04faba1/System.Text.Json.Tests/1/core.1001.93?sv=2019-07-07&se=2021-08-31T10%3A24%3A36Z&sr=c&sp=rl&sig=sW6tkW0yTybYwRggqu%2FJypbei5TuvxQeYGIra%2BV%2BBa8%3D
@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.Text.Json untriaged New issue has not been triaged by the area owner labels Aug 11, 2021
@ghost
Copy link

ghost commented Aug 11, 2021

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1287656&view=ms.vss-test-web.build-test-results-tab&runId=38015200&resultId=183344&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Configuration: net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e

/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 2741 of 2798 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 4)
./RunTests.sh: line 162:    93 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Wed Aug 11 10:28:18 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

Console log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57193-merge-32896558dfa04faba1/System.Text.Json.Tests/1/console.0d8519e9.log?sv=2019-07-07&se=2021-08-31T10%3A24%3A36Z&sr=c&sp=rl&sig=sW6tkW0yTybYwRggqu%2FJypbei5TuvxQeYGIra%2BV%2BBa8%3D

Dump: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57193-merge-32896558dfa04faba1/System.Text.Json.Tests/1/core.1001.93?sv=2019-07-07&se=2021-08-31T10%3A24%3A36Z&sr=c&sp=rl&sig=sW6tkW0yTybYwRggqu%2FJypbei5TuvxQeYGIra%2BV%2BBa8%3D

Author: ViktorHofer
Assignees: -
Labels:

area-System.Text.Json, untriaged
Milestone: -

@ViktorHofer ViktorHofer mentioned this issue Aug 11, 2021
Merged
@eiriktsarpalis
Copy link
Member

Seems like a one-off occurrence so far. We can evaluate if it reappears.

@eiriktsarpalis eiriktsarpalis removed the untriaged New issue has not been triaged by the area owner label Aug 12, 2021
@eiriktsarpalis eiriktsarpalis added this to the 6.0.0 milestone Aug 12, 2021
@eiriktsarpalis eiriktsarpalis self-assigned this Aug 12, 2021
@ViktorHofer
Copy link
Member Author

I think I saw this multiple times. Runfo/Kusto should provide you with the data of how often this has yet happened in CI.

@eiriktsarpalis
Copy link
Member

There have been 11 or so System.Text.Json segfaults in the past month for that particular queue, but none occurring in master.

let jobs = Jobs
| where Queued > ago (30d)
// | where Source == "ci/public/dotnet/runtime/refs/heads/main"
| project Name;
WorkItems
| where Started > ago (30d)
| where JobName  in (jobs)
| where QueueName == "ubuntu.1804.armarch.open"
| where ExitCode == 139
| summarize count() by FriendlyName, QueueName

But it seems like 5 of them occurred in the past week:

let crashes =
WorkItems
| where Started > ago (30d)
| where QueueName == "ubuntu.1804.armarch.open"
| where ExitCode == 139;
let min_t = toscalar(crashes | summarize min(Started));
let max_t = toscalar(crashes | summarize max(Started));
crashes
| make-series num=count() on Started in range(min_t, max_t, 1d) by FriendlyName
| limit 5
| render timechart

image

I'll investigate the dumps.

@eiriktsarpalis
Copy link
Member

@ViktorHofer my arm64 WSL2 installation has suddenly stopped working 🤷, so I'm trying to analyze the dump on Windows using the cross DAC binaries. However following the instructions detailed here results in the following error message when I try to dump the stack or any related command:

> clrstack
Failed to load data access module, 0x80004002
Can not load or initialize mscordaccore.dll. The target runtime may not be initialized.

Perhaps I'm copying the DAC artifacts incorrectly, I'm assuming we should be copying the contents of Linux.arm64.Checked/x64 directly into the shared/Microsoft.NETCore.App/6.0.0 folder of the helix payload?

@eiriktsarpalis eiriktsarpalis added Priority:3 Work that is nice to have blocking-release labels Aug 17, 2021
@eiriktsarpalis
Copy link
Member

eiriktsarpalis commented Aug 17, 2021 •
edited
Loading

Marking as blocking-release as it's unlikely this will get addressed by the snap deadline. It's relatively low priority but we should still investigate this for 6.0.0.

@ViktorHofer
Copy link
Member Author

cc @sdmaclea for Eirik's question.

@lewing lewing mentioned this issue Aug 25, 2021
Merged
@eiriktsarpalis
Copy link
Member

Hi @sdmaclea, could you take a look at the above?

@eiriktsarpalis eiriktsarpalis modified the milestones: 6.0.0, 7.0.0 Sep 11, 2021
@eiriktsarpalis eiriktsarpalis removed blocking-release Priority:3 Work that is nice to have labels Sep 11, 2021
@am11 am11 mentioned this issue Sep 29, 2021
Merged
@jkotas
Copy link
Member

jkotas commented Sep 29, 2021 •
edited
Loading

@eiriktsarpalis, @sdmaclea does not work on .NET team anymore.

@danmoseley
Copy link
Member

Note that #57324 has a dump available and how_to_debug_dump.md to help open it. I think these time out within 30-60 days or so.

@am11
Copy link
Member

am11 commented Sep 29, 2021

Here is the stack:

(lldb) clrstack
OS Thread Id: 0x2a (1)
        Child SP               IP Call Site
0000007F82681010 0000000000000000 [DebuggerU2MCatchHandlerFrame: 0000007f82681010] 
0000007F82681198 0000000000000000 [HelperMethodFrame_PROTECTOBJ: 0000007f82681198] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Span`1<System.Object> ByRef, System.Signature, Boolean, Boolean)
0000007F82681330 0000007F10BF36C4 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) [/_/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs @ 431]
0000007F826813B0 0000007F12FBDB84 Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 257]
0000007F82681430 0000007F12FBD874 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681480 0000007F12FBD74C Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0[[System.__Canon, System.Private.CoreLib]].<InvokeTestMethodAsync>b__1()
0000007F826814D0 0000007F12FBD4C4 Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4.MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs @ 48]
0000007F82681520 0000007F12FBD364 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet]](<AggregateAsync>d__4 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681560 0000007F12FBD2AC Xunit.Sdk.ExecutionTimer.AggregateAsync(System.Func`1<System.Threading.Tasks.Task>)
0000007F826815A0 0000007F12FBD028 Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9.MoveNext() [/_/src/xunit.core/Sdk/ExceptionAggregator.cs @ 90]
0000007F826815F0 0000007F12FBCF24 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core]](<RunAsync>d__9 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681630 0000007F12FBCE6C Xunit.Sdk.ExceptionAggregator.RunAsync(System.Func`1<System.Threading.Tasks.Task>)
0000007F82681670 0000007F12FBBFCC Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]].MoveNext()
0000007F82681710 0000007F12FBBCA4 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681760 0000007F12FBBB80 Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].InvokeTestMethodAsync(System.Object)
0000007F826817B0 0000007F12FBAB24 Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 206]
0000007F82681870 0000007F12FBA714 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<RunAsync>b__47_0>d<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F826818C0 0000007F12FBA5F0 Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].<RunAsync>b__47_0()
0000007F82681920 0000007F12FBA33C Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]].MoveNext() [/_/src/xunit.core/Sdk/ExceptionAggregator.cs @ 107]
0000007F826819A0 0000007F12FBA22C System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.Decimal> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F826819E0 0000007F12FBA174 Xunit.Sdk.ExceptionAggregator.RunAsync[[System.Decimal, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.Decimal>>)
0000007F82681A20 0000007F10FD572C Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4.MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs @ 67]
0000007F82681A90 0000007F12FB2614 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]](<InvokeTestAsync>d__4 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681AD0 0000007F12FB2550 Xunit.Sdk.XunitTestRunner.InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)
0000007F82681B20 0000007F12FB1F2C Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
0000007F82681B90 0000007F12FB1A64 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681BE0 0000007F12FB1924 Xunit.Sdk.ExceptionAggregator.RunAsync[[System.__Canon, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>)
0000007F82681C30 0000007F12FB0D2C Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]].MoveNext()
0000007F82681D20 0000007F12FAFF1C System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__43<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681D70 0000007F12FAFDE8 Xunit.Sdk.TestRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
0000007F82681DD0 0000007F11548ED8 Xunit.Sdk.XunitTheoryTestCaseRunner+<RunTestAsync>d__11.MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTheoryTestCaseRunner.cs @ 144]
0000007F82681E80 0000007F11544B54 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTheoryTestCaseRunner+<RunTestAsync>d__11, xunit.execution.dotnet]](<RunTestAsync>d__11 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681EC0 0000007F11544ACC System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.XunitTheoryTestCaseRunner+<RunTestAsync>d__11, xunit.execution.dotnet]](<RunTestAsync>d__11 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82681EF0 0000007F11544A78 Xunit.Sdk.XunitTheoryTestCaseRunner.RunTestAsync()
0000007F82681F50 0000007F10FD1048 Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestCaseRunner.cs @ 82]
0000007F82682070 0000007F10FD0C2C System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F826820D0 0000007F10FD0B58 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682110 0000007F10FD0AC0 Xunit.Sdk.TestCaseRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
0000007F82682170 0000007F11524EC8 Xunit.Sdk.XunitTheoryTestCase.RunAsync(Xunit.Abstractions.IMessageSink, Xunit.Sdk.IMessageBus, System.Object[], Xunit.Sdk.ExceptionAggregator, System.Threading.CancellationTokenSource) [/_/src/xunit.execution/Sdk/Frameworks/XunitTheoryTestCase.cs @ 51]
0000007F826821D0 0000007F10FD0570 Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(Xunit.Sdk.IXunitTestCase) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestMethodRunner.cs @ 45]
0000007F82682220 0000007F10FD01EC Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs @ 136]
0000007F82682290 0000007F10FCFBFC System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F826822F0 0000007F10FCFB28 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682330 0000007F10FCFA94 Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestCasesAsync()
0000007F826823A0 0000007F10FCF078 Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs @ 106]
0000007F826824B0 0000007F10FCEC8C System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82682510 0000007F10FCEBB8 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682550 0000007F10FCEB20 Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
0000007F826825B0 0000007F10FCE6B0 Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(Xunit.Abstractions.ITestMethod, Xunit.Abstractions.IReflectionMethodInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Object[]) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestClassRunner.cs @ 168]
0000007F82682630 0000007F10FCA24C Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs @ 213]
0000007F826826D0 0000007F10FC9994 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82682730 0000007F10FC98C0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682770 0000007F10FC9420 Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestMethodsAsync()
0000007F826827E0 0000007F10FC782C Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs @ 171]
0000007F82682930 0000007F10FC6E9C System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__37<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82682990 0000007F10FC6DC8 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__37<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F826829D0 0000007F10FC6D30 Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
0000007F82682A30 0000007F10FC67DC Xunit.Sdk.XunitTestCollectionRunner.RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestCollectionRunner.cs @ 158]
0000007F82682AB0 0000007F10FC6100 Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs @ 130]
0000007F82682B30 0000007F10FC5714 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestClassesAsync>d__28<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82682B90 0000007F10FC5640 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestClassesAsync>d__28<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682BD0 0000007F10FC55AC Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestClassesAsync()
0000007F82682C40 0000007F10FC271C Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs @ 101]
0000007F82682D90 0000007F10FC1A04 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__27<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82682DF0 0000007F10FC18F8 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__27<System.__Canon> ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682E30 0000007F10FC1488 Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
0000007F82682E90 0000007F10FBFD10 Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Threading.CancellationTokenSource) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs @ 235]
0000007F82682EF0 0000007F10FBFC3C Xunit.Sdk.XunitTestAssemblyRunner+<>c__DisplayClass14_2.<RunTestCollectionsAsync>b__2() [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs @ 184]
0000007F82682F30 0000007F0E800BB8 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].InnerInvoke() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs @ 503]
0000007F82682F70 0000007F0E50F774 System.Threading.Tasks.Task+<>c.<.cctor>b__271_0(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2367]
0000007F82682F90 0000007F0E4E22A8 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
0000007F82682FE0 0000007F0E507214 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2327]
0000007F82683060 0000007F0E506FA0 System.Threading.Tasks.Task.ExecuteEntry() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2240]
0000007F82683080 0000007F0E516B24 System.Threading.Tasks.SynchronizationContextTaskScheduler+<>c.<.cctor>b__8_0(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskScheduler.cs @ 630]
0000007F826830A0 0000007F10FBFB7C Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object) [/_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs @ 106]
0000007F826830E0 0000007F10FBF5B0 Xunit.Sdk.MaxConcurrencySyncContext+<>c__DisplayClass11_0.<WorkerThreadProc>b__0(System.Object) [/_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs @ 96]
0000007F82683120 0000007F0E4E22A8 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
0000007F82683170 0000007F10FB9DDC Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc() [/_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs @ 96]
0000007F826831B0 0000007F10BC7388 Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__5_0(System.Object) [/_/src/common/XunitWorkerThread.cs @ 37]
0000007F826831F0 0000007F0E5076C8 System.Threading.Tasks.Task.InnerInvoke() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2387]
0000007F82683220 0000007F0E50F774 System.Threading.Tasks.Task+<>c.<.cctor>b__271_0(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2367]
0000007F82683240 0000007F0E4E22A8 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
0000007F82683290 0000007F0E507214 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2327]
0000007F82683310 0000007F0E50708C System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2265]
0000007F82683330 0000007F0E517648 System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs @ 35]
0000007F82683350 0000007F0E4CDF8C System.Threading.Thread.StartCallback() [/_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 105]
0000007F82683568 0000007f88365dcc [DebuggerU2MCatchHandlerFrame: 0000007f82683568]

@am11

This comment has been minimized.

Sign in to view
@jkotas
Copy link
Member

jkotas commented Sep 29, 2021

@am11 It looks like the crash is in the unmanaged runtime code. Are you able to get unmanaged stacktrace too?

@am11
Copy link
Member

am11 commented Sep 29, 2021 •
edited
Loading

Yup, it looks like:

(lldb) clrstack -f
OS Thread Id: 0x2a (1)
        Child SP               IP Call Site
0000007F82680970 0000000000000000 
0000007F82680970 0000007F1251FA1C 
0000007F82680980 0000007F1251F830 
0000007F826809F0 0000007F88365DCC libcoreclr.so!CallDescrWorkerInternal + 132 at /__w/1/s/src/coreclr/vm/arm64/calldescrworkerarm64.S:73
0000007F82680CD0 0000007F8811707C libcoreclr.so!CallDescrWorkerWithHandler(CallDescrData*, int) + 364 at /__w/1/s/src/coreclr/pal/inc/pal.h:4648
0000007F82680D20 0000007F88238030 libcoreclr.so!CallDescrWorkerReflectionWrapper(CallDescrData*, Frame*) + 128 at /__w/1/s/src/coreclr/pal/inc/pal.h:4648
0000007F82680D20 0000007F88238024 libcoreclr.so!CallDescrWorkerReflectionWrapper(CallDescrData*, Frame*) + 116 at /__w/1/s/src/coreclr/vm/reflectioninvocation.cpp:471
0000007F82680DA0 0000007F882398C8 libcoreclr.so!RuntimeMethodHandle::InvokeMethod(Object*, Span<OBJECTREF>*, SignatureNative*, bool, bool) + 4984 at /__w/1/s/src/coreclr/pal/inc/pal.h:4648
0000007F82681010                  [DebuggerU2MCatchHandlerFrame: 0000007f82681010] 
0000007F82681198                  [HelperMethodFrame_PROTECTOBJ: 0000007f82681198] System.Private.CoreLib.dll!System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Span`1<System.Object> ByRef, System.Signature, Boolean, Boolean)
0000007F82681330 0000007F10BF36C4 System.Private.CoreLib.dll!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) + 196 [/_/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs @ 431]
0000007F82681340 0000007F12FBDB84 
0000007F826813B0 0000007F12FBDB84 xunit.execution.dotnet.dll!Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]].MoveNext() + 596 [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 257]
0000007F826813C0 0000007F12FBD874 
0000007F82681430 0000007F12FBD874 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef) + 132 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681440 0000007F12FBD74C 
0000007F82681480 0000007F12FBD74C xunit.execution.dotnet.dll!Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0[[System.__Canon, System.Private.CoreLib]].<InvokeTestMethodAsync>b__1() + 108
0000007F82681490 0000007F12FBD4C4 
0000007F826814D0 0000007F12FBD4C4 xunit.execution.dotnet.dll!Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4.MoveNext() + 156 [/_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs @ 48]
0000007F826814E0 0000007F12FBD364 
0000007F82681520 0000007F12FBD364 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet]](<AggregateAsync>d__4 ByRef) + 84 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681530 0000007F12FBD2AC 
0000007F82681560 0000007F12FBD2AC xunit.execution.dotnet.dll!Xunit.Sdk.ExecutionTimer.AggregateAsync(System.Func`1<System.Threading.Tasks.Task>) + 52
0000007F82681570 0000007F12FBD028 
0000007F826815A0 0000007F12FBD028 xunit.core.dll!Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9.MoveNext() + 64 [/_/src/xunit.core/Sdk/ExceptionAggregator.cs @ 90]
0000007F826815B0 0000007F12FBCF24 
0000007F826815F0 0000007F12FBCF24 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core]](<RunAsync>d__9 ByRef) + 84 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681600 0000007F12FBCE6C 
0000007F82681630 0000007F12FBCE6C xunit.core.dll!Xunit.Sdk.ExceptionAggregator.RunAsync(System.Func`1<System.Threading.Tasks.Task>) + 52
0000007F82681640 0000007F12FBBFCC 
0000007F82681670 0000007F12FBBFCC xunit.execution.dotnet.dll!Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]].MoveNext() + 540
0000007F82681680 0000007F12FBBCA4 
0000007F82681710 0000007F12FBBCA4 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef) + 132 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681720 0000007F12FBBB80 
0000007F82681760 0000007F12FBBB80 xunit.execution.dotnet.dll!Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].InvokeTestMethodAsync(System.Object) + 112
0000007F82681770 0000007F12FBAB24 
0000007F826817B0 0000007F12FBAB24 xunit.execution.dotnet.dll!Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]].MoveNext() + 852 [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 206]
0000007F826817C0 0000007F12FBA714 
0000007F82681870 0000007F12FBA714 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<RunAsync>b__47_0>d<System.__Canon> ByRef) + 132 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681880 0000007F12FBA5F0 
0000007F826818C0 0000007F12FBA5F0 xunit.execution.dotnet.dll!Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].<RunAsync>b__47_0() + 112
0000007F826818D0 0000007F12FBA33C 
0000007F82681920 0000007F12FBA33C xunit.core.dll!Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]].MoveNext() + 76 [/_/src/xunit.core/Sdk/ExceptionAggregator.cs @ 107]
0000007F82681930 0000007F12FBA22C 
0000007F826819A0 0000007F12FBA22C System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.Decimal> ByRef) + 84 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F826819B0 0000007F12FBA174 
0000007F826819E0 0000007F12FBA174 xunit.core.dll!Xunit.Sdk.ExceptionAggregator.RunAsync[[System.Decimal, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.Decimal>>) + 52
0000007F826819F0 0000007F10FD572C 
0000007F82681A20 0000007F10FD572C xunit.execution.dotnet.dll!Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4.MoveNext() + 228 [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs @ 67]
0000007F82681A30 0000007F12FB2614 
0000007F82681A90 0000007F12FB2614 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]](<InvokeTestAsync>d__4 ByRef) + 84 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681AA0 0000007F12FB2550 
0000007F82681AD0 0000007F12FB2550 xunit.execution.dotnet.dll!Xunit.Sdk.XunitTestRunner.InvokeTestAsync(Xunit.Sdk.ExceptionAggregator) + 56
0000007F82681AE0 0000007F12FB1F2C 
0000007F82681B20 0000007F12FB1F2C xunit.core.dll!Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]].MoveNext() + 68
0000007F82681B30 0000007F12FB1A64 
0000007F82681B90 0000007F12FB1A64 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.__Canon> ByRef) + 132 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681BA0 0000007F12FB1924 
0000007F82681BE0 0000007F12FB1924 xunit.core.dll!Xunit.Sdk.ExceptionAggregator.RunAsync[[System.__Canon, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>) + 124
0000007F82681BF0 0000007F12FB0D2C 
0000007F82681C30 0000007F12FB0D2C xunit.execution.dotnet.dll!Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]].MoveNext() + 836
0000007F82681C40 0000007F12FAFF1C 
0000007F82681D20 0000007F12FAFF1C System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__43<System.__Canon> ByRef) + 132 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681D30 0000007F12FAFDE8 
0000007F82681D70 0000007F12FAFDE8 xunit.execution.dotnet.dll!Xunit.Sdk.TestRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() + 112
0000007F82681D80 0000007F11548ED8 
0000007F82681DD0 0000007F11548ED8 xunit.execution.dotnet.dll!Xunit.Sdk.XunitTheoryTestCaseRunner+<RunTestAsync>d__11.MoveNext() + 224 [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTheoryTestCaseRunner.cs @ 144]
0000007F82681DE0 0000007F11544B54 
0000007F82681E80 0000007F11544B54 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTheoryTestCaseRunner+<RunTestAsync>d__11, xunit.execution.dotnet]](<RunTestAsync>d__11 ByRef) + 100 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82681E90 0000007F11544ACC 
0000007F82681EC0 0000007F11544ACC System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.XunitTheoryTestCaseRunner+<RunTestAsync>d__11, xunit.execution.dotnet]](<RunTestAsync>d__11 ByRef) + 28 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82681ED0 0000007F11544A78 
0000007F82681EF0 0000007F11544A78 xunit.execution.dotnet.dll!Xunit.Sdk.XunitTheoryTestCaseRunner.RunTestAsync() + 96
0000007F82681F00 0000007F10FD1048 
0000007F82681F50 0000007F10FD1048 xunit.execution.dotnet.dll!Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]].MoveNext() + 560 [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestCaseRunner.cs @ 82]
0000007F82681F60 0000007F10FD0C2C 
0000007F82682070 0000007F10FD0C2C System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef) + 180 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82682080 0000007F10FD0B58 
0000007F826820D0 0000007F10FD0B58 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef) + 96 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F826820E0 0000007F10FD0AC0 
0000007F82682110 0000007F10FD0AC0 xunit.execution.dotnet.dll!Xunit.Sdk.TestCaseRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() + 168
0000007F82682120 0000007F11524EC8 
0000007F82682170 0000007F11524EC8 xunit.execution.dotnet.dll!Xunit.Sdk.XunitTheoryTestCase.RunAsync(Xunit.Abstractions.IMessageSink, Xunit.Sdk.IMessageBus, System.Object[], Xunit.Sdk.ExceptionAggregator, System.Threading.CancellationTokenSource) + 176 [/_/src/xunit.execution/Sdk/Frameworks/XunitTheoryTestCase.cs @ 51]
0000007F82682188 0000007F10FD0570 
0000007F826821D0 0000007F10FD0570 xunit.execution.dotnet.dll!Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(Xunit.Sdk.IXunitTestCase) + 168 [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestMethodRunner.cs @ 45]
0000007F826821E0 0000007F10FD01EC 
0000007F82682220 0000007F10FD01EC xunit.execution.dotnet.dll!Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]].MoveNext() + 324 [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs @ 136]
0000007F82682230 0000007F10FCFBFC 
0000007F82682290 0000007F10FCFBFC System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef) + 180 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F826822A0 0000007F10FCFB28 
0000007F826822F0 0000007F10FCFB28 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef) + 96 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682300 0000007F10FCFA94 
0000007F82682330 0000007F10FCFA94 xunit.execution.dotnet.dll!Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestCasesAsync() + 172
0000007F82682340 0000007F10FCF078 
0000007F826823A0 0000007F10FCF078 xunit.execution.dotnet.dll!Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]].MoveNext() + 344 [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs @ 106]
0000007F826823B0 0000007F10FCEC8C 
0000007F826824B0 0000007F10FCEC8C System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef) + 180 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F826824C0 0000007F10FCEBB8 
0000007F82682510 0000007F10FCEBB8 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef) + 96 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682520 0000007F10FCEB20 
0000007F82682550 0000007F10FCEB20 xunit.execution.dotnet.dll!Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() + 168
0000007F82682560 0000007F10FCE6B0 
0000007F826825B0 0000007F10FCE6B0 xunit.execution.dotnet.dll!Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(Xunit.Abstractions.ITestMethod, Xunit.Abstractions.IReflectionMethodInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Object[]) + 224 [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestClassRunner.cs @ 168]
0000007F826825D0 0000007F10FCA24C 
0000007F82682630 0000007F10FCA24C xunit.execution.dotnet.dll!Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]].MoveNext() + 1036 [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs @ 213]
0000007F82682640 0000007F10FC9994 
0000007F826826D0 0000007F10FC9994 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef) + 180 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F826826E0 0000007F10FC98C0 
0000007F82682730 0000007F10FC98C0 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef) + 96 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682740 0000007F10FC9420 
0000007F82682770 0000007F10FC9420 xunit.execution.dotnet.dll!Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestMethodsAsync() + 168
0000007F82682780 0000007F10FC782C 
0000007F826827E0 0000007F10FC782C xunit.execution.dotnet.dll!Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]].MoveNext() + 612 [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs @ 171]
0000007F826827F0 0000007F10FC6E9C 
0000007F82682930 0000007F10FC6E9C System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__37<System.__Canon> ByRef) + 180 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82682940 0000007F10FC6DC8 
0000007F82682990 0000007F10FC6DC8 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__37<System.__Canon> ByRef) + 96 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F826829A0 0000007F10FC6D30 
0000007F826829D0 0000007F10FC6D30 xunit.execution.dotnet.dll!Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() + 168
0000007F826829E0 0000007F10FC67DC 
0000007F82682A30 0000007F10FC67DC xunit.execution.dotnet.dll!Xunit.Sdk.XunitTestCollectionRunner.RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>) + 228 [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestCollectionRunner.cs @ 158]
0000007F82682A50 0000007F10FC6100 
0000007F82682AB0 0000007F10FC6100 xunit.execution.dotnet.dll!Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]].MoveNext() + 896 [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs @ 130]
0000007F82682AC0 0000007F10FC5714 
0000007F82682B30 0000007F10FC5714 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestClassesAsync>d__28<System.__Canon> ByRef) + 180 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82682B40 0000007F10FC5640 
0000007F82682B90 0000007F10FC5640 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestClassesAsync>d__28<System.__Canon> ByRef) + 96 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682BA0 0000007F10FC55AC 
0000007F82682BD0 0000007F10FC55AC xunit.execution.dotnet.dll!Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestClassesAsync() + 172
0000007F82682BE0 0000007F10FC271C 
0000007F82682C40 0000007F10FC271C xunit.execution.dotnet.dll!Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]].MoveNext() + 612 [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs @ 101]
0000007F82682C50 0000007F10FC1A04 
0000007F82682D90 0000007F10FC1A04 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__27<System.__Canon> ByRef) + 180 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
0000007F82682DA0 0000007F10FC18F8 
0000007F82682DF0 0000007F10FC18F8 System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__27<System.__Canon> ByRef) + 96 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 36]
0000007F82682E00 0000007F10FC1488 
0000007F82682E30 0000007F10FC1488 xunit.execution.dotnet.dll!Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() + 168
0000007F82682E40 0000007F10FBFD10 
0000007F82682E90 0000007F10FBFD10 xunit.execution.dotnet.dll!Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Threading.CancellationTokenSource) + 176 [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs @ 235]
0000007F82682EA0 0000007F10FBFC3C 
0000007F82682EF0 0000007F10FBFC3C xunit.execution.dotnet.dll!Xunit.Sdk.XunitTestAssemblyRunner+<>c__DisplayClass14_2.<RunTestCollectionsAsync>b__2() + 148 [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs @ 184]
0000007F82682F00 0000007F0E800BB8 
0000007F82682F30 0000007F0E800BB8 System.Private.CoreLib.dll!System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].InnerInvoke() + 152 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs @ 503]
0000007F82682F40 0000007F0E50F774 
0000007F82682F70 0000007F0E50F774 System.Private.CoreLib.dll!System.Threading.Tasks.Task+<>c.<.cctor>b__271_0(System.Object) + 116 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2367]
0000007F82682F80 0000007F0E4E22A8 
0000007F82682F90 0000007F0E4E22A8 System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) + 136 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
0000007F82682FA0 0000007F0E507214 
0000007F82682FE0 0000007F0E507214 System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) + 244 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2327]
0000007F82682FF0 0000007F0E506FA0 
0000007F82683060 0000007F0E506FA0 System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteEntry() + 160 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2240]
0000007F82683070 0000007F0E516B24 
0000007F82683080 0000007F0E516B24 System.Private.CoreLib.dll!System.Threading.Tasks.SynchronizationContextTaskScheduler+<>c.<.cctor>b__8_0(System.Object) + 132 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskScheduler.cs @ 630]
0000007F82683090 0000007F10FBFB7C 
0000007F826830A0 0000007F10FBFB7C xunit.execution.dotnet.dll!Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object) + 76 [/_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs @ 106]
0000007F826830B0 0000007F10FBF5B0 
0000007F826830E0 0000007F10FBF5B0 xunit.execution.dotnet.dll!Xunit.Sdk.MaxConcurrencySyncContext+<>c__DisplayClass11_0.<WorkerThreadProc>b__0(System.Object) + 96 [/_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs @ 96]
0000007F826830F0 0000007F0E4E22A8 
0000007F82683120 0000007F0E4E22A8 System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) + 136 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
0000007F82683130 0000007F10FB9DDC 
0000007F82683170 0000007F10FB9DDC xunit.execution.dotnet.dll!Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc() + 404 [/_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs @ 96]
0000007F82683180 0000007F10BC7388 
0000007F826831B0 0000007F10BC7388 xunit.execution.dotnet.dll!Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__5_0(System.Object) + 88 [/_/src/common/XunitWorkerThread.cs @ 37]
0000007F826831C0 0000007F0E5076C8 
0000007F826831F0 0000007F0E5076C8 System.Private.CoreLib.dll!System.Threading.Tasks.Task.InnerInvoke() + 200 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2387]
0000007F82683200 0000007F0E50F774 
0000007F82683220 0000007F0E50F774 System.Private.CoreLib.dll!System.Threading.Tasks.Task+<>c.<.cctor>b__271_0(System.Object) + 116 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2367]
0000007F82683230 0000007F0E4E22A8 
0000007F82683240 0000007F0E4E22A8 System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) + 136 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
0000007F82683250 0000007F0E507214 
0000007F82683290 0000007F0E507214 System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) + 244 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2327]
0000007F826832A0 0000007F0E50708C 
0000007F82683310 0000007F0E50708C System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) + 140 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2265]
0000007F82683320 0000007F0E517648 
0000007F82683330 0000007F0E517648 System.Private.CoreLib.dll!System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object) + 136 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs @ 35]
0000007F82683340 0000007F0E4CDF8C 
0000007F82683350 0000007F0E4CDF8C System.Private.CoreLib.dll!System.Threading.Thread.StartCallback() + 236 [/_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 105]
0000007F82683360 0000007F88365DCC libcoreclr.so!CallDescrWorkerInternal + 132 at /__w/1/s/src/coreclr/vm/arm64/calldescrworkerarm64.S:73
0000007F826833A0 0000007F8811707C libcoreclr.so!CallDescrWorkerWithHandler(CallDescrData*, int) + 364 at /__w/1/s/src/coreclr/pal/inc/pal.h:4648
0000007F826833F0 0000007F8811739C libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) + 220 at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218
0000007F826834D0 0000007F8813A664 libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) + 224 at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0
0000007F82683540 0000007F880C2434 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 564 at /__w/1/s/src/coreclr/vm/threads.cpp:7349
0000007F82683540 0000007F880C242C libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 556 at /__w/1/s/src/coreclr/vm/threads.cpp:7312
0000007F82683540 0000007F880C2308 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 264 at /__w/1/s/src/coreclr/vm/threads.inl:37
0000007F82683540 0000007F880C2304 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 260 at /__w/1/s/src/coreclr/vm/threads.cpp:7514
0000007F82683540 0000007F880C22C0 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 192 at /__w/1/s/src/coreclr/vm/threads.cpp:7512
0000007F82683568                  [DebuggerU2MCatchHandlerFrame: 0000007f82683568] 
0000007F826836C0 0000007F880C2B40 libcoreclr.so!ManagedThreadBase::KickOff(void (*)(void*), void*) + 48 at /__w/1/s/src/coreclr/vm/threads.cpp:0
0000007F826836C0 0000007F880C2B30 libcoreclr.so!ManagedThreadBase::KickOff(void (*)(void*), void*) + 32 at /__w/1/s/src/coreclr/vm/threads.cpp:7287
0000007F826836F0 0000007F8813A828 libcoreclr.so!ThreadNative::KickOffThread(void*) + 304 at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230
0000007F82683770 0000007F88597CFC libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) + 888 at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1829
0000007F82683820 0000007F88A25BAC ld-musl-aarch64.so.1 + -1
0000007F82683830 0000007F88A243EC ld-musl-aarch64.so.1 + -1

@am11
Copy link
Member

am11 commented Sep 29, 2021

Updated ⬆️ with clrstack -f output (which captures both managed and native frames).

@danmoseley

This comment has been minimized.

Sign in to view
@jkotas
Copy link
Member

jkotas commented Sep 30, 2021

The stacktrace of the crash is

IP=0, but it should have been System.Text.Json.Utf8JsonReader.Read()
System.Text.Json.Tests.Utf8JsonReaderTests.ReadWithCommentsHelper(System.Text.Json.Utf8JsonReader ByRef, Int32, Boolean)
System.Text.Json.Tests.Utf8JsonReaderTests.ReadJsonStringsWithCommentsAndTrailingCommas(System.String)

So something bad happened while executing Utf8JsonReader.Read method and we failed to return from it; or we have never reached it in the first place

@jkotas
Copy link
Member

jkotas commented Sep 30, 2021

Based on the content of the stack, there was an exception throw from Utf8JsonReader.Read, the catch block got executed, but we failed to resume the normal execution in ReadWithCommentsHelper after the catch block (jumped to IP=0 instead). I am not able to tell more from the dump.

Another interesting detail is that there is a GC thread suspension in progress that may be related and triggering the problem.

@janvorli Could you please take a look?

@jkotas jkotas added area-ExceptionHandling-coreclr os-linux-musl Linux distributions using musl library. and removed area-System.Text.Json labels Sep 30, 2021
@jkotas
Copy link
Member

jkotas commented Sep 30, 2021

I see this in RtlRestoreContext:

    ldr x17, [x16, CONTEXT_Sp]
    mov sp, x17
    ldr x17, [x16, CONTEXT_Pc]

Can this be a problem if the signal gets delivered right after mov sp, x17, x16 points to the portion of the stack that was just unwound and [x16, CONTEXT_Pc] gets overwritten by the signal handler?

@jkotas jkotas added the bug label Sep 30, 2021
@janvorli
Copy link
Member

Yes, it seems that this is problematic w.r.t. to interruptions of all kinds, including async signals. I'll take a look.

@jkotas jkotas mentioned this issue Oct 1, 2021
Merged
@jkotas jkotas closed this as completed Oct 1, 2021
@danmoseley
Copy link
Member

Nice to see a successful diagnosis based on a test run dump. Thanks folks!

@ghost ghost locked as resolved and limited conversation to collaborators Dec 2, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm64 area-ExceptionHandling-coreclr bug os-linux-musl Linux distributions using musl library.
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
6 participants
@eiriktsarpalis @am11 @danmoseley @jkotas @ViktorHofer @janvorli