Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Collections.Concurrent.Tests crashing in CI #45517

Closed
ViktorHofer opened this issue Dec 3, 2020 · 8 comments
Closed

System.Collections.Concurrent.Tests crashing in CI #45517

ViktorHofer opened this issue Dec 3, 2020 · 8 comments
Labels
area-VM-coreclr tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Milestone
6.0.0

Comments

@ViktorHofer
Copy link
Member

ViktorHofer commented Dec 3, 2020 •
edited by runfoapp bot
Loading

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=905607&view=ms.vss-test-web.build-test-results-tab&runId=28901114&resultId=182589&paneView=attachments

Configuration: net6.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open

how-to-debug-dump.md:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-b6482f3963824bb38a/System.Collections.Concurrent.Tests/how-to-debug-dump.md?sv=2019-07-07&se=2020-12-22T10%3A40%3A07Z&sr=c&sp=rl&sig=l5N76%2FlDXHLoRkWIFox8OOiSkZPdUawXGM9N0cBe86A%3D

core.1000.22024:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-b6482f3963824bb38a/System.Collections.Concurrent.Tests/core.1000.22024?sv=2019-07-07&se=2020-12-22T10%3A40%3A07Z&sr=c&sp=rl&sig=l5N76%2FlDXHLoRkWIFox8OOiSkZPdUawXGM9N0cBe86A%3D

console.8cc118e5.log:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-b6482f3963824bb38a/System.Collections.Concurrent.Tests/console.8cc118e5.log?sv=2019-07-07&se=2020-12-22T10%3A40%3A07Z&sr=c&sp=rl&sig=l5N76%2FlDXHLoRkWIFox8OOiSkZPdUawXGM9N0cBe86A%3D

Runfo Tracking Issue: system.collections.concurrent.tests crashes

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1082899 runtime PR 51099 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88 console.log core dump
1072066 runtime PR 50364 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88 console.log core dump runclient.py
1072066 runtime PR 50364 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1071508 runtime PR 50364 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88 console.log core dump
1071508 runtime PR 50364 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1067051 runtime PR 50479 net6.0-OSX-Debug-x64-Mono_release-OSX.1014.Amd64.Open console.log core dump runclient.py
1067051 runtime PR 50479 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log core dump
1067051 runtime PR 50479 net6.0-Linux-Debug-x64-mono_interpreter_release-Debian.9.Amd64.Open console.log core dump
1067051 runtime PR 50479 net6.0-Linux-Debug-x64-Mono_release-(Centos.8.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:centos-8-helix-20201229003624-c1bf759 console.log core dump
1067051 runtime PR 50479 net6.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open console.log core dump
1067051 runtime PR 50479 net6.0-Linux-Debug-x64-Mono_release-(Debian.10.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-amd64-bfcd90a-20200121150006 console.log core dump
1067051 runtime PR 50479 net6.0-Linux-Debug-x64-Mono_release-Ubuntu.1604.Amd64.Open console.log core dump
1067051 runtime PR 50479 net6.0-Linux-Debug-x64-Mono_release-Ubuntu.1804.Amd64.Open console.log core dump
1067051 runtime PR 50479 net6.0-Linux-Debug-x64-Mono_release-SLES.15.Amd64.Open console.log core dump
1067051 runtime PR 50479 net6.0-Linux-Debug-x64-Mono_release-(Fedora.30.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:fedora-30-helix-20200512010621-4f8cef7 console.log core dump
1067051 runtime PR 50479 net6.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-20210106155927-56c6673 console.log core dump
1066426 runtime PR 50364 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88 console.log core dump
1066426 runtime PR 50364 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1059736 runtime PR 50232 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open console.log core dump runclient.py
1059736 runtime PR 50232 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log runclient.py
1059736 runtime PR 50232 net6.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open console.log runclient.py
1059736 runtime PR 50232 net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1059736 runtime PR 50232 net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59 console.log core dump runclient.py
1059736 runtime PR 50232 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump runclient.py
1059736 runtime PR 50232 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88 console.log core dump runclient.py
1059736 runtime PR 50232 net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e console.log core dump runclient.py
1050603 runtime PR 49990 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log
1050314 runtime PR 48601 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log runclient.py
1050243 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log
1050243 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open console.log
1047146 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log runclient.py
1047146 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open console.log
1047146 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log
1047146 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open console.log
1046026 runtime PR 49740 net6.0-Linux-Debug-x64-mono_interpreter_release-Debian.9.Amd64.Open console.log core dump
1046026 runtime PR 49740 net6.0-Linux-Debug-x64-Mono_release-(Centos.8.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:centos-8-helix-20201229003624-c1bf759 console.log core dump
1046026 runtime PR 49740 net6.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open console.log core dump runclient.py
1046026 runtime PR 49740 net6.0-Linux-Debug-x64-Mono_release-(Debian.10.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-amd64-bfcd90a-20200121150006 console.log core dump
1046026 runtime PR 49740 net6.0-Linux-Debug-x64-Mono_release-Ubuntu.1604.Amd64.Open console.log core dump
1046026 runtime PR 49740 net6.0-Linux-Debug-x64-Mono_release-Ubuntu.1804.Amd64.Open console.log core dump
1046026 runtime PR 49740 net6.0-Linux-Debug-x64-Mono_release-SLES.15.Amd64.Open console.log core dump
1046026 runtime PR 49740 net6.0-Linux-Debug-x64-Mono_release-(Fedora.30.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:fedora-30-helix-20200512010621-4f8cef7 console.log core dump
1046026 runtime PR 49740 net6.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-20210106155927-56c6673 console.log core dump
1046026 runtime PR 49740 net6.0-OSX-Debug-x64-Mono_release-OSX.1014.Amd64.Open console.log
1046026 runtime PR 49740 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log core dump
1045420 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log runclient.py
1045420 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open console.log
1042619 runtime Rolling net5.0-Linux-Release-x64-Mono_release-SLES.15.Amd64.Open console.log runclient.py
1041040 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log
1041040 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open console.log
1041040 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log
1041040 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open console.log
1041040 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log
1041040 runtime PR 49635 net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open console.log
1039858 runtime PR 43706 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88 console.log core dump
1039858 runtime PR 43706 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1038213 runtime PR 49511 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log core dump test results
1038213 runtime PR 49511 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log core dump test results
1033540 runtime PR 48601 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log runclient.py
1027303 runtime PR 49072 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log
1027303 runtime PR 49072 net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open console.log
1027001 runtime PR 49072 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log runclient.py
1026754 runtime PR 49256 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open console.log runclient.py
1026754 runtime PR 49256 net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e console.log core dump
1026754 runtime PR 49256 net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump
1026754 runtime PR 49256 net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59 console.log core dump
1025834 runtime PR 49260 net6.0-OSX-Debug-x64-Mono_release-OSX.1014.Amd64.Open console.log core dump
1025834 runtime PR 49260 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log core dump
1025834 runtime PR 49260 net6.0-Linux-Debug-x64-Mono_release-(Centos.8.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:centos-8-helix-20201229003624-c1bf759 console.log core dump
1025834 runtime PR 49260 net6.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open console.log core dump
1025834 runtime PR 49260 net6.0-Linux-Debug-x64-Mono_release-(Debian.10.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-amd64-bfcd90a-20200121150006 console.log core dump
1025834 runtime PR 49260 net6.0-Linux-Debug-x64-Mono_release-Ubuntu.1604.Amd64.Open console.log core dump
1025834 runtime PR 49260 net6.0-Linux-Debug-x64-Mono_release-Ubuntu.1804.Amd64.Open console.log core dump
1025834 runtime PR 49260 net6.0-Linux-Debug-x64-Mono_release-SLES.15.Amd64.Open console.log core dump
1025834 runtime PR 49260 net6.0-Linux-Debug-x64-Mono_release-(Fedora.30.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:fedora-30-helix-20200512010621-4f8cef7 console.log core dump
1025834 runtime PR 49260 net6.0-windows-Debug-x64-Mono_release-Windows.81.Amd64.Open console.log
1025834 runtime PR 49260 net6.0-windows-Debug-x64-Mono_release-Windows.10.Amd64.Server19H1.ES.Open console.log
1025677 runtime PR 49257 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log
1025677 runtime PR 49257 net6.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open console.log
1025677 runtime PR 49257 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open console.log
1025677 runtime PR 49257 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88 console.log core dump
1025677 runtime PR 49257 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log
1025677 runtime PR 49257 net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e console.log core dump runclient.py
1025677 runtime PR 49257 net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump
1025677 runtime PR 49257 net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59 console.log core dump
1024591 runtime PR 48601 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log runclient.py
1024540 runtime PR 47269 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump runclient.py
1024540 runtime PR 47269 net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e console.log core dump
1024540 runtime PR 47269 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88 console.log core dump
1024540 runtime PR 47269 net6.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open console.log
1022669 runtime PR 49072 net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open console.log
1022669 runtime PR 49072 net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open console.log
1019817 runtime PR 47864 net6.0-OSX-Debug-arm64-Mono_release-OSX.1100.ARM64.Open console.log core dump runclient.py
1017879 runtime PR 48923 net6.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open console.log runclient.py
1016780 runtime PR 48908 net6.0-OSX-Debug-x64-Mono_release-OSX.1014.Amd64.Open console.log runclient.py
1016780 runtime PR 48908 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log core dump
1016780 runtime PR 48908 net6.0-Linux-Debug-x64-Mono_release-(Centos.8.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:centos-8-helix-20201229003624-c1bf759 console.log core dump
1016780 runtime PR 48908 net6.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open console.log core dump
1016780 runtime PR 48908 net6.0-Linux-Debug-x64-Mono_release-(Debian.10.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-amd64-bfcd90a-20200121150006 console.log core dump
1016780 runtime PR 48908 net6.0-Linux-Debug-x64-Mono_release-Ubuntu.1604.Amd64.Open console.log core dump

Displaying 100 of 143 results

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
1 1 16
@ViktorHofer ViktorHofer added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-System.Collections labels Dec 3, 2020
@ghost
Copy link

ghost commented Dec 3, 2020

Tagging subscribers to this area: @eiriktsarpalis
See info in area-owners.md if you want to be subscribed.

Issue Details

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=905607&view=ms.vss-test-web.build-test-results-tab&runId=28901114&resultId=182589&paneView=attachments

Configuration: net6.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open

how-to-debug-dump.md:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-b6482f3963824bb38a/System.Collections.Concurrent.Tests/how-to-debug-dump.md?sv=2019-07-07&se=2020-12-22T10%3A40%3A07Z&sr=c&sp=rl&sig=l5N76%2FlDXHLoRkWIFox8OOiSkZPdUawXGM9N0cBe86A%3D

core.1000.22024:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-b6482f3963824bb38a/System.Collections.Concurrent.Tests/core.1000.22024?sv=2019-07-07&se=2020-12-22T10%3A40%3A07Z&sr=c&sp=rl&sig=l5N76%2FlDXHLoRkWIFox8OOiSkZPdUawXGM9N0cBe86A%3D

console.8cc118e5.log:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-b6482f3963824bb38a/System.Collections.Concurrent.Tests/console.8cc118e5.log?sv=2019-07-07&se=2020-12-22T10%3A40%3A07Z&sr=c&sp=rl&sig=l5N76%2FlDXHLoRkWIFox8OOiSkZPdUawXGM9N0cBe86A%3D

Author: ViktorHofer
Assignees: -
Labels:

area-System.Collections, blocking-clean-ci
Milestone: -

@ViktorHofer
Copy link
Member Author

cc @jkotas in case you are interested in the dump

@jkotas
Copy link
Member

jkotas commented Dec 3, 2020

Crash during background GC handle table scan:

(lldb) bt
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV
  * frame #0: 0x00007f0e1aef3ed0 libcoreclr.so`WKS::gc_heap::background_mark_simple1(unsigned char*) [inlined] MethodTable::GetFlag(this=<unavailable>, flag=enum_flag_HasComponentSize) const at methodtable.h:3705:16
    frame #1: 0x00007f0e1aef3ed0 libcoreclr.so`WKS::gc_heap::background_mark_simple1(unsigned char*) [inlined] MethodTable::HasComponentSize(this=<unavailable>) const at methodtable.h:1688
    frame #2: 0x00007f0e1aef3ed0 libcoreclr.so`WKS::gc_heap::background_mark_simple1(unsigned char*) [inlined] WKS::my_get_size(ob=<unavailable>) at gc.cpp:9693
    frame #3: 0x00007f0e1aef3ec9 libcoreclr.so`WKS::gc_heap::background_mark_simple1(oo="i�\xa2\r\x7f") at gc.cpp:20333
    frame #4: 0x00007f0e1aef4882 libcoreclr.so`WKS::gc_heap::background_promote(Object**, ScanContext*, unsigned int) at gc.cpp:20528:17
    frame #5: 0x00007f0e1aef482e libcoreclr.so`WKS::gc_heap::background_promote(ppObject=0x00007f0e1c966110, sc=<unavailable>, flags=<unavailable>) at gc.cpp:20608
    frame #6: 0x00007f0e1ae8245d libcoreclr.so`BlockScanBlocksWithoutUserData(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*) [inlined] ScanConsecutiveHandlesWithoutUserData(pValue=0x00007f0e1c966110, pLast=0x00007f0e1c966200, pInfo=0x00007f0d427fbb70, (null)=<unavailable>) at handletablescan.cpp:443:13
    frame #7: 0x00007f0e1ae82424 libcoreclr.so`BlockScanBlocksWithoutUserData(pSegment=0x00007f0e1c960000, uBlock=40, uCount=1, pInfo=0x00007f0d427fbb70) at handletablescan.cpp:555
    frame #8: 0x00007f0e1ae834b0 libcoreclr.so`xxxTableScanQueuedBlocksAsync(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) [inlined] ProcessScanQNode(pAsyncInfo=<unavailable>, pQNode=<unavailable>) at handletablescan.cpp:1231:9
    frame #9: 0x00007f0e1ae83482 libcoreclr.so`xxxTableScanQueuedBlocksAsync(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1201
    frame #10: 0x00007f0e1ae83436 libcoreclr.so`xxxTableScanQueuedBlocksAsync(pTable=<unavailable>, pSegment=<unavailable>, pCrstHolder=0x00007f0d427fbba0)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1312
    frame #11: 0x00007f0e1ae842ca libcoreclr.so`xxxTableScanHandlesAsync(HandleTable*, unsigned int const*, unsigned int, TableSegment* (*)(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*), void (*)(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*), ScanCallbackInfo*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) [inlined] xxxAsyncSegmentIterator(pTable=0x00005636d227bea0, pPrevSegment=<unavailable>, pCrstHolder=0x00007f0d427fbba0)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1537:9
    frame #12: 0x00007f0e1ae842a6 libcoreclr.so`xxxTableScanHandlesAsync(HandleTable*, unsigned int const*, unsigned int, TableSegment* (*)(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*), void (*)(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*), ScanCallbackInfo*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) [inlined] TableScanHandles(pTable=<unavailable>, puType=<unavailable>, uTypeCount=<unavailable>, pfnSegmentIterator=<unavailable>, pfnBlockHandler=<unavailable>, pInfo=<unavailable>, pCrstHolder=<unavailable>)(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*), void (*)(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*), ScanCallbackInfo*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1699
    frame #13: 0x00007f0e1ae83e32 libcoreclr.so`xxxTableScanHandlesAsync(pTable=0x00005636d227bea0, puType=<unavailable>, uTypeCount=<unavailable>, pfnSegmentIterator=<unavailable>, pfnBlockHandler=<unavailable>, pInfo=<unavailable>, pCrstHolder=0x00007f0d427fbba0)(HandleTable*, TableSegment*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*), void (*)(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*), ScanCallbackInfo*, Holder<CrstBase*, &(CrstBase::AcquireLock(CrstBase*)), &(CrstBase::ReleaseLock(CrstBase*)), 0ul, &(int CompareDefault<CrstBase*>(CrstBase*, CrstBase*)), true>*) at handletablescan.cpp:1781
    frame #14: 0x00007f0e1ae7ee71 libcoreclr.so`HndScanHandlesForGC(hTable=0x00005636d227bea0, scanProc=(libcoreclr.so`PromoteObject(Object**, unsigned long*, unsigned long, unsigned long) at objecthandle.cpp:305:5), param1=139694926969976, param2=139698558158720, types=0x00007f0d427fbc28, typeCount=<unavailable>, condemned=2, maxgen=2, flags=2)(Object**, unsigned long*, unsigned long, unsigned long), unsigned long, unsigned long, unsigned int const*, unsigned int, unsigned int, unsigned int, unsigned int) at handletable.cpp:789:9
    frame #15: 0x00007f0e1ae85dfd libcoreclr.so`Ref_TraceNormalRoots(condemned=2, maxgen=2, sc=0x00007f0d427fbc78, fn=(libcoreclr.so`WKS::gc_heap::background_promote(Object**, ScanContext*, unsigned int) at gc.cpp:20553))(Object**, ScanContext*, unsigned int)) at objecthandle.cpp:1059:21
    frame #16: 0x00007f0e1aee7504 libcoreclr.so`WKS::gc_heap::background_mark_phase() at gc.cpp:27932:9
    frame #17: 0x00007f0e1aee6400 libcoreclr.so`WKS::gc_heap::gc1() at gc.cpp:17500:13
    frame #18: 0x00007f0e1af013ac libcoreclr.so`WKS::gc_heap::bgc_thread_function() at gc.cpp:29072:9
    frame #19: 0x00007f0e1adac366 libcoreclr.so`(anonymous namespace)::CreateSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::__invoke(void*) [inlined] (anonymous namespace)::CreateSuspendableThread(this=<unavailable>, argument=<unavailable>)(void*), void*, char16_t const*)::$_0::operator()(void*) const at gcenv.ee.cpp:1376:17
    frame #20: 0x00007f0e1adac301 libcoreclr.so`(anonymous namespace)::CreateSuspendableThread(argument=<unavailable>)(void*), void*, char16_t const*)::$_0::__invoke(void*) at gcenv.ee.cpp:1356

@jkotas
Copy link
Member

jkotas commented Dec 3, 2020

GC heap is corrupted. The corrupted objects are System.Xml.Linq.XAttributes used by xunit harness for report.

(lldb) verifyheap
object 00007f0d6c2bffc0 contains free object 00007F0D6C2BFE10 at 00007F0D6C2BFFD8
Last good object: 00007F0D6C2BFE10.
object 00007f0d6c2c7d20 contains free object 00007F0D6C2C7D58 at 00007F0D6C2C7D48
Last good object: 00007F0D6C2C7CE0.
object 00007f0d6c2c83c8 contains free object 00007F0D6C2C8400 at 00007F0D6C2C83E0
Last good object: 00007F0D6C2C8390.
object 00007f0d6c2c89a0 contains free object 00007F0D6C2C89D8 at 00007F0D6C2C89B8
Last good object: 00007F0D6C2C8818.

(lldb) dumpobj 00007f0d6c2bffc0
Name:        System.Xml.Linq.XAttribute
MethodTable: 00007f0da202d968
EEClass:     00007f0da20354e8
Size:        56(0x38) bytes
File:        /home/helixbot/work/A19B08C0/p/shared/Microsoft.NETCore.App/6.0.0/System.Private.Xml.Linq.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f0da1cdeb58  40001cf        8 ...l.Linq.XContainer  0 instance 00007f0d6c2bc4a8 parent
00007f0da19d0c18  40001d0       10        System.Object  0 instance 0000000000000000 annotations
00007f0da202d968  400006a       18 ...l.Linq.XAttribute  0 instance 00007f0d6c2bfe10 next
00007f0da202d418  400006b       20 ...em.Xml.Linq.XName  0 instance 00007f0d6c1c9820 name
00007f0da1a97c28  400006c       28        System.String  0 instance 00007f0d6c2bfff8 value
(lldb) gcroot 00007f0d6c2bffc0
Thread 5608:
    00007FFD9F287900 00007F0DA1F58113 Xunit.ConsoleClient.ConsoleRunner.ExecuteAssembly(System.Object, Xunit.XunitProjectAssembly, Boolean, Boolean, System.Nullable`1<Boolean>, System.Nullable`1<Int32>, Boolean, Boolean, System.Nullable`1<Xunit.AppDomainSupport>, Boolean, Boolean, Xunit.XunitFilters, Boolean)
        r14:
            ->  00007F0D6C1B0560 Xunit.DelegatingLongRunningTestDetectionSink
            ->  00007F0D6C1B03F0 Xunit.DelegatingXmlCreationSink
            ->  00007F0D6C1B0480 System.Collections.Generic.Dictionary`2[[System.Guid, System.Private.CoreLib],[System.Xml.Linq.XElement, System.Private.Xml.Linq]]
            ->  00007F0D6C3D9240 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, System.Private.CoreLib],[System.Xml.Linq.XElement, System.Private.Xml.Linq]][]
            ->  00007F0D6C2BC4A8 System.Xml.Linq.XElement
            ->  00007F0D6C2BFFC0 System.Xml.Linq.XAttribute

Found 1 unique roots (run 'gcroot -all' to see all roots).
(lldb)

@jkotas
Copy link
Member

jkotas commented Dec 3, 2020

This needs more dumps to develop pattern. I do not see any more clues.

@jkotas jkotas added area-VM-coreclr tenet-reliability Reliability/stability related issue (stress, load problems, etc.) needs more info and removed area-System.Collections labels Dec 3, 2020
@runfoapp runfoapp bot mentioned this issue Dec 3, 2020
Closed
@stephentoub stephentoub removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Mar 5, 2021
@mangod9
Copy link
Member

mangod9 commented Jul 6, 2021

Doesnt look like the test was disabled, so assume this is no longer reproing?

@mangod9 mangod9 added this to the 6.0.0 milestone Jul 6, 2021
@mangod9
Copy link
Member

mangod9 commented Jul 10, 2021

@ViktorHofer ok to close assuming this hasnt reproed in a while?

@mangod9
Copy link
Member

mangod9 commented Jul 21, 2021

Closing for now

@mangod9 mangod9 closed this as completed Jul 21, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Aug 20, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-VM-coreclr tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Projects
None yet
Milestone
6.0.0
Development

No branches or pull requests
4 participants
@stephentoub @jkotas @ViktorHofer @mangod9