Try to fix a time-sensitive thread pool test

[kouvel]

I guess the test may occasionally time out if it's running in parallel with other tests, depending on what those other tests are doing. Subscribed to thread adjustment events to see if progress is being made to extend the timeout.

Hopefully fixes #66852

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I guess the test may occasionally time out if it's running in parallel with other tests, depending on what those other tests are doing. Subscribed to thread adjustment events to see if progress is being made to extend the timeout.

Hopefully fixes #66755

Author:	kouvel
Assignees:	kouvel
Labels:	area-System.Threading
Milestone:	7.0.0

[am11]

If parallelization is the root-cause of failure, there is DisableTestParallelization behavior to disable it, e.g.

runtime/src/libraries/System.Configuration.ConfigurationManager/tests/AssemblyInfo.cs

Line 7 in 38af2b8

[assembly: CollectionBehavior(DisableTestParallelization = true, MaxParallelThreads = 1)]

(requires <Compile Include="$(CommonTestPath)TestUtilities\System\DisableParallelizationPerAssembly.cs" /> in csproj)

[kouvel]

If parallelization is the root-cause of failure, there is DisableTestParallelization behavior to disable it

Does that disable parallelization only for this test assembly, or does it also disable parallelization between tests from different assemblies?

[am11]

I am not sure but I think on helix, we only execute one test assembly at a time? @adamsitnik might know the definite answer.

[adamsitnik]

I am not sure but I think on helix, we only execute one test assembly at a time?

Afaik this is true (cc @ericstj who is our ultimate source of truth).

DisableParallelizationPerAssembly

by using it it's ensured that all tests in given assembly are run sequentially.

DisableParallelization runs all other tests in parallel first, and then at the end sequentially tests from types marked with this attribute (this is typically what I am using).

[agocke]

@kouvel What's the status of this change? Will it be done today? nvm, missed that the test was disabled

[hoyosjs]

Helix assemblies for libraries do run on a project basis: https://helix.dot.net/api/jobs/f8d1561b-e515-4958-8114-6d2a0900b0de/workitems?api-version=2019-06-17 each one of those runs separately.

[ericstj]

Afaik this is true (cc @ericstj who is our ultimate source of truth).

Ha! I'm not the source of truth -- the source is. I just read it. It can change at any time so I go back and re-read it. Sometimes I don't read the source and instead read the test logs if I don't trust my eyeball-debugger. cc @dotnet/runtime-infrastructure

In libraries we don't parallelize at the assembly level. Each test assembly gets launched once and helix only runs one job at a time on the machine. You can examine how that works here:

runtime/eng/testing/tests.targets

Line 104 in 96ef47b

<GenerateRunScript RunCommands="@(RunScriptCommands)"

https://github.com/dotnet/runtime/search?q=RunScriptCommand

The coreclr tests are a little different. They intentionally parallelize (and that actually has caused problems). You can see that here:

runtime/src/tests/Common/testgrouping.proj

Lines 284 to 303 in d8f8527

	<XUnitWrapperGrouping Include="$(TestBinDir)baseservices\compilerservices\*.XUnitWrapper.dll;
	$(TestBinDir)baseservices\exceptions\*.XUnitWrapper.dll;
	$(TestBinDir)baseservices\multidimmarray\*.XUnitWrapper.dll;
	$(TestBinDir)baseservices\TieredCompilation\*.XUnitWrapper.dll;
	$(TestBinDir)baseservices\typeequivalence\*.XUnitWrapper.dll;
	$(TestBinDir)baseservices\varargs\*.XUnitWrapper.dll;
	$(TestBinDir)CoreMangLib\**\*.XUnitWrapper.dll;
	$(TestBinDir)Exceptions\**\*.XUnitWrapper.dll;
	$(TestBinDir)GC\**\*.XUnitWrapper.dll;
	$(TestBinDir)hosting\**\*.XUnitWrapper.dll;
	$(TestBinDir)ilasm\**\*.XUnitWrapper.dll;
	$(TestBinDir)Loader\**\*.XUnitWrapper.dll;
	$(TestBinDir)managed\**\*.XUnitWrapper.dll;
	$(TestBinDir)readytorun\**\*.XUnitWrapper.dll;
	$(TestBinDir)reflection\**\*.XUnitWrapper.dll;
	$(TestBinDir)Regressions\**\*.XUnitWrapper.dll;
	$(TestBinDir)tracing\**\*.XUnitWrapper.dll"
	Exclude="$(TestBinDir)Loader\classloader\Loader.classloader.XUnitWrapper.dll">
	<PayloadGroup>PayloadGroup0</PayloadGroup>
	</XUnitWrapperGrouping>

runtime/src/tests/Common/helixpublishwitharcade.proj

Line 518 in 90ec876

<XUnitWrapperDlls>$([System.String]::Join(' ', $([System.IO.Directory]::GetFiles(%(FullPath), '*.XUnitWrapper.dll', SearchOption.AllDirectories))).Replace($([MSBuild]::EnsureTrailingSlash(%(FullPath))),''))</XUnitWrapperDlls>

[kouvel]

Thanks all! I'll try disabling parallelization for this test.

[kouvel]

Still failing and appears to not be a parallelization issue. I can repro the issue on the latest macOS, seems to be an issue with GC.GetGCMemoryInfo() reporting that the memory load is >= the memory limit. Considering that it started failing somewhat suddenly, I suspect it has something to do with GC or OS changes. I'll file a separate issue.