Some tests timing out on helix

[radical]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618&view=results
Build error leg or test failing: Aspire.Hosting.Elasticsearch.Tests.WorkItemExecution
Pull request: #5243

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Aborting test run: test run timeout of [0-9]+ milliseconds exceeded",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618
Error message validated: [Aborting test run: test run timeout of [0-9]+ milliseconds exceeded]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/11/2024 7:28:44 PM UTC

Report

Build	Definition	Test	Pull Request
863093	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution
861137	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6610
859009	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	#6589
857247	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6570
856981	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	#6057
856858	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	#6560
856341	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	#6546
856290	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
854328	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	#6390
852969	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6487
852173	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#5407
851759	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6451
851666	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
851665	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6451
850807	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6425
850804	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
849239	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6057
849346	dotnet/aspire	Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution	#6417
848037	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6057
847729	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6377
847441	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	#5783
847251	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6057
845515	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	#6344
845032	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution
844487	dotnet/aspire	Aspire.Playground.Tests.ProjectSpecificTests.Aspire.Playground.Tests.ProjectSpecificTests.WithDockerfileTest	#6334
844234	dotnet/aspire	Aspire.Playground.Tests.ProjectSpecificTests.Aspire.Playground.Tests.ProjectSpecificTests.AzureFunctionsTest	#6325
842555	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	#5250

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	2	27

[radical]

cc @eerhardt @sebastienros

One of the tests timing out is Aspire.Hosting.Elasticsearch.Tests - log.

Why is this timing out after 10minutes on helix when all the tests combined didn't take that long on the build machine?

• One reason could be that the helix agents are under-powered compared to the build machines. And these tests start up a new app per xunit test in ElasticsearchFunctionalTests.

[radical]

I chose a broader error message to match against, so we can get any tests that are timing out on helix now.

[mitchdenny]

I am still hitting this today on this PR: #5223

[radical]

I am still hitting this today on this PR: #5223

I'll bump the timeouts. We are hitting new ones as we moved new tests to helix last week.

[radical]

Aspire.Hosting.Elasticsearch.Tests - I think this is timing out because each of the functional tests starts a new app, and the elasticsearch containers take a few minutes to start up, causing the time taken for all the tests to be more than 10 minutes.