Dead lettering tests

[carlossanlop]

Build Information

Build: https://dev.azure.com/dnceng-public/public/public%20Team/_build/results?buildId=654910
Build error leg or test failing: browser-wasm windows Release LibraryTests_Smoke_AOT

Error Message

{
  "ErrorMessage" : "If you’re reading this, that means the Helix work item you’re trying to find the logs for has dead-lettered.",
  "BuildRetry" : false,
  "ExcludeConsoleLog" : false
}

• PR: Restrict indentation size in JsonWriterOptions theories. #101498
• Queue: net9.0-browser-Release-wasm-Mono_Release-WasmTestOnV8
• Job result: https://dev.azure.com/dnceng-public/public/public%20Team/_build/results?buildId=654910&view=logs&j=0eae1e18-ea37-5071-1d48-9384b3d4e672&t=7c50f1ca-0ae5-5055-6835-fe157eafc276&l=59
• Log file: https://helix.dot.net/api/2019-06-17/jobs/528aca7b-21fa-4ecf-8f47-4fa3e533bc93/workitems/WasmTestOnV8-System.Runtime.Tests/console
• Output:

If you’re reading this, that means the Helix work item you’re trying to find the logs for has dead-lettered.

What this means:

- All attempts to retry execution of this work item were unable to complete.  This can be both for infrastructure reasons (problems within Azure) or issues with the work item (for instance, causing a machine to reboot unexpectedly or killing the Helix client on the machine will force a retry).
- No further work will be done for this specific work item, and its exit code is set to an artificial -1 (since it did not complete, there is no real exit code).

Common causes:

- Disabled queue (end-of-life Helix queues are automatically forwarded to deadletter and will fail instantly)
- Unhealthy Helix Client machine(s)
- Queue has been backed up heavily by a large amount of work and was manually purged by the engineering team
- Azure issues (e.g. Service Bus is overloaded)
- Malformed payloads; if Helix cannot download and unzip all payloads successfully, work will retry until dead-lettered.

For follow up:

- Check if your Helix Queue is still enabled, either via the metadata you see by browsing to https://helix.dot.net/api/info/queues?api-version=2019-06-17 or recent emails from the .NET Engineering Infrastructure team.
- Check that all work item payloads are accessible using a browser.
- If you are sending to a non-disabled queue and find this error repeatedly occurring, please contact the dnceng team.
- If a single, specific work item dead letters and others do not, consider local debugging; it may be causing spontaneous reboot (or trigging one intentionally).

Report

Build	Definition	Test	Pull Request
710640	dotnet/runtime	System.Reflection.Tests.WorkItemExecution	#103595
710441	dotnet/runtime	Workloads-Wasm.Build.NativeRebuild.Tests.OptimizationFlagChangeTests.WorkItemExecution
710444	dotnet/runtime	System.Threading.Tasks.Parallel.Tests.WorkItemExecution
710229	dotnet/runtime	System.Security.Cryptography.Csp.Tests.WorkItemExecution	#103560
709965	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
709966	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
709954	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
709957	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
709335	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
709334	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution
709332	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
709333	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
709068	dotnet/runtime	System.Threading.Tasks.Dataflow.Tests.WorkItemExecution	#100041
708944	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#103416
708854	dotnet/runtime	System.Diagnostics.TraceSource.Tests.WorkItemExecution
708846	dotnet/runtime	System.Diagnostics.TraceSource.Tests.WorkItemExecution
708853	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
708851	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
708463	dotnet/runtime	System.Globalization.CalendarsWithConfigSwitch.Tests.WorkItemExecution
708201	dotnet/runtime	System.Diagnostics.TraceSource.Tests.WorkItemExecution
708194	dotnet/runtime	JIT.Generics.WorkItemExecution	#103484
708055	dotnet/runtime	System.ComponentModel.TypeConverter.Tests.WorkItemExecution	#102927
708065	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#103479
708015	dotnet/runtime	System.Runtime.Serialization.BinaryFormat.Tests.WorkItemExecution	#103370
708020	dotnet/runtime	System.IO.Tests.WorkItemExecution	#103337
708039	dotnet/runtime	System.CodeDom.Tests.WorkItemExecution	#103456
707938	dotnet/runtime	Workloads-Wasm.Build.Tests.SatelliteAssembliesTests.WorkItemExecution	#98494
706459	dotnet/runtime	iOS.Simulator.LibraryMode.Test.WorkItemExecution	#102748
707775	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
707765	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution
707769	dotnet/runtime	System.Diagnostics.TraceSource.Config.Tests.WorkItemExecution
707764	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
703066	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#102670
707322	dotnet/runtime	System.IO.Tests.WorkItemExecution	#103442
707058	dotnet/runtime	System.Diagnostics.Tools.Tests.WorkItemExecution
707052	dotnet/runtime	Regression_3.WorkItemExecution
707057	dotnet/runtime	System.Runtime.Extensions.Tests.WorkItemExecution
706980	dotnet/runtime	System.IO.Tests.WorkItemExecution
706568	dotnet/runtime	System.Composition.Convention.Tests.WorkItemExecution	#103411
706525	dotnet/runtime	System.Diagnostics.Tracing.Tests.WorkItemExecution
706528	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution
706532	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
706522	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
706429	dotnet/runtime	tvOS.Device.Aot.Test.WorkItemExecution	#103232
706163	dotnet/runtime	Regression_3.WorkItemExecution
706180	dotnet/runtime	System.Diagnostics.TextWriterTraceListener.Tests.WorkItemExecution	#103393
706187	dotnet/runtime	System.Collections.Specialized.Tests.WorkItemExecution	#103364
706167	dotnet/runtime	System.Diagnostics.FileVersionInfo.Tests.WorkItemExecution
705894	dotnet/runtime	Microsoft.Extensions.Diagnostics.Tests.WorkItemExecution	#103379
705827	dotnet/runtime	System.IO.Tests.WorkItemExecution	#103366
2472316	dotnet-runtime	System.Dynamic.Runtime.Tests.WorkItemExecution	#40292
2471911	dotnet-runtime	release.Partition0.WorkItemExecution
2472143	dotnet-runtime	release.Partition0.WorkItemExecution
705295	dotnet/runtime	System.IO.Tests.WorkItemExecution
2471843	dotnet-runtime	System.Formats.Cbor.Tests.WorkItemExecution	#40292
705034	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution
705035	dotnet/runtime	System.Diagnostics.Tracing.Tests.WorkItemExecution
705022	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
705026	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution
704278	dotnet/runtime	System.Collections.Specialized.Tests.WorkItemExecution	#99747
704203	dotnet/runtime	Regression_3.WorkItemExecution
704267	dotnet/runtime	System.Globalization.Tests.WorkItemExecution
704117	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
703927	dotnet/runtime	System.Diagnostics.Tools.Tests.WorkItemExecution
703923	dotnet/runtime	Regression_3.WorkItemExecution
703814	dotnet/runtime	System.Runtime.Serialization.Schema.Tests.WorkItemExecution	#103283
703591	dotnet/runtime	System.Diagnostics.TraceSource.Tests.WorkItemExecution
703592	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
703588	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
703582	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
703614	dotnet/runtime	Invariant.Tests.WorkItemExecution	#100030
703283	dotnet/runtime	Regression_3.WorkItemExecution
703219	dotnet/runtime	iOS.Simulator.LibraryMode.Test.WorkItemExecution
703291	dotnet/runtime	System.IO.Tests.WorkItemExecution	#103266
703287	dotnet/runtime	System.Diagnostics.DiagnosticSource.Switches.Tests.WorkItemExecution
703222	dotnet/runtime	System.Globalization.Calendars.Tests.WorkItemExecution
703069	dotnet/runtime	JIT_Math.WorkItemExecution
703205	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution
703121	dotnet/runtime	Regression_3.WorkItemExecution
703077	dotnet/runtime	System.Globalization.Tests.WorkItemExecution
702874	dotnet/runtime	iOS.Simulator.LibraryMode.Test.WorkItemExecution
702836	dotnet/runtime	System.Console.Tests.WorkItemExecution
702918	dotnet/runtime	Regression_3.WorkItemExecution
703011	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
702973	dotnet/runtime	System.IO.Compression.Tests.WorkItemExecution
702832	dotnet/runtime	Regression_3.WorkItemExecution
702967	dotnet/runtime	System.IO.Compression.Tests.WorkItemExecution
702934	dotnet/runtime	System.Collections.Specialized.Tests.WorkItemExecution	#103138
702711	dotnet/runtime	Regression_3.WorkItemExecution
702715	dotnet/runtime	System.Console.Tests.WorkItemExecution
2470077	dotnet-runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution	#40222
702615	dotnet/runtime	System.Diagnostics.TextWriterTraceListener.Tests.WorkItemExecution	#103159
2470076	dotnet-runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution	#40221
702578	dotnet/runtime	tvOS.Device.Aot.Test.WorkItemExecution	#103226
702185	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution
702184	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
702186	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
702182	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
701895	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.WorkItemExecution
701694	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution
Displaying 100 of 338 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
8	77	338

[dotnet-policy-service]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

[carlossanlop]

In the same PR's build where the above dead-letter failure was found, there's another run for a similar queue but is Release. While it does not dead-letter immediately, it manages to print a couple lines, then dies:

• Build error leg: browser-wasm windows Release LibraryTests_Smoke_AOT
• Queue: net9.0-browser-Release-wasm-Mono_Release-WasmTestOnChrome-
• Job result: https://dev.azure.com/dnceng-public/public/_build/results?buildId=654910&view=logs&j=bf575327-8861-5da8-d33e-e12ac4086c09&t=c85eeb3c-de58-5b19-fbd6-7d6b5ac3e88c
• Log file: https://helix.dot.net/api/2019-06-17/jobs/2736decc-83a1-4c32-9e0a-ef543e0d26f3/workitems/WasmTestOnChrome-System.Runtime.Tests/console
• Output:

Console log: 'WasmTestOnChrome-System.Runtime.Tests' from job 2736decc-83a1-4c32-9e0a-ef543e0d26f3 (windows.amd64.server2022.open.rt) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2022-helix-webassembly on a000NF0
running %HELIX_CORRELATION_PAYLOAD%\scripts\be8b1ad5c1e9498d89709f26e508c549\execute.cmd in C:\h\w\B0E6099F\w\A1EF089A\e max 3600 seconds

^ It just dies after printing the second line.

I do not want to open a KnownBuildError issue for this specific failure as it would end up grouping anything. I am concerned that people are going to get blocked on getting their PRs merged because they cannot bypass the merge on green restriction. For example, this PR is already blocked and I will only be able to merge it if I JIT elevate myself: #101498
@JulieLeeMSFT @hoyosjs @jkoritzinsky

[jkotas]

I am concerned that people are going to get blocked on getting their PRs merged because they cannot bypass the merge on green restriction. For example, this PR is already blocked and I will only be able to merge it if I JIT elevate myself:

People should be able to use https://github.com/dotnet/runtime/blob/main/docs/workflow/ci/failure-analysis.md#bypassing-build-analysis . Have you tried that before JIT elevating?

[carlossanlop]

People should be able to use https://github.com/dotnet/runtime/blob/main/docs/workflow/ci/failure-analysis.md#bypassing-build-analysis . Have you tried that before JIT elevating?

I was not aware of that. Thanks for sharing! I'll try it next time.

[agocke]

What does dead-lettering mean in this context? What is the case where this fails?

[lewing]

iirc Dead Lettering is usually infrastructure related, is that correct @steveisok ?

We often see queues fall over around branch time

If you’re reading this, that means the Helix work item you’re trying to find the logs for has dead-lettered.

What this means:

- All attempts to retry execution of this work item were unable to complete.  This can be both for infrastructure reasons (problems within Azure) or issues with the work item (for instance, causing a machine to reboot unexpectedly or killing the Helix client on the machine will force a retry).
- No further work will be done for this specific work item, and its exit code is set to an artificial -1 (since it did not complete, there is no real exit code).

Common causes:

- Disabled queue (end-of-life Helix queues are automatically forwarded to deadletter and will fail instantly)
- Unhealthy Helix Client machine(s)
- Queue has been backed up heavily by a large amount of work and was manually purged by the engineering team
- Azure issues (e.g. Service Bus is overloaded)
- Malformed payloads; if Helix cannot download and unzip all payloads successfully, work will retry until dead-lettered.

For follow up:

- Check if your Helix Queue is still enabled, either via the metadata you see by browsing to https://helix.dot.net/api/info/queues?api-version=2019-06-17 or recent emails from the .NET Engineering Infrastructure team.
- Check that all work item payloads are accessible using a browser.
- If you are sending to a non-disabled queue and find this error repeatedly occurring, please contact the dnceng team.
- If a single, specific work item dead letters and others do not, consider local debugging; it may be causing spontaneous reboot (or trigging one intentionally).

[steveisok]

iirc Dead Lettering is usually infrastructure related, is that correct @steveisok ?

We often see queues fall over around branch time

Correct. @ilyas1974 is queue dead lettering manually driven or is there some automation involved?

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[ilyas1974]

For you dead lettering question, the answer is Yes - it's a manual and automated process. We manually deadletter a queue so the changes are immediate. We then add the deadletter information to the helix configuration, so he is persistent for whenever we make changes to helix.

[agocke]

Ok, so is the expectation that tests should be re-run when deadlettering happens? That basically, that run was invalid?

[lewing]

I've removed the wasm references in the labels and title bits because wasm is no longer dominating the failures in any way (with the exception of preview4 which has known problems that are fixed in main)

[agocke]

Bypassing these tests doesn't seem appropriate. I'm closing the issue.