[mono][tvos] Libraries tests failed to build on Helix due to missing System.Runtime.Serialization.Formatters.* files

[ivanpovazan]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=786206&view=results

Affected pipeline:

• runtime-extra-platforms

Build error leg:

• tvos_arm64_release_allsubsets_mono

Work items failing:

• System.Resources.Extensions.Tests
• System.Formats.Nrbf.Tests.WorkItemExecution

Relevant failure:

Device target detected but app not found, skipping signing..

Welcome to .NET 9.0!
---------------------
SDK Version: 9.0.100-preview.7.24407.12

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate, run 'dotnet dev-certs https --trust'
Learn about HTTPS: https://aka.ms/dotnet-https

----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
MSBuild version 17.12.0-preview-24374-02+48e81c6f1 for .NET
/private/tmp/helix/working/A62F0915/w/A5930953/e/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/tmp/helix/working/A62F0915/p/build/microsoft.netcore.app.runtime.tvos-arm64/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.pdb" because it was not found.
/private/tmp/helix/working/A62F0915/w/A5930953/e/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/tmp/helix/working/A62F0915/p/build/microsoft.netcore.app.runtime.tvos-arm64/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.xml" because it was not found.
Signing bundle /tmp/helix/working/A62F0915/w/A5930953/e/System.Resources.Extensions.Tests.app
No embedded provisioning profile found at /tmp/helix/working/A62F0915/w/A5930953/e/System.Resources.Extensions.Tests.app/embedded.mobileprovision! Failed to sign the app!

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorPattern": "System\\.Runtime\\.Serialization\\.Formatters\\..* because it was not found",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=786206
Error message validated: [System\.Runtime\.Serialization\.Formatters\..* because it was not found]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/27/2024 12:45:02 PM UTC

Report

Build	Definition	Test	Pull Request
792568	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#106858
786206	dotnet/runtime	System.Resources.Extensions.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	2

[dotnet-policy-service]

Tagging subscribers to 'os-tvos': @vitek-karas, @kotlarmilos, @ivanpovazan, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @directhex, @matouskozak
See info in area-owners.md if you want to be subscribed.

[ivanpovazan]

This could have been introduces with: #106737
I will follow-up with how we build tvos apps on Helix and why this started failing.
/cc: @ericstj

[dotnet-policy-service]

Tagging subscribers to 'binaryformatter-migration': @adamsitnik, @bartonjs, @jeffhandley, @terrajobst

[ericstj]

cc @adamsitnik

Why is it trying to copy PDB and XML?

/private/tmp/helix/working/A62F0915/w/B419093C/e/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/tmp/helix/working/A62F0915/p/build/microsoft.netcore.app.runtime.tvos-arm64/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.xml" because it was not found.
/private/tmp/helix/working/A62F0915/w/B419093C/e/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/tmp/helix/working/A62F0915/p/build/microsoft.netcore.app.runtime.tvos-arm64/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.pdb" because it was not found.

Here's the source:

runtime/src/mono/msbuild/apple/data/ProxyProjectForAOTOnHelix.proj

Lines 111 to 117 in ef0c712

	<ItemGroup>
	<_RuntimePackFiles Condition="%(_AppleUsedRuntimePackFiles.Extension) == '.dll' and %(_AppleUsedRuntimePackFiles.FileName) != 'System.Private.CoreLib'" Include="@(_AppleUsedRuntimePackFiles->'$(MicrosoftNetCoreAppRuntimePackLibDir)%(FileName)%(Extension)')" />
	<_RuntimePackFiles Condition="%(_AppleUsedRuntimePackFiles.Extension) != '.dll' or %(_AppleUsedRuntimePackFiles.FileName) == 'System.Private.CoreLib'" Include="@(_AppleUsedRuntimePackFiles->'$(MicrosoftNetCoreAppRuntimePackNativeDir)%(FileName)%(Extension)')" />
	</ItemGroup>
	<Copy SourceFiles="@(_RuntimePackFiles)"
	DestinationFolder="$(OriginalPublishDir)" />

Probably there is a difference in the items copied for a project reference vs the framework reference. I suspect there are some assumptions in that copying logic that are broken by a project reference being preferred.

[ivanpovazan]

Some more information about building ios/tvos test on CI.
We split the build into:

1. Build of runtime, libs and tests on build machines
2. AOT + bundling test apps on Helix machines

For 1. we collect all required files for a particular test, zip it and send over to Helix. The logic of picking up the right set of files is a bit tricky and it happens in this MSBuild target:

runtime/eng/testing/tests.ioslike.targets

Line 60 in c1a9f26

<Target Name="_BundleAOTTestAppleAppForHelix" DependsOnTargets="PrepareForAppleBuildApp">

So what I think that happens is that the System.Runtime.Serialization.Formatters.xml System.Runtime.Serialization.Formatters.pdb get passed over as required files through _AppleUsedRuntimePackFiles item group, but they are not carried over to / or restored on Helix, this probably happens since these file are present both in output of the test project build and in the runtime pack. Will investigate further.

---

FWIW to repro the failure locally one can do the following steps:

1. Build runtime/libs/tests:

./build.sh -s mono+libs+libs.tests -os tvos -arch arm64 -c Release /p:ArchiveTests=true /p:DevTeamProvisioning=- /p:RunAOTCompilation=true /p:RunSmokeTestsOnly=False /p:EnableAdditionalTimezoneChecks=true  /p:BuildDarwinFrameworks=true /p:IsManualOrRollingBuild=true /p:BuildTestsOnHelix=true /p:BuildDarwinFrameworks=true /p:UsePortableRuntimePack=true -binaryLog

2. Unzip the folder that would be sent to Helix, and cd to the publish dir

mkdir artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests
tar -xvf artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests.zip -C artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests
pushd artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests/publish

3. AOT the test and create the app bundle (use your repo root instead <Path-to-Your-Repo-Root>)

../../../../../../dotnet.sh msbuild ProxyProjectForAOTOnHelix.proj /p:RuntimeSrcDir=<Path-to-Your-Repo-Root> /p:RuntimeConfig=Release /p:RunAOTCompilation=true /p:UsePortableRuntimePack=true /p:TargetOS=tvos /p:TargetArchitecture=arm64 /p:MonoEnableLLVM=true /p:DevTeamProvisioning=- /p:Configuration=Release /bl:test.binlog

The step 3. should fail with:

/Users/ivan/repos/runtime-mono/artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/Users/ivan/repos/runtime-mono/artifacts/bin/microsoft.netcore.app.runtime.tvos-arm64/Release/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.pdb" because it was not found.
/Users/ivan/repos/runtime-mono/artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/Users/ivan/repos/runtime-mono/artifacts/bin/microsoft.netcore.app.runtime.tvos-arm64/Release/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.xml" because it was not found.

[ericstj]

Copying the PDB is a good idea if it will actually be consumed, since that helps get more accurate stack traces in cases of crashes. If the PDB is not actually consumed, then it might be better to just ignore those items in the logic that's trying to copy them on the test machine. Looks like elsewhere there are places that are trying to only handle .dll.