[mono][ios/tvos] DynamicInvoke_DateTimeAndCustomConstantAttribute_DateTimeParameterWithMissingValue test failure on CI

[ivanpovazan]

Description

Both jobs:

• ios-arm64 Release AllSubsets_Mono
• tvos-arm64 Release AllSubsets_Mono

Are failing on runtime pipeline due to a test failure:

[FAIL] System.Tests.DelegateTests.DynamicInvoke_DateTimeAndCustomConstantAttribute_DateTimeParameterWithMissingValue   Test name: System.Tests.DelegateTests.DynamicInvoke_DateTimeAndCustomConstantAttribute_DateTimeParameterWithMissingValue
[08:03:49.0336820] 2024-01-30 08:03:49.101 System.Runtime.Tests[90748:69574085]    Assembly:  [System.Runtime.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
[08:03:49.0336900]    Exception messages: Assert.Equal() Failure: Values differ
[08:03:49.0336970] Expected: 0001-01-01T00:00:00.0000042
[08:03:49.0337050] 2024-01-30 08:03:49.101 System.Runtime.Tests[90748:69574085] Actual:   0001-01-01T00:00:00.0000043   Exception stack traces:    at System.Tests.DelegateTests.DynamicInvoke_DateTimeAndCustomConstantAttribute_DateTimeParameterWithMissingValue()
[08:03:49.0337150]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
[08:03:49.0337250] 2024-01-30 08:03:49.101 System.Runtime.Tests[90748:69574085]    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
[08:03:49.0337330]    Execution time: 0.0005927

Manual build analysis

• Last green CI build was: https://dev.azure.com/dnceng-public/public/_build/results?buildId=545633&view=logs&j=7f49df26-8126-5de3-bf2f-6ac6bde01830
  Against: d40c654

• First detected failure: https://dev.azure.com/dnceng-public/public/_build/results?buildId=545788&view=logs&j=7f49df26-8126-5de3-bf2f-6ac6bde01830
  Against: 28ad7e9

• This indicates the failure happening in the following range:
  d40c654...28ad7e9

/cc: @kotlarmilos @akoeplinger since the range above includes the change of enabling IL stripping for libraries tests, I suspect it is related to the regression (even though your PR seemed to have all tests passing, some change in the meantime must started conflicting with it?)

Additional findings

The first detected failure (build ID: 545788) was incorrectly marked by BuildAnalysis as Known Build Errors in: #82637 as the system scans only the console logs and not the device logs indicating the actual cause of failure.

This is really problematic as ios/tvos failures get masked with an incorrect (or at least not precise enough) build issue making it really difficult to track back what the actual failure was (and when it started occurring).

/cc: @matouskozak @vitek-karas

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "[FAIL] System.Tests.DelegateTests.DynamicInvoke_DateTimeAndCustomConstantAttribute_DateTimeParameterWithMissingValue",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=545633
Error message validated: [FAIL] System.Tests.DelegateTests.DynamicInvoke_DateTimeAndCustomConstantAttribute_DateTimeParameterWithMissingValue
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 2/1/2024 11:43:50 AM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[ghost]

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger, @kotlarmilos
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Both jobs:

• ios-arm64 Release AllSubsets_Mono
• tvos-arm64 Release AllSubsets_Mono

Are failing on runtime pipeline due to a test failure:

[FAIL] System.Tests.DelegateTests.DynamicInvoke_DateTimeAndCustomConstantAttribute_DateTimeParameterWithMissingValue   Test name: System.Tests.DelegateTests.DynamicInvoke_DateTimeAndCustomConstantAttribute_DateTimeParameterWithMissingValue
[08:03:49.0336820] 2024-01-30 08:03:49.101 System.Runtime.Tests[90748:69574085]    Assembly:  [System.Runtime.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
[08:03:49.0336900]    Exception messages: Assert.Equal() Failure: Values differ
[08:03:49.0336970] Expected: 0001-01-01T00:00:00.0000042
[08:03:49.0337050] 2024-01-30 08:03:49.101 System.Runtime.Tests[90748:69574085] Actual:   0001-01-01T00:00:00.0000043   Exception stack traces:    at System.Tests.DelegateTests.DynamicInvoke_DateTimeAndCustomConstantAttribute_DateTimeParameterWithMissingValue()
[08:03:49.0337150]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
[08:03:49.0337250] 2024-01-30 08:03:49.101 System.Runtime.Tests[90748:69574085]    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
[08:03:49.0337330]    Execution time: 0.0005927

Manual build analysis

• Last green CI build was: https://dev.azure.com/dnceng-public/public/_build/results?buildId=545633&view=logs&j=7f49df26-8126-5de3-bf2f-6ac6bde01830
  Against: d40c654

• First detected failure: https://dev.azure.com/dnceng-public/public/_build/results?buildId=545788&view=logs&j=7f49df26-8126-5de3-bf2f-6ac6bde01830
  Against: 28ad7e9

• This indicates the failure happening in the following range:
  d40c654...28ad7e9

/cc: @kotlarmilos @akoeplinger since the range above includes the change of enabling IL stripping for libraries tests, I suspect it is related to the regression (even though your PR seemed to have all tests passing, some change in the meantime must started conflicting with it?)

Additional findings

The first detected failure (build ID: 545788) was incorrectly marked by BuildAnalysis as Known Build Errors in: #82637 as the system scans only the console logs and not the device logs indicating the actual cause of failure.

This is really problematic as ios/tvos failures get masked with an incorrect (or at least not precise enough) build issue making it really difficult to track back what the actual failure was (and when it started occurring).

/cc: @matouskozak @vitek-karas

Author:	ivanpovazan
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure-mono, os-ios
Milestone:	9.0.0

[kotlarmilos]

Thanks for reporting this. I couldn't find any relevant runtime or library change, so I believe it could be an issue related to IL stripping. However, I don't know why it didn't appear as a failure initially.

[ivanpovazan]

@kotlarmilos based on #97870 should we close this issue?

[kotlarmilos]

I would keep it open as it seems to be ILStrip-related issue. There is another issue related to ILStrip, so I plan to create a tracking issue.