Xunit-reporter needs to use venv for cross-architecture scenarios

[MattGal]

• This issue is blocking
• This issue is causing unreasonable pain

Example log

As noted in dotnet/runtime#39244 . When Xunit-reporter was created, we didn't have any ARM arch docker machines (or even docker support, for that matter). As such, it tries to use the mounted-in python packages including native bits from the host machine, which may or may not work inside the docker container (for instance, in the case of ARM32 on ARM64, definitely not).

There's an existing pattern that can be followed used by the azure devops reporter in the same Helix SDK, which is to use Venv to run it (instead of just straight "python"). (See https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter )

[garath]

Triage: We plan to talk about this more under the context of the Resiliency effort.

[BruceForstall]

Any progress on this? AFAIK, this is preventing us from having Linux arm results show up in Kusto, right? e.g., https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-5628ce574dba4b4cba/System.Memory.Tests/console.90a1acb5.log?sv=2019-02-02&se=2020-08-14T10%3A00%3A27Z&sr=c&sp=rl&sig=eFAf6QU%2BRwGgPmhS08Hj86VIZfZIpmsPq%2Ben3XNRNqM%3D

[markwilkie]

This is telemetry used by us. Is there another path you could take to get what you need? Taking a dependency on our Kusto db may not be the best path forward.

cc/ @Chrisboh

[BruceForstall]

Maybe I'm mistaken about the impact of the failure? I thought from the description it was preventing test results from being included in the Kusto data which I regularly use to determine how frequently a test fails and how wide the impact is of a particular test failing. I use https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata for that. When you say "taking a dependency on our Kusto db", is that the one you are talking about?

[markwilkie]

I think so - but @Chrisboh can confirm. I'm wondering if there's a better way to do this... What are you using the data for from kusto?

[BruceForstall]

I (and others) use the data to analyze test failures, especially "flaky" test failures. E.g., how often does test A fail? In which pipelines does it fail? On what architectures does it fail? Kusto is, of course, very flexible, and that flexibility is very useful when doing this kind of work. It's MUCH better for this than simply using the AzDO UI, which is extremely lacking in this kind flexible analysis capability.

• @jashook

[MattGal]

@lukas-lansky any update?

[lukas-lansky]

@MattGal I will open the PR today.

[ilyas1974]

@lukas-lansky is there any information you can add to this issue regarding the PR you created? Is this issue still a problem?

[lukas-lansky]

@ilyas1974 The PR should solve the issue. Let me find someone who will help me review that and maybe provide additional context. (Also, I'm on FR next week luckily.)

[ChadNedzlek]

We are likely going to remove this reporter as part of our resiliency effort. https://github.com/dotnet/core-eng/issues/11347 (As part of that, we are also only going to record failures as individual data items).

I'm curious what sorts of reporting you are doing, because we are trying to figure out a way to only store aggregated data so that people can still be productive without us having to shuttle terabytes of data around that is mostly useless.

[BruceForstall]

I had a long conversation with @Chrisboh about this. Bottom-line: this problem means linux-arm failures don't get reported, and we can't use Kusto to see trends in test failures.

[lukas-lansky]

I verified with @alexperovich that the work is being done on removal of the problematic code. I closed my PR and I'm moving this to Tracking as discussed yesterday on FR standup. We can close this as soon as Alex's work is done without any change specific to this issue.

[lukas-lansky]

The work is now being tracked in https://github.com/dotnet/core-eng/issues/11793.