[master] Switch to Windows Server queues #2927

dougbu · 2020-02-03T20:15:38Z

agents should start much faster

dougbu · 2020-02-05T06:32:54Z

src/TestingUtils/Microsoft.AspNetCore.Testing/test/MaximumOSVersionTest.cs

@@ -42,7 +42,7 @@ public void RunTest_Win10_RS4()
            Assert.NotNull(versionKey);
            var currentVersion = (string)versionKey.GetValue("CurrentBuildNumber");
            Assert.NotNull(currentVersion);
-            Assert.True(17134 >= int.Parse(currentVersion));
+            Assert.True(17134 >= int.Parse(currentVersion), $"Unexpected build number {currentVersion} on {Environment.OSVersion}.");


@MattGal this displays very strange information when the assertion fails:

Unexpected build number 17763 on Microsoft Windows NT 6.2.9200.0.

But, according to https://en.wikipedia.org/wiki/List_of_Microsoft_Windows_versions, 17763 is the build number of the latest Windows Server 2019 release. But, 6.2.9200.0 indicates the agent is running Windows Server 2012 (not R2) and 9200 is the build number of the latest release there.

FYI I need to revise our testing attributes to handle server machines and get them working in the new world. Right now they seem to work as expected only for the client OS.

Tagging @Tratcher

Environment.OSVersion lies for back-compat reasons unless you have an app manifest. VS and dotnet test both have manifests so that's not usually a problem?

What version did this queue use to run on?

Maybe server 2019 has a new manifest guid?
https://docs.microsoft.com/en-us/windows/win32/sbscs/application-manifests

What Tratcher said. From our perspective though, whenever we say server we are aiming for "latest non-core fully patched server as of when the image was generated" now.

Being current is great, we just need to figure out why our version detection broke.

Hmm, the app manifest docs only go up to Server 2016, I can't find any information about 2019.

The owners of the app manifest docs claim the GUID is unchanged for server 2019. Might have to set up some machines and confirm.
MicrosoftDocs/feedback#2477 (comment)

If it used the same GUID, these tests wouldn't have failed and we wouldn't see 6.2.9200.0

src/TestingUtils/Microsoft.AspNetCore.Testing/test/MaximumOSVersionTest.cs

Tratcher · 2020-02-05T17:04:11Z

src/TestingUtils/Microsoft.AspNetCore.Testing/test/MaximumOSVersionTest.cs

@@ -42,7 +42,7 @@ public void RunTest_Win10_RS4()
            Assert.NotNull(versionKey);
            var currentVersion = (string)versionKey.GetValue("CurrentBuildNumber");
            Assert.NotNull(currentVersion);
-            Assert.True(17134 >= int.Parse(currentVersion));
+            Assert.True(17134 >= int.Parse(currentVersion), $"Unexpected build number {currentVersion} on {Environment.OSVersion}.");


Environment.OSVersion lies for back-compat reasons unless you have an app manifest. VS and dotnet test both have manifests so that's not usually a problem?

What version did this queue use to run on?

Maybe server 2019 has a new manifest guid?
https://docs.microsoft.com/en-us/windows/win32/sbscs/application-manifests

dougbu · 2020-02-06T07:02:14Z

All: Environment.OSVersion lies when VS 2019 is installed too. The displayed version in the assertion failure didn't change when I moved from the VS 2017 queues to the VS 2019 ones. See https://dev.azure.com/dnceng/public/_build/results?buildId=509395&view=ms.vss-test-web.build-test-results-tab&runId=16160634&resultId=100056&paneView=debug

If the problem is the manifests and we had the GUID, we could contribute a PR that updates files like https://github.com/microsoft/vstest/blob/master/src/testhost.x86/app.manifest and https://github.com/microsoft/vstest/blob/master/src/vstest.console/app.manifest Those files currently list the supportedOS GUIDs up to Windows 10 / Windows Server 2016. But, we need the answer to the question (that @Tratcher highlighted) here: https://stackoverflow.com/questions/57320490/what-is-the-supportedos-guid-for-windows-server-2019

So, @Pilchie do we have any contacts on the Windows Server team?

Another option is to stick w/ the Windows.10 queues in all repos x branches using [MaximumOSVersion(...)] or [MinimumOSVersion(...)]. The "hold back repos" would be dotnet/extensions and dotnet/aspnetcore; 'release/3.1' and 'master' are affected in both. Thoughts?

Or, @MattGal do we have any queues running Windows Server 2016?

Tratcher · 2020-02-06T14:19:21Z

I also filed MicrosoftDocs/feedback#2477 to help track this down.

Pilchie · 2020-02-06T17:30:04Z

The OS is always known to lie about version. See https://devblogs.microsoft.com/oldnewthing/20100927-00/?p=12733 and it's linked articles for some of the reasons why. AFAIK, the way to get new OS versions to be returned is to compile with a newer Windows SDK.

We should start an internal thread with some contacts to figure out what to do here.

Also, I hear that dotnet/runtime has similar test attributes, we should look at how they work.

dougbu · 2020-02-15T00:27:38Z

I suspect the relevant dotnet/runtime class is the Windows portion of PlatformDetection but I doubt it's exactly what we need. The available properties include (say) IsWindows8xOrLater but not IsWindows10OrLater and IsWindowsServerCore but not IsWindowsServer. (dotnet/runtime uses these properties in lambdas passed to [ConditionalTheory] attributes. I can't find that attribute but https://github.com/dotnet/runtime/blob/0b25063ec056b0d6668bf0848d108d8c873f3fce/src/libraries/System.Globalization/tests/CultureInfo/GetCultureInfo.cs#L12-L14 is an example of its use.)

Then again, IsWindows10Version1903OrGreater and similar properties may be useful and the externs might be what we need, especially GetProductInfo.

/cc @stephentoub and @ViktorHofer (feel free to take this offline). We're trying to get our testing attributes to treat Windows Server 2019 as if it were Windows 10 but don't have the right manifest to do it with our current approach. See MaximumOSVersionAttribute and MinimumOsVersionAttribute

Any suggestions much appreciated❕

- wait for unblock of dotnet#2927 - however, do switch to VS2019 queues

- wait for unblock of dotnet#2927

…rsionTest.cs Co-Authored-By: Chris Ross <[email protected]>

Tratcher · 2020-03-03T20:56:16Z

I wasn't able to reproduce this issue locally using the current master. I set up a Win Server 2019 VM and the tests passed fine. Another app worked fine with the normal Win10 app manifest.

Microsoft Visual Studio Professional 2019
Version 16.4.5
VisualStudio.16.Release/16.4.5+29806.167
Microsoft .NET Framework
Version 4.7.03190

Installed Version: Professional

ASP.NET and Web Tools 2019   16.4.460.23317
ASP.NET and Web Tools 2019

ASP.NET Web Frameworks and Tools 2019   16.4.460.23317
For additional information, visit https://www.asp.net/
...

Doug, I rebased your changes on master to see if anything has changed.

Tratcher · 2020-03-03T21:46:48Z

Nope, still failing. Interesting observation though, it's only failing on the net472 target, Core is fine.

Tests succeeded: F:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Testing.Tests\Debug\netcoreapp5.0\Microsoft.AspNetCore.Testing.Tests.dll [netcoreapp5.0|x64]

XUnit : error : Tests failed: F:\workspace\_work\1\s\artifacts\TestResults\Debug\Microsoft.AspNetCore.Testing.Tests_net472_x64.html [net472|x64]

@MattGal any chance of getting access to one of these agents to debug this more?

MattGal · 2020-03-03T23:36:43Z

Nope, still failing. Interesting observation though, it's only failing on the net472 target, Core is fine.
Tests succeeded: F:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Testing.Tests\Debug\netcoreapp5.0\Microsoft.AspNetCore.Testing.Tests.dll [netcoreapp5.0|x64]

XUnit : error : Tests failed: F:\workspace\_work\1\s\artifacts\TestResults\Debug\Microsoft.AspNetCore.Testing.Tests_net472_x64.html [net472|x64]
@MattGal any chance of getting access to one of these agents to debug this more?

Set @Tratcher up on teams with a machine using this image, he'll report back with findings.

Tratcher · 2020-03-04T00:32:57Z

Progress, I was able to repro this locally on Server 2019.

PS C:\git\extensions\src\TestingUtils\Microsoft.AspNetCore.Testing\test> dotnet test passes
PS C:\git\extensions\src\TestingUtils\Microsoft.AspNetCore.Testing\test> dotnet msbuild -target:test fails
build -test fails
build -test -ci fails
eng\common\CIBuild.cmd fails

Found it: dotnet msbuild -target:test` uses a different test runner on core and .net. On core it uses dotnet.exe, but on .net it uses xunit.console.exe.

Unpacked success command:

C:\git\extensions\.dotnet\dotnet.exe exec --depsfile "C:\git\extensions\artifacts\bin\Microsoft.AspNetCore.Testing.Tests\Debug\netcoreapp5.0\Microsoft.AspNetCore.Testing.Tests.deps.json" --runtimeconfig "C:\git\extensions\artifacts\bin\Microsoft.AspNetCore.Testing.Tests\Debug\netcoreapp5.0\Microsoft.AspNetCore.Testing.Tests.runtimeconfig.json"  "C:\Users\myuser\.nuget\packages\xunit.runner.console/2.4.1/tools/netcoreapp2.0/xunit.console.dll" "C:\git\extensions\artifacts\bin\Microsoft.AspNetCore.Testing.Tests\Debug\netcoreapp5.0\Microsoft.AspNetCore.Testing.Tests.dll" -noautoreporters -xml "C:\git\extensions\artifacts\TestResults\Debug\Microsoft.AspNetCore.Testing.Tests_netcoreapp5.0_x64.xml" -html "C:\git\extensions\artifacts\TestResults\Debug\Microsoft.AspNetCore.Testing.Tests_netcoreapp5.0_x64.html" -notrait "Flaky:All=true"

Unpacked failure command:

C:\Users\myuser\.nuget\packages\xunit.runner.console\2.4.1\tools\net472\xunit.console.exe "C:\git\extensions\artifacts\bin\Microsoft.AspNetCore.Testing.Tests\Debug\net472\Microsoft.AspNetCore.Testing.Tests.dll" -noshadow -xml "C:\git\extensions\artifacts\TestResults\Debug\Microsoft.AspNetCore.Testing.Tests_net472_x64.xml" -html "C:\git\extensions\artifacts\TestResults\Debug\Microsoft.AspNetCore.Testing.Tests_net472_x64.html" -notrait "Flaky:All=true"

dotnet.exe has a manifest that opts into Win10 support. xunit.console.exe by comparison doesn't have any compat information in its manifest.

Why don't these tests fail on other Win10 queues? Theory: This is our first queue that uses RS4 or later. There aren't any Win10 tests for lower versions.

@dougbu @MattGal Does this make sense?

dougbu · 2020-03-04T17:15:04Z

Does this make sense?

Sounds reasonable. Should we switch to use the PlatformDetection extern approach? Or do you have a different solution in mind?

Tratcher · 2020-03-04T17:28:32Z

Let's start by skipping this test on .NET. The attribute isn't currently used for any .NET scenarios, only Core. I'll update the PR.

Second, I'll file a bug on xunit.

Tratcher · 2020-03-04T18:36:31Z

xunit/xunit#2076

MattGal

LGTM

dougbu · 2020-03-04T23:07:07Z

@JunTaoLuo @maryamariyan FYI our [MaximumOSVersion] and [MinimumOSVersion] testing attributes do not work correctly when targeting .NET Framework. Be careful (😺) if you plan to use these in dotnet/runtime.

@Tratcher

- add debug information for `Assert` failures in `MaximumOSVersionTest` - update src/TestingUtils/Microsoft.AspNetCore.Testing/test/MaximumOSVersionTest.cs - co-Authored-By: @Tratcher - try with VS2019 queues - skip `MaximumOSVersion` tests on .NET due to xunit issue - co-authored-by: @Tratcher Commit migrated from dotnet/extensions@26ff835

Tratcher · 2020-03-16T23:43:17Z

Related: dotnet/runtime#33643

@Tratcher

- add debug information for `Assert` failures in `MaximumOSVersionTest` - update src/TestingUtils/Microsoft.AspNetCore.Testing/test/MaximumOSVersionTest.cs - co-Authored-By: @Tratcher - try with VS2019 queues - skip `MaximumOSVersion` tests on .NET due to xunit issue - co-authored-by: @Tratcher Commit migrated from dotnet/extensions@26ff835

@Tratcher

- add debug information for `Assert` failures in `MaximumOSVersionTest` - update src/TestingUtils/Microsoft.AspNetCore.Testing/test/MaximumOSVersionTest.cs - co-Authored-By: @Tratcher - try with VS2019 queues - skip `MaximumOSVersion` tests on .NET due to xunit issue - co-authored-by: @Tratcher Commit migrated from dotnet/extensions@26ff835

dougbu requested a review from a team February 3, 2020 20:15

wtgodbe approved these changes Feb 3, 2020

View reviewed changes

dougbu added the area-infrastructure label Feb 3, 2020

dougbu added this to the 5.0.0-preview1 milestone Feb 3, 2020

dougbu commented Feb 5, 2020

View reviewed changes

Tratcher reviewed Feb 5, 2020

View reviewed changes

dougbu added a commit to dotnet-maestro-bot/Common that referenced this pull request Feb 21, 2020

Revert switch to Windows Server queues

edb9b1e

- wait for unblock of dotnet#2927 - however, do switch to VS2019 queues

dougbu added a commit to dotnet-maestro-bot/Common that referenced this pull request Feb 21, 2020

Revert switch to Windows Server queues

a54ae2a

- wait for unblock of dotnet#2927

dougbu and others added 4 commits March 3, 2020 12:46

Add debug information for Assert failures in MaximumOSVersionTest

74ee448

Update src/TestingUtils/Microsoft.AspNetCore.Testing/test/MaximumOSVe…

ab0d651

…rsionTest.cs Co-Authored-By: Chris Ross <[email protected]>

Try with VS2019 queues

c3406b2

Missed a spot

28dd096

Tratcher force-pushed the dougbu/master/server.queues branch from 50cac19 to 28dd096 Compare March 3, 2020 20:50

Tratcher mentioned this pull request Mar 4, 2020

xunit.console.exe does not declare support for Win10 xunit/xunit#2076

Closed

Skip MaximumOSVersion tests on .NET due to xunit issue

ec8c1b8

Tratcher approved these changes Mar 4, 2020

View reviewed changes

MattGal approved these changes Mar 4, 2020

View reviewed changes

dougbu merged commit 26ff835 into master Mar 4, 2020

dougbu deleted the dougbu/master/server.queues branch March 4, 2020 23:12

ghost locked as resolved and limited conversation to collaborators May 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[master] Switch to Windows Server queues #2927

[master] Switch to Windows Server queues #2927

dougbu commented Feb 3, 2020

dougbu Feb 5, 2020

dougbu Feb 5, 2020

Pilchie Feb 5, 2020

Tratcher Feb 5, 2020

MattGal Feb 5, 2020

Tratcher Feb 5, 2020

Tratcher Feb 5, 2020

Tratcher Mar 3, 2020

dougbu Mar 3, 2020

Tratcher Feb 5, 2020

dougbu commented Feb 6, 2020

Tratcher commented Feb 6, 2020

Pilchie commented Feb 6, 2020

dougbu commented Feb 15, 2020

Tratcher commented Mar 3, 2020

Tratcher commented Mar 3, 2020

MattGal commented Mar 3, 2020

Tratcher commented Mar 4, 2020 •

edited

Loading

dougbu commented Mar 4, 2020

Tratcher commented Mar 4, 2020

Tratcher commented Mar 4, 2020

MattGal left a comment

dougbu commented Mar 4, 2020

Tratcher commented Mar 16, 2020

[master] Switch to Windows Server queues #2927

[master] Switch to Windows Server queues #2927

Conversation

dougbu commented Feb 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dougbu commented Feb 6, 2020

Tratcher commented Feb 6, 2020

Pilchie commented Feb 6, 2020

dougbu commented Feb 15, 2020

Tratcher commented Mar 3, 2020

Tratcher commented Mar 3, 2020

MattGal commented Mar 3, 2020

Tratcher commented Mar 4, 2020 • edited Loading

dougbu commented Mar 4, 2020

Tratcher commented Mar 4, 2020

Tratcher commented Mar 4, 2020

MattGal left a comment

Choose a reason for hiding this comment

dougbu commented Mar 4, 2020

Tratcher commented Mar 16, 2020

Tratcher commented Mar 4, 2020 •

edited

Loading