Make the plugin initialization code non cancellable

[nkolev92]

Bug

Fixes: NuGet/Home#8648
Regression: No

• Last working version:
• How are we preventing it in future:

Fix

Details: The scenario is NuGet restore with one or more protected feeds (in every report we've been dealing with Azure Devops feeds, but that's because majority of the reports are internal).

The manifestation of this is restore failing with "A task was canceled" exception.
This is something that happens with the latest NuGet (5.2, 5.3 were both tested) and the latest version of the artifacts-credprovider 0.1.18.

The current reasoning of the investigation is that we have 2 issues:

1. Early cancellation of operations that we can’t recover from (plugin startup)
2. Threadpool starvation.

A child of NuGet/Home#8528.

The plugin infra cannot handle cancellations that happen during the plugin init process so we simply don't honor those said cancellations.

##[error]Problem starting the plugin 'C:\Users\dlab14.nuget\plugins\netfx\CredentialProvider.Microsoft\CredentialProvider.Microsoft.exe'. Plugin 'CredentialProvider.Microsoft' failed within 0.310 seconds with exit code -1.
Problem starting the plugin 'C:\Users\dlab14.nuget\plugins\netfx\CredentialProvider.Microsoft\CredentialProvider.Microsoft.exe'. Plugin 'CredentialProvider.Microsoft' failed within 0.310 seconds with exit code -1.
 
This is thrown here,

NuGet.Client/src/NuGet.Core/NuGet.Protocol/Plugins/PluginManager.cs

Line 257 in dc72141

pluginCreationResult = new PluginCreationResult(

and suggests that for some reason the plugin initialization was cancelled within 0.3s.
 
Long story short, the reason for this is NuGet's internal handling of package downloading.

NuGet.Client/src/NuGet.Core/NuGet.Protocol/HttpSource/HttpRetryHandler.cs

Line 89 in dc72141

response = await TimeoutUtility.StartWithTimeout(

Basically we hit all the sources at the same time (well http ones at least), and cancel if one feed beats another.
Note that PC & PR have different download logic and since there are PC downloads here while the rest of the repros are PR, the http calling patterns are different enough that the PC case seems to be more likely to repro.

Testing/Validation

Tests Added: No
Reason for not adding tests: Difficult to test.
Validation: See details https://microsoft.sharepoint.com/teams/NuGet/_layouts/OneNote.aspx?id=%2Fteams%2FNuGet%2FTeam%2FNuGet&wd=target%28Engineering.one%7C219AABAA-18B3-46F4-83E8-D3188D83DE22%2FPlugin%3A%20Task%20was%20canceled%20investigation%7C6939BE64-BF48-43CA-BC02-BE80F590AFE7%2F%29
onenote:https://microsoft.sharepoint.com/teams/NuGet/Team/NuGet/Engineering.one#Plugin%20Task%20was%20canceled%20investigation&section-id={219AABAA-18B3-46F4-83E8-D3188D83DE22}&page-id={6939BE64-BF48-43CA-BC02-BE80F590AFE7}&end

Verified the fix on the NuGet Gallery deployment repro and the RPC.Contracts repro.
This fix seems to be addressing those issues.