dotnet build run into 429 and ignore Retry-After Header

[marcpopMSFT]

---

Issue moved from dotnet/sdk#15619

• Please respond to @fotto1.

---

From @fotto1 on Tuesday, February 2, 2021 1:11:15 PM

We using for our project docker image mcr.microsoft.com/dotnet/core/sdk:3.1 with digest mcr.microsoft.com/dotnet/core/sdk@sha256:0fece15a102530aa2dad9d247bc0d05db6790917696377fc56a8465604ef1aff to run dotnet cli.

Our nuget packages are stored on an gitlab package manager version 13.8

After there are to many http request nuget receives 429 following RFC6585 with Retry-After Header in Response like described in https://tools.ietf.org/html/rfc6585#section-4

Instead of do a retry later dotnet build continue with download next packages and exit with 1 after third download fails with 429

Microsoft (R) Build Engine version 16.7.2+b60ddb6f4 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.
  Determining projects to restore...
  Retrying 'FindPackagesByIdAsync' for source 'https://SERVER_URL/api/v4/projects/PROJECT_ID/packages/nuget/download/microsoft.extensions.configuration.binder/index.json'.
  Response status code does not indicate success: 429 (Too Many Requests).
  Retrying 'FindPackagesByIdAsync' for source 'https://SERVER_URL/api/v4/projects/PROJECT_ID/packages/nuget/download/microsoft.extensions.configuration.binder/index.json'.
  Response status code does not indicate success: 429 (Too Many Requests).
  Retrying 'FindPackagesByIdAsync' for source 'https://SERVER_URL/api/v4/projects/PROJECT_ID/packages/nuget/download/microsoft.extensions.configuration.environmentvariables/index.json'.
  Response status code does not indicate success: 429 (Too Many Requests).

Can you please check how 429 is handled in dotnet build in case nuget is configured

[marcpopMSFT]

---

Issue moved from dotnet/sdk#15619

• Please respond to @dotnet-issue-labeler[bot].

---

From @dotnet-issue-labeler[bot] on Tuesday, February 2, 2021 1:11:18 PM

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[nkolev92]

Consider using maxHttpRequestsPerSource as a client side workaround.

Sample: (tweak the connection count)

<config>
    <add key="maxHttpRequestsPerSource" value="16" />
</config>

Related: https://github.com/NuGet/docs.microsoft.com-nuget/issues/2033

[fotto1]

Thanks for the hotfix @nkolev92 , I will try this. Anyway will a proper handling of 429 ever implemented?

I work for a large organization and will be not the only one with this problem. Furthermore, I also expect that other companies will face the same issue when they use NuGet package manager.

[nkolev92]

Hey @fotto1,

We prioritize issues based on customer reports, workarounds available etc. At this point, I can't make a commitment when this will be fixed.

The NuGet client/server protocol currently does not handle 429.
Not many of the servers that the client has interacted have returned a 429 in the past. Note that in addition to 429, there are other status codes which are not implemented. This is just one of those.

[zivkan]

We might need to raise the priority of this. Our latest insertions into the .NET SDK failed because of this.

PR has failed checks: dotnet/sdk#15854
Check failed because tests failed. Test failed because NuGet restore got HTTP 429 reponses from Azure Artifacts: https://dev.azure.com/dnceng/public/_build/results?buildId=993350&view=ms.vss-test-web.build-test-results-tab&runId=31123132&resultId=100686&paneView=debug

This is in no way specific to NuGet's insertion into the .NET SDK, hence all of dnceng might be affected by these HTTP responses. We'll also need to work with the Azure Artifacts team to understand if their limits are too low. But maybe we need to implement at least a basic solution to this request.

[zivkan]

To the people who 👍 my comment, or others who would like this feature request implemented, please react 👍 to the original comment/post, so that this issue will bubble up the list of all open issues sorted by 👍

[fgreinacher]

@marcpopMSFT @nkolev92 @heng-liu Would you accept a community contribution for this?

[zivkan]

I wanted to be helpful and write an comment with links helping you to get started, but then I realized that all the product code that needs to be changed is in HttpRetryHandler, so it was actually quick enough to do myself: https://github.com/NuGet/NuGet.Client/compare/dev-zivkan-handle-http-429

However, I haven't yet tested this whatsoever.

If you'd like to try to figure out NuGet.Protocol's tests, or build and then somehow manually test this code against a server you know (or you can force) to return HTTP 429 responses, then I'd love that help.

Otherwise, I'll try to figure out unit tests over the upcoming days/weeks.

[fgreinacher]

I wanted to be helpful and write an comment with links helping you to get started, but then I realized that all the product code that needs to be changed is in HttpRetryHandler, so it was actually quick enough to do myself: https://github.com/NuGet/NuGet.Client/compare/dev-zivkan-handle-http-429

However, I haven't yet tested this whatsoever.

If you'd like to try to figure out NuGet.Protocol's tests, or build and then somehow manually test this code against a server you know (or you can force) to return HTTP 429 responses, then I'd love that help.

Otherwise, I'll try to figure out unit tests over the upcoming days/weeks.

Wow, nice!

Happy to help however I can! Is there a straightforward way to get your changes in a custom dotnet CLI?

[zivkan]

https://github.com/NuGet/NuGet.Client/blob/dev/docs/debugging.md#debugging-restore-task-in-dotnetexe
https://github.com/NuGet/NuGet.Client/blob/15be58f38274a1d78c3283560d3d57aadd3ba3dd/scripts/nuget-debug-helpers.ps1#L38

Replace msbuild with dotnet, and it's the same, as mentioned in debugging.md. Also, I'm pretty sure only NuGetRestoreTargets needs to be defined, because the other two properties are defined in nuget.targets.

git clone https://github.com/NuGet/NuGet.Client
cd NuGet.Client
git switch dev-zivkan-handle-http-429
# alternative to configure.ps1
git submodule init
git submodule update
dotnet restore build\bootstrap.proj
# if you're on windows, you might need to run `build.ps1 -SkipUnitTest` to stop strong name checking for NuGet's assemblies
# but that's probably only for .NET Framework, I think NetCoreApp might not care about partial signed assemblies.
dotnet build src\NuGet.Core\NuGet.Build.Tasks -c Release

# test something
dotnet restore -p:NuGetRestoreTargets=c:\path\to\NuGet.Client\artifacts\NuGet.Build.Tasks\bin\Release\net5.0\NuGet.targets path\to\test\solution.sln

[fgreinacher]

https://github.com/NuGet/NuGet.Client/blob/dev/docs/debugging.md#debugging-restore-task-in-dotnetexe https://github.com/NuGet/NuGet.Client/blob/15be58f38274a1d78c3283560d3d57aadd3ba3dd/scripts/nuget-debug-helpers.ps1#L38

Replace msbuild with dotnet, and it's the same, as mentioned in debugging.md. Also, I'm pretty sure only NuGetRestoreTargets needs to be defined, because the other two properties are defined in nuget.targets.

git clone https://github.com/NuGet/NuGet.Client
cd NuGet.Client
git switch dev-zivkan-handle-http-429
# alternative to configure.ps1
git submodule init
git submodule update
dotnet restore build\bootstrap.proj
# if you're on windows, you might need to run `build.ps1 -SkipUnitTest` to stop strong name checking for NuGet's assemblies
# but that's probably only for .NET Framework, I think NetCoreApp might not care about partial signed assemblies.
dotnet build src\NuGet.Core\NuGet.Build.Tasks -c Release

# test something
dotnet restore -p:NuGetRestoreTargets=c:\path\to\NuGet.Client\artifacts\NuGet.Build.Tasks\bin\Release\net5.0\NuGet.targets path\to\test\solution.sln

@zivkan Thanks a lot for the great help setting this up! It worked out great and I was able to play around with your branch a bit.

Unfortunately the changes don't seem to be effective for dotnet restore as the retry logic is disabled at the call site:

https://github.com/NuGet/NuGet.Client/blob/5c09eab016e12b2dbb083e4da87588e4ac5dfa36/src/NuGet.Core/NuGet.Protocol/Utility/FindPackagesByIdNupkgDownloader.cs#L282

... as already indicated by your comment at https://github.com/NuGet/NuGet.Client/compare/dev-zivkan-handle-http-429#diff-2ae20246c40c717c164cb40b9858455a72a6bc262b23e4d0a14fc7824fc1d860R265-R267 😀

I guess we're now left with a. either implementing the "retry-after" logic at the call site or b. remove the retry logic from the call site so that the new code becomes effective... 🤷

[fgreinacher]

@zivkan I noticed that you removed this from the sprint. Can I help bring this forward somehow? Also happy to dig deeper into the code if this does not have enough priority for you at the moment.

[zivkan]

I was looking into this in December, but I only had a day or two of time available. The fix for this is needed to also fix #11226, which is currently in this sprint, but honestly I'm not confident that I'll have enough time given all the other work I have planned for the end of the month (and this doesn't include work I have scheduled in private repos and Visual Studio's Developer Community, and Microsoft internal issue tracking. But most of our planned work is here on GitHub). If you click that link after January 2022, it'll be empty except for completed work, since all work I had assigned will be bumped to other sprints.

Anyway, my idea for the fix would be to add a Action onRetry callback to HttpRetryHandler, and therefore probably HttpSource as well. Then plumb it though everywhere that needs it. Finally, any NuGet.Protocol "resource" that does its own retry logic can then be refactored to use HttpRetryHandler's retries, stop doing its own, and whatever cleanup it needs to do between retries, it does in that callback.

Finally, HttpRetryHandler needs to include downloading/processing the stream, not just the httpClient.SendAsync, returning as soon as the headers are finished and the body is about to start. HttpSource has a bunch of ProcessStream methods, not just a Get method, so it's partly there, but from memory it does need some work. Technically this isn't needed for HTTP 429 handling, but it's exactly what (I think) causes #11226's bug. We need to reset the output stream's position to zero on retries.

This is a little tricky/time consuming because most of these APIs are public, and I feel there's a high probability that developers using NuGet.Protocol in their own apps are using these existing APIs. Therefore, we need to figure out how to implement all this without any breaking changes, and minimizing the number of new APIs added. Working on a product which is both a tool and a library is frustrating 😠

So now it's just a question if I can find time in a sprint to work on it. I currently have 11226 in my quality week, but I'm definitely also overcommitted. However, I've also made good progress on my non-quality week work this sprint, so if I'm lucky I'll finish early and have extra time for these types of issues before the end of the month.

If someone else wants to try it and send us a PR, that's fine, but I will be looking very closely at every PublicAPI.*.txt file change because NuGet's overexposure of public APIs is something that has forced us to revert changes in the past, and then all future changes are slower because now everyone's scared of breaking a public API that might need to be reverted and then reworked. So, I don't want to add new poorly designed public APIs that will cause us additional technical debt in the future.

[fgreinacher]

Thanks @zivkan for the in-depth answer, I can fully understand your situation!

Your input is surely super helpful to implement the necessary changes. It looks like a lot of work, so I am not sure when I'll find the time needed to provide a high-quality contribution. I'll definitely ping you before starting bigger changes!

[nkolev92]

Hey all,

In 6.2, we added a fix that changed how NuGet did retries and added an exponential back-off, NuGet/NuGet.Client@dc37ed5, NuGet/NuGet.Client@73c1eea.

While this isn't a fix specific to the 429 status code, it is very likely to improve this scenario as well.

Please let us know if you are still seeing similar issues after this fix by upvoting this comment.

[kasperk81]

@danmoseley in reference to dotnet/sdk#27843 (comment).

as noted above. proper handling of 429 means respecting the Retry-After response header (value is in milliseconds). can be handled internally with a backoff # of times property default to 5. optionally, connect the property to configuration so user can specify a different value. cc @jeffkl

[danmoseley]

Thanks @kasperk81 . IIRC the changes linked by @nkolev92 were actually originally motivated by making our CI builds more stable, and we certainly have them already. Are you suggesting the HttpRetryHandler should add special treatment for 429 (HttpStatusCode.TooManyRequests)?
cc @MattGal who I think wrote this originally.

[MattGal]

Thanks @kasperk81 . IIRC the changes linked by @nkolev92 were actually originally motivated by making our CI builds more stable, and we certainly have them already. Are you suggesting the HttpRetryHandler should add special treatment for 429 (HttpStatusCode.TooManyRequests)? cc @MattGal who I think wrote this originally.

While I fully support honoring the retry-after field and am happy to comment, I only added retries and backoff in places where it was lacking. I made no efforts to try to honor the Retry-After field as there was nothing in place as far as I can tell to handle HTTP 429s at the time, and I was trying to change as little as possible while working around intermittent known issues with Azure DevOps package feeds which in most cases was achieved by having any retry-with-delay at all.

[danmoseley]

@MattGal is there any way to query how often our infra fails with 429? eg grep the world for Response status code does not indicate success: 429

[MattGal]

Ulises on my team (can't tag him here, but linked the issue) has been working on the ability to grep through build logs for this. Short answer is "I don't think so" but I will check out what I can find in the 1ES AzureDevOps Kusto instance, and get back to you.

[danmoseley]

tagging should work here. @ulisesh

[danmoseley]

Memorializing the failure in question here, anyway, until we have data on whether it's common --

Process ID: 3712
    Microsoft.NET.Build.Tests.GivenThatWeWantToUseAnalyzers.It_resolves_multitargeted_analyzers [FAIL]
      Expected command to pass but it did not.
      File Name: C:\h\w\BDDE09E9\p\d\dotnet.exe
      Arguments: msbuild /t:WriteValuesToFile C:\h\w\BDDE09E9\t\dotnetSdkTests\1v3w1tea.knb\It_resolves_m---47570CA8\It_resolves_multitargeted_analyzers\It_resolves_multitargeted_analyzers.csproj /p:ValueName=Analyzer /restore -p:TargetFramework=net472
      Exit Code: 1
      StdOut:
Running C:\h\w\BDDE09E9\p\d\dotnet.exe msbuild /t:Build C:\h\w\BDDE09E9\t\dotnetSdkTests\1v3w1tea.knb\It_handles_co---209447AF\ContentFiles\ContentFiles.csproj /restore
      MSBuild version 17.4.0-preview-22458-02+a3b647d76 for .NET
        Determining projects to restore...
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/system.runtime.compilerservices.unsafe/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/49a1bb2b-12b0-475f-adbd-1560fc76be38/nuget/v3/flat2/microsoft.bcl.asyncinterfaces/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/516521bf-6417-457e-9a9c-0a4bdfde03e7/nuget/v3/flat2/microsoft.bcl.asyncinterfaces/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/4828dfac-e9f8-49bc-acb6-319be99331fc/nuget/v3/flat2/system.buffers/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/825db618-e3eb-4426-ba54-b1d6e6c944d8/nuget/v3/flat2/system.memory/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/4828dfac-e9f8-49bc-acb6-319be99331fc/nuget/v3/flat2/microsoft.bcl.asyncinterfaces/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/system.threading.tasks.extensions/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/825db618-e3eb-4426-ba54-b1d6e6c944d8/nuget/v3/flat2/system.buffers/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/1a5f89f6-d8da-4080-b15f-242650c914a8/nuget/v3/flat2/microsoft.bcl.asyncinterfaces/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/516521bf-6417-457e-9a9c-0a4bdfde03e7/nuget/v3/flat2/system.runtime.compilerservices.unsafe/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/516521bf-6417-457e-9a9c-0a4bdfde03e7/nuget/v3/flat2/system.text.encodings.web/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/825db618-e3eb-4426-ba54-b1d6e6c944d8/nuget/v3/flat2/system.runtime.compilerservices.unsafe/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/49e5305d-d845-4a14-9d69-6f5dbfb9570c/nuget/v3/flat2/microsoft.bcl.asyncinterfaces/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/system.valuetuple/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/system.memory/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/e31c6eea-0277-49f3-8194-142be67a9f72/nuget/v3/flat2/system.valuetuple/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/1a5f89f6-d8da-4080-b15f-242650c914a8/nuget/v3/flat2/system.numerics.vectors/index.json'.
        Response status code does not indicate success: 429 ().
        Failed to download package 'System.Numerics.Vectors.4.5.0' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.numerics.vectors/4.5.0/system.numerics.vectors.4.5.0.nupkg'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/system.numerics.vectors/index.json'.
        Response status code does not indicate success: 429 ().
        Failed to download package 'System.Numerics.Vectors.4.5.0' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.numerics.vectors/4.5.0/system.numerics.vectors.4.5.0.nupkg'.
        Response status code does not indicate success: 429 ().
        Failed to download package 'System.Memory.4.5.3' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.memory/4.5.3/system.memory.4.5.3.nupkg'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/825db618-e3eb-4426-ba54-b1d6e6c944d8/nuget/v3/flat2/system.buffers/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/system.numerics.vectors/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/system.memory/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/1a5f89f6-d8da-4080-b15f-242650c914a8/nuget/v3/flat2/system.numerics.vectors/index.json'.
        Response status code does not indicate success: 429 ().
        Failed to download package 'System.Memory.4.5.3' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.memory/4.5.3/system.memory.4.5.3.nupkg'.
        Response status code does not indicate success: 429 ().
        Failed to download package 'System.Numerics.Vectors.4.5.0' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.numerics.vectors/4.5.0/system.numerics.vectors.4.5.0.nupkg'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/system.numerics.vectors/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/1a5f89f6-d8da-4080-b15f-242650c914a8/nuget/v3/flat2/system.numerics.vectors/index.json'.
Process ID: 6920
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/system.memory/index.json'.
        Response status code does not indicate success: 429 ().
        Failed to download package 'System.Memory.4.5.3' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.memory/4.5.3/system.memory.4.5.3.nupkg'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/825db618-e3eb-4426-ba54-b1d6e6c944d8/nuget/v3/flat2/system.buffers/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/system.numerics.vectors/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/1a5f89f6-d8da-4080-b15f-242650c914a8/nuget/v3/flat2/system.numerics.vectors/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/system.memory/index.json'.
        Response status code does not indicate success: 429 ().
        Failed to download package 'System.Memory.4.5.3' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.memory/4.5.3/system.memory.4.5.3.nupkg'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/system.numerics.vectors/index.json'.
        Response status code does not indicate success: 429 ().
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/825db618-e3eb-4426-ba54-b1d6e6c944d8/nuget/v3/flat2/system.buffers/index.json'.
        Response status code does not indicate success: 429 ().
        Failed to download package 'System.Memory.4.5.3' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.memory/4.5.3/system.memory.4.5.3.nupkg'.
        Response status code does not indicate success: 429 ().
      C:\h\w\BDDE09E9\t\dotnetSdkTests\1v3w1tea.knb\It_resolves_m---47570CA8\It_resolves_multitargeted_analyzers\It_resolves_multitargeted_analyzers.csproj : error NU1301: Failed to retrieve information about 'System.Numerics.Vectors' from remote source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/system.numerics.vectors/index.json'. [TargetFramework=net472]
      C:\h\w\BDDE09E9\t\dotnetSdkTests\1v3w1tea.knb\It_resolves_m---47570CA8\It_resolves_multitargeted_analyzers\It_resolves_multitargeted_analyzers.csproj : error NU1301: Failed to retrieve information about 'System.Numerics.Vectors' from remote source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/system.numerics.vectors/index.json'. [TargetFramework=net472]
        Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/825db618-e3eb-4426-ba54-b1d6e6c944d8/nuget/v3/flat2/system.buffers/index.json'.
        Response status code does not indicate success: 429 ().
        Failed to restore C:\h\w\BDDE09E9\t\dotnetSdkTests\1v3w1tea.knb\It_resolves_m---47570CA8\It_resolves_multitargeted_analyzers\It_resolves_multitargeted_analyzers.csproj (in 1.67 min).

For completeness we do not know of course whether doing anything special for 429 for this server would help. Rerunning the build succeeded.

[marcpopMSFT]

I will add that the SDK still hits 429s quite regularly. This was more common before we started predownloading many packages (we can't pre-download current version runtime specific packages easily as we'd be constantly chasing those versions). However I still see this pretty much every time we have branding or branching as usually those trigger lots of PRs all at the same time.

[ulisesh]

@MattGal is there any way to query how often our infra fails with 429? eg grep the world for Response status code does not indicate success: 429

Just in dotnet/sdk repo, I found 377 failed working items with the string success: 429 out of the 4485 workitems that have failed in the last 7 days in the repo

[ulisesh]

@marcpopMSFT @danmoseley is it ok if I create a known issue (in this repo) to track how often we get 429s?

[danmoseley]

@ulisesh it's not my repo, but I can't imagine how they'd mind. Seems like a good idea if it helps.

[kasperk81]

I found 377 failed working items with the string success: 429 out of the 4485 workitems that have failed in the last 7 days in the repo

8.4% of failures are 429, worth looking into. cloud services are known to issue 429 errors based on load and scale. honoring retry-after header in client might not "fix" the issue, but it will put the blame on cosmosdb or whichever storage provider is used under the hoods of azure devops nuget packages. with that we can bounce the ball in their court.

[danmoseley]

@aortiz-msft do you know who might know when https://pkgs.dev.azure.com/ issues a 429 and whether it provides and respects a retry-after? (I can't repro so I can't say)

[zivkan]

I just started an internal email thread about this, since it's discussing Azure DevOps and Azure services, that are not on GitHub.