dotnet nuget push fails when pushing a relatively large package (3.4MB) to Github

[datvm]

NuGet Product Used

dotnet.exe

Product Version

.NET 7.0.401, Nuget 6.7.0.127

Worked before?

No response

Impact

I'm unable to use this version

Repro Steps & Context

When pushing a relatively large (but in no way huge, about 3.4MB) Nuget package to a custom feed hosted by Github, it fails with the following error:

dotnet nuget push --source "github" --skip-duplicate $file.FullName

Pushing [packagename] to 'https://nuget.pkg.github.com/[org]'...
  PUT https://nuget.pkg.github.com/[org]/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/[org]/'. The request will now be retried.
Error while copying content to a stream.
  Unable to write data to the transport connection: An established connection was aborted by the software in your host machine..
  An established connection was aborted by the software in your host machine.

All retries fail as well.

HOWEVER, either using Nuget CLI client (downloaded at https://www.nuget.org/downloads) or manually crafting a PUT request using cUrl or HTTP client works well and the package is uploaded successfully. It only happens to dotnet nuget. This happens consistently and not randomly as well. I have two packages, it consistently uploads the smaller one successfully but fails on the bigger one. The successful standalone nuget command is this:

nuget push -Source "github" -SkipDuplicate $file.FullName

Further technical details

dotnet --info:

.NET SDK:
 Version:   7.0.401
 Commit:    eb26aacfec

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22621
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\7.0.401\

Host:
  Version:      7.0.11
  Architecture: x64
  Commit:       ecb34f85ec

.NET SDKs installed:
  7.0.401 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 7.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

dotnet nuget --version:

NuGet Command Line
6.7.0.127

nuget version (the standalone one):

> nuget help
NuGet Version: 6.7.0.127

Verbose Logs

No response

[donnie-msft]

Hello,
Our first guess is that something on your machine like an Antivirus or Firewall is restricting the traffic. Could you check both of these and see of nuget.exe is on the allow list, while dotnet.exe is not?
Thanks!

[datvm]

@donnie-msft Hi, I just checked. I only have Windows Firewall and both dotnet and nuget are not on the list. I don't think firewall should be the issue because as mentioned, the smaller package is successfully uploaded without issue (with dotnet nuget) so dotnet does have access to the Internet (Windows Firewall doesn't discriminate traffic, it either blocks all or allow all AFAIK).

Do you need any log file I can provide? I can reproduce it consistently, one successful upload for a smaller file and one fail upload for the bigger file.

[donnie-msft]

If you can try running a Fiddler trace when performing the dotnet nuget push, it may provide insight where the HTTP communication is breaking down. There's information on how to setup Fiddler for HTTPS traffic in https://learn.microsoft.com/en-us/power-query/web-connection-fiddler.

My guess at the difference you're seeing is due to the difference in HTTP handling between nuget.exe (.NET Framework) and dotnet.exe (.NET Core). What exactly that may be isn't obvious, especially since I'd expect more reports of this issue. In the meantime, I'll reach out to the .NET runtime team to see if they have ideas.

[datvm]

Hi, I have tried pushing with Fiddler on, unfortunately it fails with HTTP/1.1 504 Fiddler - Send Failure for both nuget and dotnet nuget. After closing Fiddler so it does not interfere with the traffic anymore, dotnet nuget fails with the previous error and nuget successfully uploads the package. Here are the requests (the body seems to be identical):

dotnet nuget:

PUT https://nuget.pkg.github.com/[my-organization]/ HTTP/1.1
Host: nuget.pkg.github.com
Transfer-Encoding: chunked
X-NuGet-Session-Id: 84a32052-268e-49fa-9ea1-e65dafd40190
User-Agent: NuGet xplat/6.7.0 (Microsoft Windows 10.0.22621)
X-NuGet-Client-Version: 6.7.0
Accept-Language: en-US
Accept-Encoding: gzip, deflate
Content-Type: multipart/form-data; boundary="3519f204-5d5c-4e33-963d-dc3e19dc6fdf"

nuget:

PUT https://nuget.pkg.github.com/[my-organization]/ HTTP/1.1
X-NuGet-Session-Id: dc757060-0530-463a-8eae-2615fe745772
user-agent: NuGet Command Line/6.7.0 (Microsoft Windows NT 10.0.22621.0)
X-NuGet-Client-Version: 6.7.0
Accept-Language: en-US
Content-Type: multipart/form-data; boundary="94233684-24e8-457c-94f0-e187dc0e8442"
Host: nuget.pkg.github.com
Transfer-Encoding: chunked
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

Fiddler error:

HTTP/1.1 504 Fiddler - Send Failure

[Fiddler] SendRequest() failed: System.IO.IOException Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. < An established connection was aborted by the software in your host machine

Do you think the Connection: Keep-Alive header is the difference here? Also when I crafted the curl request by myself, I don't really have those headers, just the Authorization header and it was successful (I didn't test it with curl this time though).

[donnie-msft]

I believe Fiddler has some guidance on working with HTTPS connections, so not sure if that's part of the errors you're seeing with it. It's interesting that the IOException is the same "aborted" message you saw from dotnet nuget push. Did you say you have no 3rd party Antivirus on this machine?

A few questions coming to mind:

1. Can you write an app to create an HttpWebRequest and reproduce the same problem?
   1. I don't suspect the header is a problem, but this would make it easy to test out.
   2. https://learn.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest?view=netframework-4.8
2. What OS are you on, and is it a container, etc? Can you successfully push from a different machine on the same network?
3. Worth a reminder that TLS 1.2 is required in Windows https://devblogs.microsoft.com/nuget/nuget-org-will-permanently-remove-support-for-tls-1-0-and-1-1-on-june-15th/
   1. I'd check the Fiddler Options for anything obvious (eg, make sure the HTTPS / Protocol settings have TLS 1.2 enabled.

[datvm]

Hi, after some testing, here are some more info:

• I tried using HttpClient in .NET 7, same issue with the same error. I sent a request to https://check-tls.akamai.io/v1/tlsinfo.json to check for TLS info and the default value is 1.3:

{
"tls_sni_status": "present",
"scheme": "https",
"tls_version": "tls1.3",
"tls_sni_value": "check-tls.akamai.io",
"host_header_value": "check-tls.akamai.io",
"tls_cipher_name": "TLS_AES_256_GCM_SHA384",
"alpn": "",
"client_ip": "[removed]",
"client_ip_version": "ipv4",
"client_port": [removed],
"user_agent": "None",
"output_version": "0.1.21",
"timestamp": 1697556916
}

I also manually set the TLS version to 1.2. Both 1.2 and 1.3 has the same issue:

Unhandled exception. System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|153_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn)

Interestingly, this time instead of curl I used another HTTP client Insomnia, and it successfully pushes the package as well. No header needed except the Authorization and Content-Type for multipart.

* Preparing request to https://nuget.pkg.github.com/[organization]/
* Current time is 2023-10-17T15:34:35.986Z
* Enable automatic URL encoding
* Using HTTP 1.1
* Enable timeout of 30000ms
* Disable SSL validation
* Enable cookie sending with jar of 2 cookies
*   Trying 140.82.114.34:443...
* Connected to nuget.pkg.github.com (140.82.114.34) port 443 (#0)
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.pkg.github.com
*  start date: Mar 16 00:00:00 2023 GMT
*  expire date: Apr 12 23:59:59 2024 GMT
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):

> PUT /LukeVApps/ HTTP/1.1
> Host: nuget.pkg.github.com
> Content-Type: multipart/form-data; boundary=X-INSOMNIA-BOUNDARY
> Authorization: Basic [RemovedToken]
> Accept: */*
> Content-Length: 3704572

* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):

| (64 KB hidden)

* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):

| (64 KB hidden)

* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):

// More like the above

| (64 KB hidden)

* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):

| (33.7 KB hidden)

* We are completely uploaded and fine
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse

< HTTP/1.1 200 OK
< access-control-allow-methods: OPTIONS, PUT
< Access-Control-Allow-Origin: *
< Content-Security-Policy: default-src 'none';
< Server: GitHub Registry
< Strict-Transport-Security: max-age=31536000;
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< x-nuget-warning: Please use the --api-key option when publishing to GitHub Packages
< X-XSS-Protection: 1; mode=block
< Date: Tue, 17 Oct 2023 15:34:52 GMT
< Content-Length: 59
< Content-Type: text/plain; charset=utf-8
< X-GitHub-Request-Id: 880D:741B:335D76:48766B:652EA98C


* Received 59 B chunk
* Connection #0 to host nuget.pkg.github.com left intact

I also suspected it's due to the File Stream, so I loaded it into a MemoryStream (and reset it with memoryStream.Seek(0, SeekOrigin.Begin); before passing) and it doesn't work still. Here's my attempt:

using System.Net;
using System.Text;

const string Package = @"D:\Temp\Test.Assets.1.0.0.nupkg";
const string Source = "https://nuget.pkg.github.com/[organization]/";
//var url = $"{Source}api/v2/package";
var url = $"{Source}";

const string User = "[email]";
var key = Environment.GetEnvironmentVariable("GithubKey");

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

using var handler = new HttpClientHandler();

handler.SslProtocols = System.Security.Authentication.SslProtocols.Tls12;
using var http = new HttpClient(handler);

//using var http = new HttpClient();

Console.WriteLine(await http.GetStringAsync("https://check-tls.akamai.io/v1/tlsinfo.json"));

using var req = new HttpRequestMessage(HttpMethod.Put, url);
var auth = Convert.ToBase64String(Encoding.ASCII.GetBytes($"{User}:{key}"));
//req.Headers.Add("Authorization", $"Basic {auth}");

using var form = new MultipartFormDataContent();
var packageStream = File.OpenRead(Package);

// Just to be sure, load it into memory
var memoryStream = new MemoryStream();
await packageStream.CopyToAsync(memoryStream);
memoryStream.Seek(0, SeekOrigin.Begin);

form.Add(new StreamContent(memoryStream, (int)memoryStream.Length));

req.Content = form;

using var res = await http.SendAsync(req);
Console.WriteLine(res.StatusCode);
Console.WriteLine(await res.Content.ReadAsStringAsync());

Here's the file if you want to test. It's just a font and a JS in there for testing:

Test.Assets.1.0.0.nupkg.zip

[zivkan]

I think a wireshark capture would be most useful here. Fiddler deals with HTTP requests and responses, but it doesn't give us enough information about what's happening within a single request, at the tcp packet level. Specifically, I'm curious at what point the "An established connection was aborted by the software in your host machine." error message is thrown. Was the header sent? Was the body started? did the body complete? How much of the file was sent? Was any response headers received? Wireshark, or any other packet capture, should tell us more.

Sysinternals ProcMon might also be helpful, because it'll tell us about every file read and network read/write API call, so we can check if Windows is giving any additional information that .NET is losing. It won't give us as much TCP/HTTP information as wireshark, but it will tell us what position with the nupkg the filestream was at, and how many bytes were read/written to the TCP socket.

We're also reaching out to .NET internally, to see if we can get any additional information.

Finally, I downloaded your package, edited it to make the repo URL look like it's under my own account, and then I tried to push. But no matter what I try, I keep getting HTTP 401 unauthorized responses. I made the the token I saved has write:packages scope, but it just won't work 😕 So, I wanted to check if I'm experiencing the same issue, but I can't even authenticate

[datvm]

@zivkan oh I forgot to mention a small details, to check if it actually reached Github, I commented out the Authorization header (as you see in my code above) so I can see a 401 response but I didn't as well so apparently the response was cut even before authorization. If you get 401 response from using my code above, make sure you uncomment the Authorization header 😅

[zivkan]

I wasn't using your test program, but just dotnet nuget push. I tried again today, and I found a bug in GitHub's server implementation. When NuGet has cached the service index (index.json), then the first, and only, HTTP that NuGet sends is the PUT request to the push endpoint. In this case, GitHub is not sending the WWW-Authenticate header. According to mozilla's docs (I'm too lazy to check the RFC, mozilla's docs is quicker to find), the header is mandatory. Therefore, GitHub's server may be violating the HTTP spec.

However, when I run dotnuget nuget locals http-cache --clear and then try push again, the test package works, and I successfully pushed the package.

Therefore @datvm, I can't reproduce your problem. Given the error message explicitly says "aborted by the software in your host machine", I'd like to reiterate that this appears to be an issue local to your machine. I don't understand why it only happens with the dotnet CLI and not with nuget.exe, that doesn't make sense to me. But if it's only happening on your machine, then it's not a .NET or NuGet issue, and the win32 error saying it's "your host machine" suggests it's some software issue on your machine, like anti-virus or firewall software.

[ghost]

This issue has been automatically marked as stale because we have not received a response in 14 days. It will be closed if no further activity occurs within another 14 days of this comment.

[KillyMXI]

I've had the same issue with even bigger package (~6MB).
I was able to push it successfully once in a few dozens of attempts. So, it is consistent, but not completely consistent.

The discussion under #9775 was fruitful for me.
I see no mention of --api-key here, so the reason might be the same.