[BUG] Too many parallel network connections open while installation

[ankurparihar]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

Hi,

For a project npm install opens nearly ~2100 connections in parallel, the same project installed via pnpm only takes about 18 concurrent connection.

This causes a huge bandwidth issue for our internal CI/CD pipelines.

Expected Behavior

Please provide a way to limit concurrent connection.

Steps To Reproduce

1. In this environment

• MacOS / Linux

2. First clean cache

• npm cache clean --force
• pnpm store prune

2. Run npm i / pnpm i in any project containing package.json
3. in parallel run watch -n 0.2 "ss -tunap | wc -l" - to monitor open connections

Environment

• npm: 10.2.4
• Node.js: v20.11.1
• OS Name: MacOS Sonoma 14.2.1 (23C71)
• System Model Name: Macbook Pro
• npm config:

; node bin location = /usr/local/bin/node
; node version = v20.11.1
; npm local prefix = /home
; npm version = 10.2.4
; cwd = /home
; HOME = /root
; Run `npm config ls -l` to show all defaults.

[ankurparihar]

Update:

• Node 16.20.2, NPM: 8.19.4 ~25 max concurrent connections
• NODE: 18.19.1, NPM: 10.2.4 also makes ~2100 connections

It seems this was increased from Node 18

[ankurparihar]

It seems maxsockets is also not being respected, setting this to a lower value like 10 or 1 didn't have any effect.

[mhdawson]

I'm not sure if this is related, but we've seen some teams running into problems moving between versions of Node.js with npm 9 versus 10 and running out of file handles which sounds like it could be related.

[lukekarrys]

Update:

• Node 16.20.2, NPM: 8.19.4 ~25 max concurrent connections

• NODE: 18.19.1, NPM: 10.2.4 also makes ~2100 connections

It seems this was increased from Node 18

Thanks for the report, I will take a look at this.

[ChristophLindemann]

@lukekarrys check out #7072 which has been open since December. It might have some more insights.

[ankurparihar]

Thank you for the prompt response, let me know if I can do some help, even in testing and all.

[melroy89]

Update:

• Node 16.20.2, NPM: 8.19.4 ~25 max concurrent connections
• NODE: 18.19.1, NPM: 10.2.4 also makes ~2100 connections

It seems this was increased from Node 18

Thanks for the report, I will take a look at this.

That would be much appreciated, I'm using Node LTS (v20.11.1), which is using NPM v10.2.4. Where this problem is still present, and causing a lot of issues during CI/CD pipelines.

[silverwind]

Imho, it's definitely related to maxsockets having not effect in npm 10.

[dominicfraser]

@neelance commented something interesting in #7072

neelance
So... I went into the rabbit hole... It went very very deep... But I finally found the root cause!

npm-reqistry-fetch uses make-fetch-happen, see here.
make-fetch-happen uses minipass-fetch, see here.
minipass-fetch uses Node's builtin http/https, see here.
The maxSockets parameter gets passed through all these layers just fine, no issue here.
However, the implementation of http.Agent in Node.js is suboptimal: It checks for maxSockets here, but does not immediately increase the checked value. Instead it calls this.createSocket and assumes the function pushes to this.sockets synchronously. Typical risk for a race condition.
Now back up the stack again: make-fetch-happen uses @npmcli/agent, see here
@npmcli/agent uses agent-base, see here
agent-base subclasses http.Agent and overwrites createSocket with an implementation that is asynchronous, see here.
🫠

[neelance]

I added a workaround to #7072.

[silverwind]

Can we close this duplicate issue in favor of #7072?

[ljharb]

Duplicate of #7072