Actions fail intermittently due to NPM ECONNRESET #3737

rijkvanzanten · 2021-07-15T19:39:44Z

Description

Every now and again, our actions randomly fail due to an ECONNRESET error coming from NPM, see for example https://github.com/directus/directus/runs/3080202193?check_suite_focus=true, https://github.com/directus/directus/pull/5754/checks?check_run_id=2617592226, https://github.com/directus/directus/runs/2611005993 or https://github.com/directus/directus/runs/2602577476.

I reached out to GitHub Support about this, who referenced me to this repo.

Virtual environments affected

Image version and build link

n/a. Getting a Error: We are currently unable to download the log. Please try again later. message now..

Is it regression?

No response

Expected behavior

The connection to NPM shouldn't fail

Actual behavior

The connection to NPM fails

Repro steps

Setup a workflow that runs npm ci
See it fail intermittently

The text was updated successfully, but these errors were encountered:

miketimofeev · 2021-07-16T05:38:56Z

Hi @rijkvanzanten!
It shouldn't be related to the images themselves, we will take a look. Where have you seen connection failures since there is an error with logs in provided runs?

miketimofeev · 2021-07-16T09:53:48Z

Well, found the link with connreset
https://github.com/directus/directus/runs/3080883194?check_suite_focus=true

cschleiden · 2021-07-16T09:53:51Z

A more recent run which seems to hit the same error: https://github.com/directus/directus/runs/3080883194?check_suite_focus=true

miketimofeev · 2021-07-16T11:21:59Z

@rijkvanzanten old logs are not available as the retention policy has been configured to 5 days.
How often do you see such behavior? Could you please provide minimal repro steps? We assume that something is wrong on the NPM side from time to time.

rijkvanzanten · 2021-07-16T13:36:38Z

Hi y'all! Thanks for the swift replies!

It shouldn't be related to the images themselves, we will take a look

I would agree, it feels like a weird intermittent connection issue between Actions and NPM. Not sure on what side of the equation it goes wrong though..

We assume that something is wrong on the NPM side from time to time.

I could see this, however, I run npm install (and/or CI, doesn't seem to matter) locally dozens of times anywhere between 1 and 50 times a day, and have never seen this specific ECONNRESET error 🤔 I can't recall (personally) seeing the same exact error outside of GitHub Actions. (Though that's obviously very anecdotal evidence)

old logs are not available as the retention policy has been configured to 5 days.

Gotcha, the error message threw me off there. Should I open a separate ticket here/elsewhere to pick that up?

Could you please provide minimal repro steps?

It seems to happen on a regular npm install, so the minimal repro should be as simple as

name: Debug NPM connection
jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: '16'
      - name: Install dependencies
        run: |
          npm install

How often do you see such behavior?

Hard to tell with exact accuracy due to the logs retention, though I would estimate between 1-5% of the runs.

LeonidLapshin · 2021-07-21T10:23:01Z

Hey, @rijkvanzanten !
I have forked your repo and tried to reproduce the problem but have no luck with it :( Seems that it's really rare and no action needed. I'll close this issue but please feel free to reopen it if you have any concerns.
Thanks!

rijkvanzanten · 2021-07-21T14:07:56Z

Makes sense, I understand this is too much of an edge case to be able to efficiently research and resolve. As for any concerns, the only one I have is that I'd rather not have our CI fail randomly of course 😅

Is there anything I can do from our side to debug this issue further when it happens?

miketimofeev · 2021-07-21T14:19:58Z

@rijkvanzanten please let us know as soon as you get it and provide the links to the failed runs. We will check the underlying infrastructure for the particular run.
Thank you!

rijkvanzanten · 2021-07-21T14:22:06Z

Will do!

AaronFriel · 2022-10-12T00:33:38Z

Hey folks, bumping an old thread here, we're seeing a lot of ECONNRESET issues, primarily in one action: actions/setup-dotnet#333

No recent change in the action, is this an issue with the virtual environment?

kmahyyg · 2024-01-27T10:19:26Z

Multiple runs failed from a long time ago, especially in QEMU-based ARM64 docker image building, but amd64 builder is working:

ARM64 using QEMU:
https://github.com/kmahyyg/ztncui-aio/actions/runs/7677880679/job/20927134240
https://github.com/kmahyyg/ztncui-aio/actions/runs/7677807879/job/20926977995
https://github.com/kmahyyg/ztncui-aio/actions/runs/7504706695/job/20449416696

Successful runs using AMD64 directly (same commit):
https://github.com/kmahyyg/ztncui-aio/actions/runs/7677880679/job/20927134295
https://github.com/kmahyyg/ztncui-aio/actions/runs/7677831444/job/20927028783
https://github.com/kmahyyg/ztncui-aio/actions/runs/7504706695/job/20449416799

@miketimofeev

kmahyyg · 2024-01-27T10:20:21Z

This can be reproduced stably using GH-Hosted runner on my side.

gramnarine · 2024-02-01T15:04:12Z

I have been seeing this intermittently in my org as well.

Self hosted runners, running version 2.312.0 on Ubuntu 22.04.03

austinpray-mixpanel · 2024-02-01T17:18:28Z

Seeing this pretty often using stock actions/setup-node@v4

npm ERR! code ECONNRESET
npm ERR! network aborted
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

under

Current runner version: '2.312.0'
Operating System
  Ubuntu
  20.04.6
  LTS
Runner Image
  Image: ubuntu-20.04
  Version: 20240126.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu20/20240126.1/images/ubuntu/Ubuntu2004-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu20%2F20240126.1

LeaBroudoEcho3d · 2024-02-20T16:37:47Z

Also seeing this pretty often now, I'm getting the following log:

npm ERR! code ECONNRESET
npm ERR! network aborted
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

For the following action (the error comes specifically from the npm ci command):

name: XX
on: [push]
jobs:
  run-defined-frontend-tests:
    runs-on: ubuntu-latest
    name: Webapp Unit Tests
    defaults:
      run:
        working-directory: ./webapp_new
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '18'
      - run: npm ci --legacy-peer-deps
      - run: npm run test:ci
        env:
          CI: true

undergroundwires · 2024-03-21T22:12:05Z

I get this on macOS runners:

Current runner version: '2.314.1'
Operating System
  macOS
  12.7.3
  21H1015
Runner Image
  Image: macos-12
  Version: 20240218.1
  Included Software: https://github.com/actions/runner-images/blob/macOS-12/20240218.1/images/macos/macos-12-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20240218.1
Runner Image Provisioner
  2.0.341.1+ea3907f30f87ddf37e44fa45b753abddb03811b4

bcheidemann · 2024-03-27T13:34:44Z

Getting the below error intermittently but regularly.

npm ERR! code ECONNRESET
npm ERR! network aborted
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

EDIT: using runner ubuntu-latest.

dadebue · 2024-03-28T12:54:44Z

I'm getting a similar (and reproducible) error for an QEMU-based arm64 build on ubuntu-latest GitHub hosted runners in the docker/build-push-action@v5 action npm ci command:

#19 [build-frontend 5/7] RUN npm ci
#19 519.0 npm ERR! code ECONNRESET
#19 519.1 npm ERR! errno ECONNRESET
#19 519.1 npm ERR! network request to https://registry.npmjs.org/wgs84/-/wgs84-0.0.0.tgz failed, reason: socket hang up
#19 519.1 npm ERR! network This is a problem related to network connectivity.
#19 519.1 npm ERR! network In most cases you are behind a proxy or have bad network settings.
#19 519.1 npm ERR! network 
#19 519.1 npm ERR! network If you are behind a proxy, please make sure that the
#19 519.1 npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

Dockerfile

FROM node:lts as build-frontend

WORKDIR /build

COPY package.json package-lock.json ./
RUN mkdir node_modules
RUN npm ci

COPY . .

RUN npm run build

CICD yaml build job:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@v3
        with:
          install: true
          platforms: linux/arm64

      - name: log in to GitHub container registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: build and push container image
        uses: docker/build-push-action@v5
        with:
          context: .
          platforms: linux/arm64
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          build-args: |
            VITE_GIT_COMMIT=${{ github.sha }}

Any way to fix it @miketimofeev?

dadebue · 2024-04-02T22:03:47Z

FYI: Upgrading to Node 20 in the Dockerfile fixed the problem for me...

undergroundwires · 2024-04-04T11:43:30Z

I solved this by upgrading to Node 20 and enforcing IPv4, see force-ipv4 action.

austinpray-mixpanel · 2024-04-04T15:16:19Z

It's worth noting that npm just closed a bug (pending release) where too many connections were being opened during install:

buanet · 2024-04-05T06:04:53Z

I got the same error, exactly as @dadebue describes.
But as I have to stay at Node 18 at the moment, I needed another solution than upgrading to Node 20 in my Dockerfile.
So I just update NPM to the new 10.5.1 release and the GHA build is working again.
Thank you all for commenting this old, closed issue, helping me to fix that! 😄

Regards,
André

dadebue · 2024-04-09T04:39:55Z

The problem is occurring again for me now. Even with Node 20 🤔

zburgermeiszter · 2024-04-23T17:30:07Z

I also hit this issue often.
As a temporary solution I leveraged the nick-fields/retry action.

      - name: Install npm dependencies
        uses: nick-fields/retry@v3
        with:
          timeout_minutes: 10
          max_attempts: 3
          command: npm ci

rijkvanzanten added the needs triage label Jul 15, 2021

miketimofeev added Area: JavaScript and Node.js investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Ubuntu and removed needs triage labels Jul 16, 2021

AlenaSviridenko assigned LeonidLapshin Jul 19, 2021

LeonidLapshin closed this as completed Jul 21, 2021

bap14 mentioned this issue Apr 3, 2024

GitHub Workflow Running out of disk space wardenenv/images#24

Closed

rbren mentioned this issue Apr 17, 2024

Docker build is flaky for the app container All-Hands-AI/OpenHands#1192

Closed

rbren mentioned this issue Apr 23, 2024

[WIP] fix the docker build All-Hands-AI/OpenHands#1320

Closed

This was referenced Apr 24, 2024

chore(common): add retry checks around npm install (and any other npm network activities) keymanapp/keyman#10350

Closed

chore(web): CI builds are currently extremely unreliable keymanapp/keyman#10494

Closed

Geod24 mentioned this issue Sep 18, 2024

IPv6 support dlang-community/setup-dlang#84

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Actions fail intermittently due to NPM ECONNRESET #3737

Actions fail intermittently due to NPM ECONNRESET #3737

rijkvanzanten commented Jul 15, 2021

miketimofeev commented Jul 16, 2021

miketimofeev commented Jul 16, 2021

cschleiden commented Jul 16, 2021

miketimofeev commented Jul 16, 2021

rijkvanzanten commented Jul 16, 2021

LeonidLapshin commented Jul 21, 2021

rijkvanzanten commented Jul 21, 2021

miketimofeev commented Jul 21, 2021

rijkvanzanten commented Jul 21, 2021

AaronFriel commented Oct 12, 2022

kmahyyg commented Jan 27, 2024

kmahyyg commented Jan 27, 2024

gramnarine commented Feb 1, 2024 •

edited

Loading

austinpray-mixpanel commented Feb 1, 2024

LeaBroudoEcho3d commented Feb 20, 2024

undergroundwires commented Mar 21, 2024

bcheidemann commented Mar 27, 2024 •

edited

Loading

dadebue commented Mar 28, 2024 •

edited

Loading

dadebue commented Apr 2, 2024

undergroundwires commented Apr 4, 2024

austinpray-mixpanel commented Apr 4, 2024

buanet commented Apr 5, 2024

dadebue commented Apr 9, 2024

zburgermeiszter commented Apr 23, 2024 •

edited

Loading

Actions fail intermittently due to NPM ECONNRESET #3737

Actions fail intermittently due to NPM ECONNRESET #3737

Comments

rijkvanzanten commented Jul 15, 2021

Description

Virtual environments affected

Image version and build link

Is it regression?

Expected behavior

Actual behavior

Repro steps

miketimofeev commented Jul 16, 2021

miketimofeev commented Jul 16, 2021

cschleiden commented Jul 16, 2021

miketimofeev commented Jul 16, 2021

rijkvanzanten commented Jul 16, 2021

LeonidLapshin commented Jul 21, 2021

rijkvanzanten commented Jul 21, 2021

miketimofeev commented Jul 21, 2021

rijkvanzanten commented Jul 21, 2021

AaronFriel commented Oct 12, 2022

kmahyyg commented Jan 27, 2024

kmahyyg commented Jan 27, 2024

gramnarine commented Feb 1, 2024 • edited Loading

austinpray-mixpanel commented Feb 1, 2024

LeaBroudoEcho3d commented Feb 20, 2024

undergroundwires commented Mar 21, 2024

bcheidemann commented Mar 27, 2024 • edited Loading

dadebue commented Mar 28, 2024 • edited Loading

dadebue commented Apr 2, 2024

undergroundwires commented Apr 4, 2024

austinpray-mixpanel commented Apr 4, 2024

buanet commented Apr 5, 2024

dadebue commented Apr 9, 2024

zburgermeiszter commented Apr 23, 2024 • edited Loading

gramnarine commented Feb 1, 2024 •

edited

Loading

bcheidemann commented Mar 27, 2024 •

edited

Loading

dadebue commented Mar 28, 2024 •

edited

Loading

zburgermeiszter commented Apr 23, 2024 •

edited

Loading