Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Platform-specific optional dependencies not being included in package-lock.json when reinstalling with node_modules present #4828

Open
2 tasks done
JustinChristensen opened this issue Apr 29, 2022 · 195 comments
Labels
Bug thing that needs fixing Needs Triage needs review for next steps Release 8.x work is associated with a specific npm 8 release

Comments

@JustinChristensen
Copy link

JustinChristensen commented Apr 29, 2022

Is there an existing issue for this?

  • I have searched the existing issues

This issue exists in the latest npm version

  • I am using the latest npm

Current Behavior

[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'

I'm working on a team that utilizes a mix of x64-based and m1-based macs, and has CI build processes that uses musl. We're seeing that npm is skipping platform-specific optional dependencies for packages such as @swc/core as a result of the package-lock.json file being generated without all of them included. In our case, this then causes linting to throw an exception, because one of our eslint plugins depends on @swc, which depends on having the platform specific @swc package also installed.

There seems to be at least two stages of cause to this. Firstly, when installing @swc/core from a clean slate working directory npm generates a package-lock.json with all of the optional dependencies for @swc/core listed:

[user@host:foo] $ npm install @swc/core
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-android-arm-eabi": {
    "node_modules/@swc/core-android-arm64": {
    "node_modules/@swc/core-darwin-arm64": {
    "node_modules/@swc/core-darwin-x64": {
    "node_modules/@swc/core-freebsd-x64": {
    "node_modules/@swc/core-linux-arm-gnueabihf": {
    "node_modules/@swc/core-linux-arm64-gnu": {
    "node_modules/@swc/core-linux-arm64-musl": {
    "node_modules/@swc/core-linux-x64-gnu": {
    "node_modules/@swc/core-linux-x64-musl": {
    "node_modules/@swc/core-win32-arm64-msvc": {
    "node_modules/@swc/core-win32-ia32-msvc": {
    "node_modules/@swc/core-win32-x64-msvc": {

And it only installs the platform specific package:

[user@host:foo] $ ls -l node_modules/@swc/
total 0
drwxr-xr-x  22 user  staff  704 Apr 29 15:39 core
drwxr-xr-x   6 user  staff  192 Apr 29 15:39 core-darwin-arm64

If I then remove my package-lock.json, leave my node_modules directory as-is, and then reinstall, I get:

[user@host:foo] $ rm -rf package-lock.json
[user@host:foo] $ npm install
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-darwin-arm64": {

That is, it then generates a package-lock.json with only the platform-specific dependency that was installed on this machine, and not with the other optional dependencies that should also be listed.

If you delete both node_modules AND package-lock.json, and then re-run npm install, it generates the correct lockfile with all of those optional dependencies listed.

The problem is that then, If the package-lock.json with the missing optional platform-specific dependencies gets checked into git and an x64 user pulls it down, or vice-versa, npm fails to detect that your platform's optional dependencies are missing in the lockfile and just silently skips installing the platform-specific dependency. For example, when I've got a package-lock.json that only contains the x64 @swc package because of the above problem (generated by my coworker on his x64 machine):

[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'
>
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-darwin-x64": {
[user@host:foo] $ ls
package-lock.json package.json

And I then install:

[user@host:foo] $ npm install
added 1 package in 341ms

1 package is looking for funding
  run `npm fund` for details
[user@host:foo] $ ls node_modules/@swc/
core

You can see that it fails to install the arm64 dependency or warn me in any way that the package-lock.json is missing my platform's dependency.

So yeah, two problems:

  1. npm is generating an inconsistent package-lock.json when node_modules has your platform-specific dependency installed.
  2. When installing from this inconsistent package-lock.json, npm fails to try to correct the problem by comparing the optional dependencies to what's listed upstream

Expected Behavior

  1. npm should preserve the full set of platform-specific optional deps for a package like @swc when rebuilding package-lock.json from an existing node_modules tree
  2. npm install should warn if the package-lock.json becomes inconsistent because of the first case

Steps To Reproduce

See above.

Environment

  • npm: 8.8.0
  • Node.js:
  • OS Name: OSX
  • System Model Name: Macbook Pro
[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node -v
v16.14.2
[user@host:foo] $ uname -a
Darwin host.foo.com. 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T8101 arm64
[user@host] $ npm config ls
; "user" config from /Users/user/.npmrc
; node bin location = /Users/user/.nvm/versions/node/v16.14.2/bin/node
; node version = v16.14.2
; npm local prefix = /Users/user/Development/foo
; npm version = 8.8.0
; cwd = /Users/user/Development/foo
; HOME = /Users/user
; Run `npm config ls -l` to show all defaults.
@JustinChristensen JustinChristensen added Bug thing that needs fixing Needs Triage needs review for next steps Release 8.x work is associated with a specific npm 8 release labels Apr 29, 2022
@JustinChristensen JustinChristensen changed the title [BUG] Platform-specific optional dependencies not being included in package-lock when reinstalling with node_modules/ present [BUG] Platform-specific optional dependencies not being included in package-lock when reinstalling with node_modules present Apr 29, 2022
@JustinChristensen JustinChristensen changed the title [BUG] Platform-specific optional dependencies not being included in package-lock when reinstalling with node_modules present [BUG] Platform-specific optional dependencies not being included in package-lock.json when reinstalling with node_modules present Apr 29, 2022
@JustinChristensen
Copy link
Author

@nlf

Sorry to ping you out of the blue, but this issue has been open for 11 days now without any movement. Is there anyone working on npm right now that might have the bandwidth to at least validate that this is indeed a problem as I've described it?

Just so that when someone does become available to do some development work they know that this is in the queue?

Please and thank you.

@JustinChristensen
Copy link
Author

Bump

@RobbieClarken
Copy link

I'm also encountering this issue with a Next.js project:

  • Deleting package-lock.json and running npm install on an M1 Mac results in a package-lock.json file that is no longer able to build the app on x86.
  • This can be fixed by deleting package-lock.json and node_modules and re-running npm install.

Unfortunately developers often don't realise the package-lock.json file is broken because everything continues to run fine on their machine. It is only when the build runs in CI that we learn it is broken.

Here is a reproduction:

$ node --version
v16.13.0
$ npm --version
8.12.1
$ npx create-next-app@latest
What is your project named? … my-app
Creating a new Next.js app in /Users/robbie/demo/my-app.
$ cd my-app/
$ npm install

up to date, audited 223 packages in 480ms

68 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$ git status
On branch main
nothing to commit, working tree clean
$ rm package-lock.json
$ npm install

up to date, audited 223 packages in 579ms

68 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$ # ************ package-lock.json is now incompatible with x86 ************
$ git diff
diff --git a/package-lock.json b/package-lock.json
index cbbf946..a87c1e5 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -96,36 +96,6 @@
         "glob": "7.1.7"
       }
     },
-    "node_modules/@next/swc-android-arm-eabi": {
-      "version": "12.1.6",
-      "resolved": "https://registry.npmjs.org/@next/swc-android-arm-eabi/-/swc-android-arm-eabi-12.1.6.tgz",
-      "integrity": "sha512-BxBr3QAAAXWgk/K7EedvzxJr2dE014mghBSA9iOEAv0bMgF+MRq4PoASjuHi15M2zfowpcRG8XQhMFtxftCleQ==",
-      "cpu": [
-        "arm"
-      ],
-      "optional": true,
-      "os": [
-        "android"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@next/swc-android-arm64": {
-      "version": "12.1.6",
-      "resolved": "https://registry.npmjs.org/@next/swc-android-arm64/-/swc-android-arm64-12.1.6.tgz",
-      "integrity": "sha512-EboEk3ROYY7U6WA2RrMt/cXXMokUTXXfnxe2+CU+DOahvbrO8QSWhlBl9I9ZbFzJx28AGB9Yo3oQHCvph/4Lew==",
-      "cpu": [
-        "arm64"
-      ],
-      "optional": true,
-      "os": [
-        "android"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
[...]
$ rm -r package-lock.json node_modules
$ npm install

added 222 packages, and audited 223 packages in 2s

68 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$ # ************ package-lock.json is now ok again ************
$ git status
On branch main
nothing to commit, working tree clean

@pete55104
Copy link

I am also having this issue. I'm trying to run tests using jest with swc. The test runner is a linux image, but my dev machine is darwin. I can get it to work by either using --force to install the linux dependency, or I can install packages from inside the container... but github CI stands up the docker container in such a way that I can't easily install packages from in there, and that also prevents me from maintaining a cached node modules etc.

@johnculviner
Copy link

bump

@nikkhn
Copy link

nikkhn commented Jul 12, 2022

bump - cannot get optional dependencies (namely @swc/core-linux-arm64-gnu) to install on my linux distro

@sgoodluck
Copy link

bump

@alcuadrado
Copy link

Confirming that this issue is still present. It's particularly important for projects using NAPI modules, as tons of them use platform-specific packages.

jfsoul added a commit to guardian/prosemirror-typerighter that referenced this issue Sep 26, 2022
@AboldUSER
Copy link

Ran into this issue when creating a CI process for a repo where I use a Windows machine and the CI process is using Linux. My quick "fix" for now is to start the CI process by deleting the package-lock.json and running npm install instead of npm ci. I know this is not good practice, so looking forward to a real fix to come through.

@eliotSmithNYC
Copy link

bump

@douglassllc
Copy link

I am having a similar issue. My project uses @ffmpeg-installer/ffmpeg. While using npm v6 all optional dependencies (arch specific) are installed. After my upgrade to npm v8 the optional dependencies no longer install. Per the npm documentation I attempted using --include=optional, but this did not resolve the issue.

What has changed between v6 and v8 and is there an npm config option that will have v8 work similar to v6 when it comes to optional dependencies?

richford added a commit to yeatmanlab/roar-dashboard that referenced this issue Oct 15, 2024
* Introduce useOrgUsersQuery composable

* Add unit tests for useOrgUsersQuery

* Fix linter offenses

* Introduce useTasksQuery composable

* Extend withSetup test util to accept app options

* Add unit tests

* Introduce useAddTaskMutation

* Improve task addition error handling

* Improve view model handling

* Introduce useUpdateTaskMutation and add mutation keys

* Add mutation unit tests

* Introduce useAddTaskVariantMutation and useUpdateTaskVariantMutation composables

* Fix useTasksQuery and implement on Manage Variants page

* Renamed task variants query

* Fix ref val usage in useTaskVariantsQuery

* Add unit tests for useTaskVariantsQuery

* Introduce useTasksQuery and optimise conditional task loading

* Extend useUserType for super admin handling

* Apply super admin user type check changes to HomeSelector

* Introduce useAdministrationsQuery composable

* Remove obsolete user claims query

* Fix administrations loading spinner

* Introduce useUpdateUserMutation

* Cleanup and remove obsolete logging

* Add unit tests for useUpdateUserMutation

* Add useUserDataQuery unit tests

* Leverage useUserType for isSuperAdmin status

* Add missing import

* Regenerate package-lock due to npm/cli#4828

* Prepare for composable queries

* Replace user claims query with useUserClaimsQuery

* Use userType composable

* Introduce useAdministrationsQuery composable and apply on administrator homepage

* Fix loading state

* Fix linter offenses

* Restore itemsPerPage to original value

* Restore package files

* Upgrade tanstack and install dev tools

* Remove comment

* Remove incorrect parameters

* Introduce useUserAssignments composable

* Fix reactivity and minor issues

* Introduce revised useAdministrationsQuery composable

* Extend useTasksQuery to fetch by task IDs

* Move useTasksQuery function to helper

* Fix tasks query fetching and constants

* Update useTasksQuery unit tests

* Fix loading & query reactivity

* Introduce useSurveyReponsesQuery composable

* Add useSurveyResponsesQuery unit tests

* Add useUserAssignmentsQuery unit tests

* Fix reactivity in query composables

* Remove duplications in favour of computeQueryOverrides helper

* Fix vue warning

* Improve computeQueryOverride reactivity and testability

* Add unit tests for useUserStudentDataQuery

* Overhaul useUserDataQuery unit tests

* Add useUserClaimsQuery unit tests

* Overhaul useUserClaimsQuery unit tests

* Simplify useTaskVariantsQuery unit tests

* Align useTasksQuery unit test names

* Overhaul useSurveyResponsesQuery unit tests

* Add useAdministrationsQuery unit tests

* Add useDsgfOrgQuery unit tests

* Add useLegalDocsQuery unit tests

* Add computeQueryOverrides unit tests

* Switch from uid to roarUid for survey response querying

* Migrate score report queries to existing query composables

* Introduce useAdministrationAssignmentsQuery composable

* Add useDistrictsQuery unit tests

* Split district query into separate composables

* Introduce useSchoolQuery composable

* Introduce useClassQuery composable

* Introduce useGroupQuery composable

* Fix doc comments

* Introduce useFamilyQuery composable

* Fix useAdministrationsQuery unit tests

* Introduce new fetchDocumentsById query helper

* Replace useClassQuery with useClassesQuery composable

* Restructure district queries

* Replace useSchoolQuery with useSchoolsQuery composable

* Replace useGroupQuery with useGroupsQuery composable

* Replace useFamilyQuery with useFamiliesQuery

* Extend useDistrictsListQuery tests

* Replace manual administration query with useAdministrationsQuery

* Introduce new useAdministrationsStatsQuery composable

* Migrate ProgressReport to individual org queries

* Introduce reusable useOrgQuery composable for dynamic org querying

* Migrate remaining page queries to composables

* Switch to router based report routing

* Add unit tests

* Renamed page to match naming

* Adapt useUserDataQuery to optionally fetch by manual user ID

* Introduce useUserAdministrationAssignmentsQuery composable

* Replace manual query with useAdministrationsQuery

* Introduce tempoary useUsrRunPageQuery composable

* Add useUserAdministrationAssignmentsQuery unit tests

* Fix linter offenses

* Fix comment typos

* Make TanStack dev tools load conditionally

* Fix administration assignments query

* Remove staging conditional in favour of env var

* Temporarily disable progress and report filtering tests

* Replace administrations query with useAdministrationsQuery composable

* Replace districts and schools queries with composables

* Align query omposables and fix reactivity issues

* Replace classes query with useClassesQuery composable

* Fix linter offenses

* Fix query state reactivity

* Replace groups query with useGroupsQuery composable

* Replace families query with useFamiliesQuery composable

* Refactor CreateAdminitration page data handling

* Fix query state handling

* Remove console logs and obsolete comments

* Refactor consent handling for administrators

* Apply Consent Modal changes to remaining pages

* Fix query reactivity

* Fix document path destructuring

* Add markdown/html sanitization

* Add remaining consent types

* Add loop breaker

* Remove obsolete import

* Add useSignOutMutation

* Fix and improve signout mutation

* Apply signOut mutation across pages

* Remove query key indexes from queries

* Combine ClassLink and Clever into single SSO landing page

* Refactor store sso source

* Add (intentional) typos

* Fix home selector routing for sso flow

* Introduce useTasksDictionaryQuery composable

* Replace authStore tasks dictionary in favour of query composable

* Remove obsolete import

* Remove onUpdated hook for task dictionary refresh

* Use AUTH_SSO_PROVIDERS instead of string literals

* Clear query data before sign-out redirect

* Fix vue-query hook usage error

* Improve error handling and logging

* Add inline initStateFromRedirect comments

* Add unit tests

* Introduce useDeleteAdministrationMutation mutation

* Improve administration loader

* Fix error handling in fetchByDocId

* Update JSDoc

* Update src/components/auth/RegisterStudent.vue

Co-authored-by: Kyle Montville <[email protected]>

* Update src/components/auth/RegisterStudent.vue

Co-authored-by: Kyle Montville <[email protected]>

* Improve error handling

* Updated copy to match form option

* Revert "Improve error handling"

This reverts commit 4fa5c46.

* Fix post-signout firekit initialisation

* Fix linter offenses

* Fix SWR task loading

* Fix Vocab task loading

* Fix SRE task loading

* Fix Ran task loading

* Fix PA task loading

* Fix Multichoice task loading

* Fix MEP task loading

* Fix Levante task loading

* Fix Letter task loading

* Fix Fluency task loading

* Fix Crowding task loading

* Fix tasks loading (#883)

* Fix SWR task loading

* Fix Vocab task loading

* Fix SRE task loading

* Fix Ran task loading

* Fix PA task loading

* Fix Multichoice task loading

* Fix MEP task loading

* Fix Levante task loading

* Fix Letter task loading

* Fix Fluency task loading

* Fix Crowding task loading

* Fix administration selection

* Fix administrations list rendering (#888)

* Fix participant homepage loading state (#889)

* Fix flashing loading states on participant homepage

* Fix refs

---------

Co-authored-by: Emily Arteaga <[email protected]>
Co-authored-by: Kyle Montville <[email protected]>
Co-authored-by: Adam Richie-Halford <[email protected]>
@anand-TV
Copy link

bump

@schjetne
Copy link

At this point I don't think bumping this thread actually does anything. NPM clearly has an issue with differing chip architectures and they're not doing anything about it.

So we'll have to keep doing our workarounds…

@mhintzke
Copy link

mhintzke commented Oct 16, 2024

The "easiest" (but maybe not cleanest) workaround for me has been to delete the node_modules folder and the package-lock.json file and then perform a npm install --package-lock-only which will generate the package-lock.json without following any of the install "rules" that come with multi-architecture supported packages. This will properly generate the package-lock with all optional dependencies for all of the various architectures.

Yes, I understand it sucks to have to delete your package-lock which means you likely need to do exhaustive testing to ensure no runtime errors exist, but it does work.

There is a second, controversially dangerous option which is to copy just the JSON needed to add the new packages for the package at-hand, then restore your old package-lock.json and paste in the new packages to surgically fix the file without completely altering your other packages.

@tavareshenrique
Copy link

tavareshenrique commented Oct 17, 2024

I've tried everything here to solve this problem, but without success.

I'm trying to run it using docker, and I always get the error bellow.

I have no idea what to do anymore.

I tried running it with NPM, PNPM and nothing works.

app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:59
project-1  |            throw new Error(
project-1  |                  ^
project-1  | 
project-1  | Error: Cannot find module @rollup/rollup-linux-arm64-gnu. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.
project-1  |     at requireWithFriendlyError (/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:59:9)
project-1  |     at Object.<anonymous> (/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:68:76)
project-1  |     ... 3 lines matching cause stack trace ...
project-1  |     at Module._load (node:internal/modules/cjs/loader:1104:12)
project-1  |     at cjsLoader (node:internal/modules/esm/translators:346:17)
project-1  |     at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:286:7)
project-1  |     at ModuleJob.run (node:internal/modules/esm/module_job:234:25)
project-1  |     at async ModuleLoader.import (node:internal/modules/esm/loader:473:24) {
project-1  |   [cause]: Error: Cannot find module '@rollup/rollup-linux-arm64-gnu'
project-1  |   Require stack:
project-1  |   - /app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js
project-1  |       at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
project-1  |       at Module._load (node:internal/modules/cjs/loader:1051:27)
project-1  |       at Module.require (node:internal/modules/cjs/loader:1311:19)
project-1  |       at require (node:internal/modules/helpers:179:18)
project-1  |       at requireWithFriendlyError (/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:41:10)
project-1  |       at Object.<anonymous> (/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:68:76)
project-1  |       at Module._compile (node:internal/modules/cjs/loader:1469:14)
project-1  |       at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
project-1  |       at Module.load (node:internal/modules/cjs/loader:1288:32)
project-1  |       at Module._load (node:internal/modules/cjs/loader:1104:12) {
project-1  |     code: 'MODULE_NOT_FOUND',
project-1  |     requireStack: [
project-1  |       '/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js'
project-1  |     ]
project-1  |   }
project-1  | }
project-1  | 
project-1  | Node.js v20.18.0
project-1  |  ELIFECYCLE  Command failed with exit code 1.
server-1   | --watch/-w can be omitted, JSON Server 1+ watches for file changes by default

This is my Dockerfile:

FROM node:20.18.0-slim

WORKDIR /app

COPY package.json pnpm-lock.yaml ./

RUN npm install -g pnpm

RUN pnpm install

COPY . .

EXPOSE 3000
EXPOSE 3001

CMD ["sh", "-c", "if [ \"$SERVICE\" = 'project' ]; then pnpm dev; else pnpm init:db; fi"]

@dmail
Copy link

dmail commented Oct 17, 2024

To be un blocked you can do this:

After npm install (or npm ci) you run the command to install the platform specific dependency.

muodov added a commit to duckduckgo/autoconsent that referenced this issue Oct 17, 2024
@tavareshenrique
Copy link

Thanks for your answer @dmail, I've already tried, but it still persists. I changed my dockerfile to adapt to this scenario.

FROM node:20-buster

WORKDIR /app

RUN apt-get update && apt-get install -y \
  build-essential \
  python3 \
  && rm -rf /var/lib/apt/lists/*

COPY package.json package-lock.json ./

RUN npm ci

RUN npm install @rollup/rollup-linux-arm64-gnu

COPY . .

EXPOSE 3000
EXPOSE 3001

CMD ["sh", "-c", "if [ \"$SERVICE\" = 'front' ]; then npm run dev; else npm run init:db; fi"]

@tavareshenrique
Copy link

I don't know if it helps, but I had to install the two dependencies below to solve my scenario:

"@esbuild/linux-arm64": "0.20.2",
"@rollup/rollup-linux-arm64-gnu": "4.24.0",

@ken717w
Copy link

ken717w commented Oct 18, 2024

My case was running bun run build failed on a Apple Silicon Mac, and none of the workarounds above work for me.

At the end of the day I changed my default Node version which installed via NPM, from 16 to 20, and then all errors are gone!

trentm added a commit to trentm/opentelemetry-js-contrib that referenced this issue Oct 19, 2024
The browser tests failed in CI with this error message:

    Error: Cannot find module @rollup/rollup-linux-x64-gnu. npm has a bug related to optional dependencies (npm/cli#4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.
carloscasalar added a commit to casrpg/traveller-rpg-tools that referenced this issue Oct 19, 2024
@majdarov
Copy link

I run Docker development containers based on the node:20 image. An error occurred in the backend container related to the @sws/core and @swc/cli packages. There is also an error in the frontend container: "Error: I can't find the @rollup/rollup-linux-x64-musl " module.

After several days of searching and temporary solutions, I realized that the error occurs when using the node:20 image based on alpine. If you use the node:20-bookworm or node:20-bullseye images, there will be no error. Maybe this will be useful to someone.

vitebo added a commit to quero-edu/quero-challenge-frontend-vue that referenced this issue Oct 23, 2024
npm has a bug related to optional dependencies (npm/cli#4828).
vitebo added a commit to quero-edu/quero-challenge-frontend-react that referenced this issue Oct 23, 2024
npm has a bug related to optional dependencies (npm/cli#4828).
scristobal added a commit to scristobal/webgpu-experiments that referenced this issue Oct 26, 2024
@saiichihashimoto
Copy link

For whatever reason, after every merge conflict I have to fix this and I'll eventually get it fixed, only to fail in my CI. I'm having to delete package-lock.json and node_modules locally, npm install, npm run build, git push, only to have it fail again in CI and repeat until it works for every merge.

saiichihashimoto added a commit to saiichihashimoto/sanity-typed that referenced this issue Oct 28, 2024
saiichihashimoto added a commit to saiichihashimoto/sanity-typed that referenced this issue Oct 28, 2024
@lpknv
Copy link

lpknv commented Oct 29, 2024

I am also experiencing a similar issue using node:22-alpine for my sveltekit / vite docker deployment and currently trying out different node docker version (bookworm, slim etc).

@hrishikeshs
Copy link

Running into this issue on my gitlab instance for CI runners. Our project is quite large and caching node_modules when package-lock.json hasn't changed makes a significant difference (time taken reduces from 30 mins to around 10 mins). This issue is preventing us from setting up our pipeline to me more efficient :(

saiichihashimoto added a commit to saiichihashimoto/sanity-typed that referenced this issue Oct 29, 2024
dan-j added a commit to versori/versori-js-sdk that referenced this issue Oct 30, 2024
@Abadii
Copy link

Abadii commented Oct 30, 2024

I encountered a similar issue, and here’s how I resolved it.

I’m working on a MacBook Pro with an M2 Pro chip, while my GitLab pipeline runs on a Linux (x64) environment.

In my case, the problem stemmed from the rollup package not installing certain optional dependencies:

"optionalDependencies": {
        "@rollup/rollup-android-arm-eabi": "4.22.4",
        "@rollup/rollup-android-arm64": "4.22.4",
        "@rollup/rollup-darwin-arm64": "4.22.4",
        "@rollup/rollup-darwin-x64": "4.22.4",
        "@rollup/rollup-linux-arm-gnueabihf": "4.22.4",
        "@rollup/rollup-linux-arm-musleabihf": "4.22.4",
        "@rollup/rollup-linux-arm64-gnu": "4.22.4",
        "@rollup/rollup-linux-arm64-musl": "4.22.4",
        "@rollup/rollup-linux-powerpc64le-gnu": "4.22.4",
        "@rollup/rollup-linux-riscv64-gnu": "4.22.4",
        "@rollup/rollup-linux-s390x-gnu": "4.22.4",
        "@rollup/rollup-linux-x64-gnu": "4.22.4",
        "@rollup/rollup-linux-x64-musl": "4.22.4",
        "@rollup/rollup-win32-arm64-msvc": "4.22.4",
        "@rollup/rollup-win32-ia32-msvc": "4.22.4",
        "@rollup/rollup-win32-x64-msvc": "4.22.4",
        "fsevents": "~2.3.2"
      }

This package list is required by: @angular-devkit/[email protected] -> @angular/[email protected] -> [email protected].

In my package-lock.json, the necessary @rollup/rollup-darwin-arm64 was missing, even though it seemed required. Oddly, it was marked as optional. Running npm install --include=optional fixed this locally, updating my package-lock.json to include the needed packages, allowing ng serve to run smoothly.

To ensure consistency, I deleted the package-lock.json (a potentially risky step) and ran a fresh npm install --include=optional. This worked on my local machine but not in the pipeline, where the required linux-x64 packages were still missing.

To address this, I deleted package-lock.json again and ran npm install --platform=linux --arch=x64, adding include=optional to my .npmrc file. This generated a package-lock.json with all the necessary OS/architecture packages. Following this, npm install succeeded both locally and in the pipeline. Each environment correctly installed only the relevant packages (Darwin for macOS locally and Linux for the pipeline).

If you don’t mind deleting package-lock.json, this method might work for you too. Ensure the required packages are in package-lock.json before rerunning npm install with include=optional.

Best of luck!

@Abadii
Copy link

Abadii commented Oct 30, 2024

I encountered a similar issue, and here’s how I resolved it.

I’m working on a MacBook Pro with an M2 Pro chip, while my GitLab pipeline runs on a Linux (x64) environment.

In my case, the problem stemmed from the rollup package not installing certain optional dependencies:

"optionalDependencies": {
        "@rollup/rollup-android-arm-eabi": "4.22.4",
        "@rollup/rollup-android-arm64": "4.22.4",
        "@rollup/rollup-darwin-arm64": "4.22.4",
        "@rollup/rollup-darwin-x64": "4.22.4",
        "@rollup/rollup-linux-arm-gnueabihf": "4.22.4",
        "@rollup/rollup-linux-arm-musleabihf": "4.22.4",
        "@rollup/rollup-linux-arm64-gnu": "4.22.4",
        "@rollup/rollup-linux-arm64-musl": "4.22.4",
        "@rollup/rollup-linux-powerpc64le-gnu": "4.22.4",
        "@rollup/rollup-linux-riscv64-gnu": "4.22.4",
        "@rollup/rollup-linux-s390x-gnu": "4.22.4",
        "@rollup/rollup-linux-x64-gnu": "4.22.4",
        "@rollup/rollup-linux-x64-musl": "4.22.4",
        "@rollup/rollup-win32-arm64-msvc": "4.22.4",
        "@rollup/rollup-win32-ia32-msvc": "4.22.4",
        "@rollup/rollup-win32-x64-msvc": "4.22.4",
        "fsevents": "~2.3.2"
      }

This package list is required by: @angular-devkit/[email protected] -> @angular/[email protected] -> [email protected].

In my package-lock.json, the necessary @rollup/rollup-darwin-arm64 was missing, even though it seemed required. Oddly, it was marked as optional. Running npm install --include=optional fixed this locally, updating my package-lock.json to include the needed packages, allowing ng serve to run smoothly.

To ensure consistency, I deleted the package-lock.json (a potentially risky step) and ran a fresh npm install --include=optional. This worked on my local machine but not in the pipeline, where the required linux-x64 packages were still missing.

To address this, I deleted package-lock.json again and ran npm install --platform=linux --arch=x64, adding include=optional to my .npmrc file. This generated a package-lock.json with all the necessary OS/architecture packages. Following this, npm install succeeded both locally and in the pipeline. Each environment correctly installed only the relevant packages (Darwin for macOS locally and Linux for the pipeline).

If you don’t mind deleting package-lock.json, this method might work for you too. Ensure the required packages are in package-lock.json before rerunning npm install with include=optional.

Best of luck!

I encountered a similar issue, and here’s how I resolved it.

I’m working on a MacBook Pro with an M2 Pro chip, while my GitLab pipeline runs on a Linux (x64) environment.

In my case, the problem stemmed from the rollup package not installing certain optional dependencies:

"optionalDependencies": {
        "@rollup/rollup-android-arm-eabi": "4.22.4",
        "@rollup/rollup-android-arm64": "4.22.4",
        "@rollup/rollup-darwin-arm64": "4.22.4",
        "@rollup/rollup-darwin-x64": "4.22.4",
        "@rollup/rollup-linux-arm-gnueabihf": "4.22.4",
        "@rollup/rollup-linux-arm-musleabihf": "4.22.4",
        "@rollup/rollup-linux-arm64-gnu": "4.22.4",
        "@rollup/rollup-linux-arm64-musl": "4.22.4",
        "@rollup/rollup-linux-powerpc64le-gnu": "4.22.4",
        "@rollup/rollup-linux-riscv64-gnu": "4.22.4",
        "@rollup/rollup-linux-s390x-gnu": "4.22.4",
        "@rollup/rollup-linux-x64-gnu": "4.22.4",
        "@rollup/rollup-linux-x64-musl": "4.22.4",
        "@rollup/rollup-win32-arm64-msvc": "4.22.4",
        "@rollup/rollup-win32-ia32-msvc": "4.22.4",
        "@rollup/rollup-win32-x64-msvc": "4.22.4",
        "fsevents": "~2.3.2"
      }

This package list is required by: @angular-devkit/[email protected] -> @angular/[email protected] -> [email protected].

In my package-lock.json, the necessary @rollup/rollup-darwin-arm64 was missing, even though it seemed required. Oddly, it was marked as optional. Running npm install --include=optional fixed this locally, updating my package-lock.json to include the needed packages, allowing ng serve to run smoothly.

To ensure consistency, I deleted the package-lock.json (a potentially risky step) and ran a fresh npm install --include=optional. This worked on my local machine but not in the pipeline, where the required linux-x64 packages were still missing.

To address this, I deleted package-lock.json again and ran npm install --platform=linux --arch=x64, adding include=optional to my .npmrc file. This generated a package-lock.json with all the necessary OS/architecture packages. Following this, npm install succeeded both locally and in the pipeline. Each environment correctly installed only the relevant packages (Darwin for macOS locally and Linux for the pipeline).

If you don’t mind deleting package-lock.json, this method might work for you too. Ensure the required packages are in package-lock.json before rerunning npm install with include=optional.

Best of luck!

To clarify, now my package-lock.json includes the following:

......
"node_modules/@rollup/rollup-darwin-arm64": {
      "version": "4.22.4",
      "resolved": "xxx",
      "integrity": "xxx",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
        "darwin"
      ]
    },
    "node_modules/@rollup/rollup-darwin-x64": {
      "version": "4.22.4",
       "resolved": "xxx",
      "integrity": "xxx",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
        "darwin"
      ]
    },
    "node_modules/@rollup/rollup-linux-arm-gnueabihf": {
      "version": "4.22.4",
       "resolved": "xxx",
      "integrity": "xxx",,
      "cpu": [
        "arm"
      ],
      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
        "linux"
      ]
    },
    "node_modules/@rollup/rollup-linux-arm-musleabihf": {
      "version": "4.22.4",
       "resolved": "xxx",
      "integrity": "xxx",
      "cpu": [
        "arm"
      ],
      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
        "linux"
      ]
    },
    "node_modules/@rollup/rollup-linux-arm64-gnu": {
      "version": "4.22.4",
       "resolved": "xxx",
      "integrity": "xxx",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
        "linux"
      ]
    },
etc ....

Before, these packages were missing in the package-lock.json file.

@lpknv
Copy link

lpknv commented Oct 30, 2024

I encountered a similar issue, and here’s how I resolved it.

I’m working on a MacBook Pro with an M2 Pro chip, while my GitLab pipeline runs on a Linux (x64) environment.

In my case, the problem stemmed from the rollup package not installing certain optional dependencies:

"optionalDependencies": {
        "@rollup/rollup-android-arm-eabi": "4.22.4",
        "@rollup/rollup-android-arm64": "4.22.4",
        "@rollup/rollup-darwin-arm64": "4.22.4",
        "@rollup/rollup-darwin-x64": "4.22.4",
        "@rollup/rollup-linux-arm-gnueabihf": "4.22.4",
        "@rollup/rollup-linux-arm-musleabihf": "4.22.4",
        "@rollup/rollup-linux-arm64-gnu": "4.22.4",
        "@rollup/rollup-linux-arm64-musl": "4.22.4",
        "@rollup/rollup-linux-powerpc64le-gnu": "4.22.4",
        "@rollup/rollup-linux-riscv64-gnu": "4.22.4",
        "@rollup/rollup-linux-s390x-gnu": "4.22.4",
        "@rollup/rollup-linux-x64-gnu": "4.22.4",
        "@rollup/rollup-linux-x64-musl": "4.22.4",
        "@rollup/rollup-win32-arm64-msvc": "4.22.4",
        "@rollup/rollup-win32-ia32-msvc": "4.22.4",
        "@rollup/rollup-win32-x64-msvc": "4.22.4",
        "fsevents": "~2.3.2"
      }

This package list is required by: @angular-devkit/[email protected] -> @angular/[email protected] -> [email protected].

In my package-lock.json, the necessary @rollup/rollup-darwin-arm64 was missing, even though it seemed required. Oddly, it was marked as optional. Running npm install --include=optional fixed this locally, updating my package-lock.json to include the needed packages, allowing ng serve to run smoothly.

To ensure consistency, I deleted the package-lock.json (a potentially risky step) and ran a fresh npm install --include=optional. This worked on my local machine but not in the pipeline, where the required linux-x64 packages were still missing.

To address this, I deleted package-lock.json again and ran npm install --platform=linux --arch=x64, adding include=optional to my .npmrc file. This generated a package-lock.json with all the necessary OS/architecture packages. Following this, npm install succeeded both locally and in the pipeline. Each environment correctly installed only the relevant packages (Darwin for macOS locally and Linux for the pipeline).

If you don’t mind deleting package-lock.json, this method might work for you too. Ensure the required packages are in package-lock.json before rerunning npm install with include=optional.

Best of luck!

Thanks for this Infos. I will give it a try because I am experiencing similar issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug thing that needs fixing Needs Triage needs review for next steps Release 8.x work is associated with a specific npm 8 release
Projects
None yet
Development

Successfully merging a pull request may close this issue.