-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Platform-specific optional dependencies not being included in package-lock.json
when reinstalling with node_modules
present
#4828
Comments
node_modules
present
node_modules
presentpackage-lock.json
when reinstalling with node_modules
present
Sorry to ping you out of the blue, but this issue has been open for 11 days now without any movement. Is there anyone working on npm right now that might have the bandwidth to at least validate that this is indeed a problem as I've described it? Just so that when someone does become available to do some development work they know that this is in the queue? Please and thank you. |
Bump |
I'm also encountering this issue with a Next.js project:
Unfortunately developers often don't realise the Here is a reproduction:
|
I am also having this issue. I'm trying to run tests using jest with swc. The test runner is a linux image, but my dev machine is darwin. I can get it to work by either using --force to install the linux dependency, or I can install packages from inside the container... but github CI stands up the docker container in such a way that I can't easily install packages from in there, and that also prevents me from maintaining a cached node modules etc. |
bump |
bump - cannot get optional dependencies (namely |
bump |
Confirming that this issue is still present. It's particularly important for projects using NAPI modules, as tons of them use platform-specific packages. |
Ran into this issue when creating a CI process for a repo where I use a Windows machine and the CI process is using Linux. My quick "fix" for now is to start the CI process by deleting the package-lock.json and running npm install instead of npm ci. I know this is not good practice, so looking forward to a real fix to come through. |
bump |
I am having a similar issue. My project uses What has changed between v6 and v8 and is there an npm config option that will have v8 work similar to v6 when it comes to optional dependencies? |
* Introduce useOrgUsersQuery composable * Add unit tests for useOrgUsersQuery * Fix linter offenses * Introduce useTasksQuery composable * Extend withSetup test util to accept app options * Add unit tests * Introduce useAddTaskMutation * Improve task addition error handling * Improve view model handling * Introduce useUpdateTaskMutation and add mutation keys * Add mutation unit tests * Introduce useAddTaskVariantMutation and useUpdateTaskVariantMutation composables * Fix useTasksQuery and implement on Manage Variants page * Renamed task variants query * Fix ref val usage in useTaskVariantsQuery * Add unit tests for useTaskVariantsQuery * Introduce useTasksQuery and optimise conditional task loading * Extend useUserType for super admin handling * Apply super admin user type check changes to HomeSelector * Introduce useAdministrationsQuery composable * Remove obsolete user claims query * Fix administrations loading spinner * Introduce useUpdateUserMutation * Cleanup and remove obsolete logging * Add unit tests for useUpdateUserMutation * Add useUserDataQuery unit tests * Leverage useUserType for isSuperAdmin status * Add missing import * Regenerate package-lock due to npm/cli#4828 * Prepare for composable queries * Replace user claims query with useUserClaimsQuery * Use userType composable * Introduce useAdministrationsQuery composable and apply on administrator homepage * Fix loading state * Fix linter offenses * Restore itemsPerPage to original value * Restore package files * Upgrade tanstack and install dev tools * Remove comment * Remove incorrect parameters * Introduce useUserAssignments composable * Fix reactivity and minor issues * Introduce revised useAdministrationsQuery composable * Extend useTasksQuery to fetch by task IDs * Move useTasksQuery function to helper * Fix tasks query fetching and constants * Update useTasksQuery unit tests * Fix loading & query reactivity * Introduce useSurveyReponsesQuery composable * Add useSurveyResponsesQuery unit tests * Add useUserAssignmentsQuery unit tests * Fix reactivity in query composables * Remove duplications in favour of computeQueryOverrides helper * Fix vue warning * Improve computeQueryOverride reactivity and testability * Add unit tests for useUserStudentDataQuery * Overhaul useUserDataQuery unit tests * Add useUserClaimsQuery unit tests * Overhaul useUserClaimsQuery unit tests * Simplify useTaskVariantsQuery unit tests * Align useTasksQuery unit test names * Overhaul useSurveyResponsesQuery unit tests * Add useAdministrationsQuery unit tests * Add useDsgfOrgQuery unit tests * Add useLegalDocsQuery unit tests * Add computeQueryOverrides unit tests * Switch from uid to roarUid for survey response querying * Migrate score report queries to existing query composables * Introduce useAdministrationAssignmentsQuery composable * Add useDistrictsQuery unit tests * Split district query into separate composables * Introduce useSchoolQuery composable * Introduce useClassQuery composable * Introduce useGroupQuery composable * Fix doc comments * Introduce useFamilyQuery composable * Fix useAdministrationsQuery unit tests * Introduce new fetchDocumentsById query helper * Replace useClassQuery with useClassesQuery composable * Restructure district queries * Replace useSchoolQuery with useSchoolsQuery composable * Replace useGroupQuery with useGroupsQuery composable * Replace useFamilyQuery with useFamiliesQuery * Extend useDistrictsListQuery tests * Replace manual administration query with useAdministrationsQuery * Introduce new useAdministrationsStatsQuery composable * Migrate ProgressReport to individual org queries * Introduce reusable useOrgQuery composable for dynamic org querying * Migrate remaining page queries to composables * Switch to router based report routing * Add unit tests * Renamed page to match naming * Adapt useUserDataQuery to optionally fetch by manual user ID * Introduce useUserAdministrationAssignmentsQuery composable * Replace manual query with useAdministrationsQuery * Introduce tempoary useUsrRunPageQuery composable * Add useUserAdministrationAssignmentsQuery unit tests * Fix linter offenses * Fix comment typos * Make TanStack dev tools load conditionally * Fix administration assignments query * Remove staging conditional in favour of env var * Temporarily disable progress and report filtering tests * Replace administrations query with useAdministrationsQuery composable * Replace districts and schools queries with composables * Align query omposables and fix reactivity issues * Replace classes query with useClassesQuery composable * Fix linter offenses * Fix query state reactivity * Replace groups query with useGroupsQuery composable * Replace families query with useFamiliesQuery composable * Refactor CreateAdminitration page data handling * Fix query state handling * Remove console logs and obsolete comments * Refactor consent handling for administrators * Apply Consent Modal changes to remaining pages * Fix query reactivity * Fix document path destructuring * Add markdown/html sanitization * Add remaining consent types * Add loop breaker * Remove obsolete import * Add useSignOutMutation * Fix and improve signout mutation * Apply signOut mutation across pages * Remove query key indexes from queries * Combine ClassLink and Clever into single SSO landing page * Refactor store sso source * Add (intentional) typos * Fix home selector routing for sso flow * Introduce useTasksDictionaryQuery composable * Replace authStore tasks dictionary in favour of query composable * Remove obsolete import * Remove onUpdated hook for task dictionary refresh * Use AUTH_SSO_PROVIDERS instead of string literals * Clear query data before sign-out redirect * Fix vue-query hook usage error * Improve error handling and logging * Add inline initStateFromRedirect comments * Add unit tests * Introduce useDeleteAdministrationMutation mutation * Improve administration loader * Fix error handling in fetchByDocId * Update JSDoc * Update src/components/auth/RegisterStudent.vue Co-authored-by: Kyle Montville <[email protected]> * Update src/components/auth/RegisterStudent.vue Co-authored-by: Kyle Montville <[email protected]> * Improve error handling * Updated copy to match form option * Revert "Improve error handling" This reverts commit 4fa5c46. * Fix post-signout firekit initialisation * Fix linter offenses * Fix SWR task loading * Fix Vocab task loading * Fix SRE task loading * Fix Ran task loading * Fix PA task loading * Fix Multichoice task loading * Fix MEP task loading * Fix Levante task loading * Fix Letter task loading * Fix Fluency task loading * Fix Crowding task loading * Fix tasks loading (#883) * Fix SWR task loading * Fix Vocab task loading * Fix SRE task loading * Fix Ran task loading * Fix PA task loading * Fix Multichoice task loading * Fix MEP task loading * Fix Levante task loading * Fix Letter task loading * Fix Fluency task loading * Fix Crowding task loading * Fix administration selection * Fix administrations list rendering (#888) * Fix participant homepage loading state (#889) * Fix flashing loading states on participant homepage * Fix refs --------- Co-authored-by: Emily Arteaga <[email protected]> Co-authored-by: Kyle Montville <[email protected]> Co-authored-by: Adam Richie-Halford <[email protected]>
bump |
At this point I don't think bumping this thread actually does anything. NPM clearly has an issue with differing chip architectures and they're not doing anything about it. So we'll have to keep doing our workarounds… |
The "easiest" (but maybe not cleanest) workaround for me has been to delete the Yes, I understand it sucks to have to delete your package-lock which means you likely need to do exhaustive testing to ensure no runtime errors exist, but it does work. There is a second, controversially dangerous option which is to copy just the JSON needed to add the new packages for the package at-hand, then restore your old |
I've tried everything here to solve this problem, but without success. I'm trying to run it using docker, and I always get the error bellow. I have no idea what to do anymore. I tried running it with NPM, PNPM and nothing works. app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:59
project-1 | throw new Error(
project-1 | ^
project-1 |
project-1 | Error: Cannot find module @rollup/rollup-linux-arm64-gnu. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.
project-1 | at requireWithFriendlyError (/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:59:9)
project-1 | at Object.<anonymous> (/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:68:76)
project-1 | ... 3 lines matching cause stack trace ...
project-1 | at Module._load (node:internal/modules/cjs/loader:1104:12)
project-1 | at cjsLoader (node:internal/modules/esm/translators:346:17)
project-1 | at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:286:7)
project-1 | at ModuleJob.run (node:internal/modules/esm/module_job:234:25)
project-1 | at async ModuleLoader.import (node:internal/modules/esm/loader:473:24) {
project-1 | [cause]: Error: Cannot find module '@rollup/rollup-linux-arm64-gnu'
project-1 | Require stack:
project-1 | - /app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js
project-1 | at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
project-1 | at Module._load (node:internal/modules/cjs/loader:1051:27)
project-1 | at Module.require (node:internal/modules/cjs/loader:1311:19)
project-1 | at require (node:internal/modules/helpers:179:18)
project-1 | at requireWithFriendlyError (/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:41:10)
project-1 | at Object.<anonymous> (/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js:68:76)
project-1 | at Module._compile (node:internal/modules/cjs/loader:1469:14)
project-1 | at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
project-1 | at Module.load (node:internal/modules/cjs/loader:1288:32)
project-1 | at Module._load (node:internal/modules/cjs/loader:1104:12) {
project-1 | code: 'MODULE_NOT_FOUND',
project-1 | requireStack: [
project-1 | '/app/node_modules/.pnpm/[email protected]/node_modules/rollup/dist/native.js'
project-1 | ]
project-1 | }
project-1 | }
project-1 |
project-1 | Node.js v20.18.0
project-1 | ELIFECYCLE Command failed with exit code 1.
server-1 | --watch/-w can be omitted, JSON Server 1+ watches for file changes by default This is my Dockerfile: FROM node:20.18.0-slim
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN npm install -g pnpm
RUN pnpm install
COPY . .
EXPOSE 3000
EXPOSE 3001
CMD ["sh", "-c", "if [ \"$SERVICE\" = 'project' ]; then pnpm dev; else pnpm init:db; fi"] |
To be un blocked you can do this: After npm install (or npm ci) you run the command to install the platform specific dependency. |
Thanks for your answer @dmail, I've already tried, but it still persists. I changed my dockerfile to adapt to this scenario. FROM node:20-buster
WORKDIR /app
RUN apt-get update && apt-get install -y \
build-essential \
python3 \
&& rm -rf /var/lib/apt/lists/*
COPY package.json package-lock.json ./
RUN npm ci
RUN npm install @rollup/rollup-linux-arm64-gnu
COPY . .
EXPOSE 3000
EXPOSE 3001
CMD ["sh", "-c", "if [ \"$SERVICE\" = 'front' ]; then npm run dev; else npm run init:db; fi"] |
I don't know if it helps, but I had to install the two dependencies below to solve my scenario: "@esbuild/linux-arm64": "0.20.2",
"@rollup/rollup-linux-arm64-gnu": "4.24.0", |
My case was running At the end of the day I changed my default Node version which installed via NPM, from 16 to 20, and then all errors are gone! |
The browser tests failed in CI with this error message: Error: Cannot find module @rollup/rollup-linux-x64-gnu. npm has a bug related to optional dependencies (npm/cli#4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.
Trying to bypass this npm bug: npm/cli#4828
I run Docker development containers based on the node:20 image. An error occurred in the backend container related to the @sws/core and @swc/cli packages. There is also an error in the frontend container: "Error: I can't find the @rollup/rollup-linux-x64-musl " module. After several days of searching and temporary solutions, I realized that the error occurs when using the node:20 image based on alpine. If you use the node:20-bookworm or node:20-bullseye images, there will be no error. Maybe this will be useful to someone. |
npm has a bug related to optional dependencies (npm/cli#4828).
npm has a bug related to optional dependencies (npm/cli#4828).
For whatever reason, after every merge conflict I have to fix this and I'll eventually get it fixed, only to fail in my CI. I'm having to delete |
I am also experiencing a similar issue using node:22-alpine for my sveltekit / vite docker deployment and currently trying out different node docker version (bookworm, slim etc). |
Running into this issue on my gitlab instance for CI runners. Our project is quite large and caching node_modules when |
I encountered a similar issue, and here’s how I resolved it. I’m working on a MacBook Pro with an M2 Pro chip, while my GitLab pipeline runs on a Linux (x64) environment. In my case, the problem stemmed from the "optionalDependencies": {
"@rollup/rollup-android-arm-eabi": "4.22.4",
"@rollup/rollup-android-arm64": "4.22.4",
"@rollup/rollup-darwin-arm64": "4.22.4",
"@rollup/rollup-darwin-x64": "4.22.4",
"@rollup/rollup-linux-arm-gnueabihf": "4.22.4",
"@rollup/rollup-linux-arm-musleabihf": "4.22.4",
"@rollup/rollup-linux-arm64-gnu": "4.22.4",
"@rollup/rollup-linux-arm64-musl": "4.22.4",
"@rollup/rollup-linux-powerpc64le-gnu": "4.22.4",
"@rollup/rollup-linux-riscv64-gnu": "4.22.4",
"@rollup/rollup-linux-s390x-gnu": "4.22.4",
"@rollup/rollup-linux-x64-gnu": "4.22.4",
"@rollup/rollup-linux-x64-musl": "4.22.4",
"@rollup/rollup-win32-arm64-msvc": "4.22.4",
"@rollup/rollup-win32-ia32-msvc": "4.22.4",
"@rollup/rollup-win32-x64-msvc": "4.22.4",
"fsevents": "~2.3.2"
} This package list is required by: In my To ensure consistency, I deleted the To address this, I deleted If you don’t mind deleting Best of luck! |
To clarify, now my ......
"node_modules/@rollup/rollup-darwin-arm64": {
"version": "4.22.4",
"resolved": "xxx",
"integrity": "xxx",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/@rollup/rollup-darwin-x64": {
"version": "4.22.4",
"resolved": "xxx",
"integrity": "xxx",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/@rollup/rollup-linux-arm-gnueabihf": {
"version": "4.22.4",
"resolved": "xxx",
"integrity": "xxx",,
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-arm-musleabihf": {
"version": "4.22.4",
"resolved": "xxx",
"integrity": "xxx",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-arm64-gnu": {
"version": "4.22.4",
"resolved": "xxx",
"integrity": "xxx",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
etc .... Before, these packages were missing in the |
Thanks for this Infos. I will give it a try because I am experiencing similar issue. |
Is there an existing issue for this?
This issue exists in the latest npm version
Current Behavior
I'm working on a team that utilizes a mix of x64-based and m1-based macs, and has CI build processes that uses musl. We're seeing that
npm
is skipping platform-specific optional dependencies for packages such as@swc/core
as a result of thepackage-lock.json
file being generated without all of them included. In our case, this then causes linting to throw an exception, because one of our eslint plugins depends on @swc, which depends on having the platform specific @swc package also installed.There seems to be at least two stages of cause to this. Firstly, when installing
@swc/core
from a clean slate working directorynpm
generates apackage-lock.json
with all of the optional dependencies for@swc/core
listed:And it only installs the platform specific package:
If I then remove my
package-lock.json
, leave mynode_modules
directory as-is, and then reinstall, I get:That is, it then generates a package-lock.json with only the platform-specific dependency that was installed on this machine, and not with the other optional dependencies that should also be listed.
If you delete both
node_modules
ANDpackage-lock.json
, and then re-runnpm install
, it generates the correct lockfile with all of those optional dependencies listed.The problem is that then, If the
package-lock.json
with the missing optional platform-specific dependencies gets checked into git and an x64 user pulls it down, or vice-versa,npm
fails to detect that your platform's optional dependencies are missing in the lockfile and just silently skips installing the platform-specific dependency. For example, when I've got a package-lock.json that only contains the x64 @swc package because of the above problem (generated by my coworker on his x64 machine):And I then install:
You can see that it fails to install the arm64 dependency or warn me in any way that the
package-lock.json
is missing my platform's dependency.So yeah, two problems:
Expected Behavior
npm
should preserve the full set of platform-specific optional deps for a package like @swc when rebuildingpackage-lock.json
from an existingnode_modules
treenpm install
should warn if thepackage-lock.json
becomes inconsistent because of the first caseSteps To Reproduce
See above.
Environment
The text was updated successfully, but these errors were encountered: