Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"NX Cannot find module '@nrwl/nx-linux-arm64-gnu'" when running angular universal in a node:18 docker #15452

Closed
jwittekind opened this issue Mar 5, 2023 · 30 comments · Fixed by #16446
Assignees
Labels

Comments

@jwittekind
Copy link

Current Behavior

I just migrated using nx migrate latest

angular packages: 15.1.1 => 15.2.1

@nrwl/angular: 15.7.2 => 15.8.5
@nrwl/webpack: 15.7.2 => 15.8.5

I'm running angular universal in development mode and start it in a docker node:18

Now I get the following error:
> NX Cannot find module '@nrwl/nx-linux-arm64-gnu'

As described in a similar issue, #8969
I tried to delete my node_modules and clear my cache.

It seems like my application works if I run my nx command outside docker:
nx run my-app:serve-ssr

When starting my docker image, it throws the error above.

If i check my package-lock.json the package is listed under "optionalDependencies":
"@nrwl/nx-linux-arm64-gnu": "15.8.5"

Expected Behavior

After the migration it may should have installed the missing module @nrwl/nx-linux-arm64-gnu and run.

GitHub Repo

No response

Steps to Reproduce

Docker Image:

FROM node:18

RUN mkdir -p /app
WORKDIR /app/

COPY package*.json ./
COPY decorate-angular-cli.js ./decorate-angular-cli.js
RUN npm install -g @angular/cli@15 @angular-devkit/build-angular@15

RUN nx run my-app:serve-ssr

Nx Report

Node : 18.14.2
   OS   : darwin arm64
   npm  : 9.5.0
   
   nx                      : 15.8.5
   @nrwl/js                : 15.8.5
   @nrwl/jest              : 15.8.5
   @nrwl/linter            : 15.8.5
   @nrwl/workspace         : 15.8.5
   @nrwl/angular           : 15.8.5
   @nrwl/cli               : 15.8.5
   @nrwl/cypress           : 15.8.5
   @nrwl/devkit            : 15.8.5
   @nrwl/eslint-plugin-nx  : 15.8.5
   @nrwl/storybook         : 15.8.5
   @nrwl/tao               : 15.8.5
   @nrwl/webpack           : 15.8.5
   typescript              : 4.9.5
   ---------------------------------------
   Community plugins:
   @nguniversal/express-engine : 15.2.0
   apollo-angular              : 4.2.1
   ng2-charts                  : 4.1.1
   @nguniversal/builders       : 15.2.0
   @storybook/angular          : 6.5.15

Failure Logs

No response

Additional Information

No response

@nguyenchu2306
Copy link

nguyenchu2306 commented Mar 9, 2023

I faced a similar issues to yours, My resolution was to do the following steps:

  • Add this block in your Jenkins build stage withEnv(['NX_NON_NATIVE_HASHER=true']).
  • Remove --ignore-optional from your yarn install if any.

@jwittekind
Copy link
Author

Hey @nguyenchu2306!

Thank you, for your reply.

Actually I'm using docker in the context of a local development environment on macOS, where there is no Jenkins, no pipeline, and I'm using npm.

I temporarily solved the error by downgrading back to my earlier version.
Nothing else was changed in my setup though.
My expectation would be, that the nx serve command works the same after an update - if there are no breaking changes.

@nguyenchu2306
Copy link

nguyenchu2306 commented Mar 9, 2023

Hi @jwittekind,
If you use other CI then Jenkins, you can added the NX_NON_NATIVE_HASHER=true to your .env file.
It'll work as well, since NX_NON_NATIVE_HASHER is introduce in Nx 15.8 that use rush hasher and setting this to true will disable the new rush hasher. Medium blog

@jwittekind
Copy link
Author

jwittekind commented Mar 9, 2023

Thank you @nguyenchu2306!
Looks like my nodejs container might be one of the missed environments to be supported.
I guess, that the support may be yet to come, right? In that Case, I'd prefer to avoid this temporary workaround.

I'll stick with the downgrade for now.

If this feature is not be planned to be widely supported - i'd prefer an "opt in" instead of an "opt out".

@AgentEnder AgentEnder added the scope: core core nx functionality label Mar 13, 2023
@Cammisuli
Copy link
Member

Cammisuli commented Mar 14, 2023

We want to support all the environments that people use. It's not too difficult for us to build binaries for different systems. But we do build binaries for linux-arm64-gnu.

What is the content of your package.json?

Also, it looks like you don't do a npm install/yarn install in the dockerfile?

@Cammisuli
Copy link
Member

related to #15380.

@jwittekind
Copy link
Author

jwittekind commented Mar 14, 2023

Hey @Cammisuli

The docker container uses a mounted volume to use the node_modules directory of the host machine.

here's my Package.json of my downgraded repository ;)

{
  "name": "my-monorepo",
  "version": "0.0.0",
  "private": true,
  "license": "MIT",
  "scripts": {
    "prepare": "husky install",
    "ng": "nx",
    "postinstall": "node ./decorate-angular-cli.js",
    "start": "public:dev",
    "build": "nx build",
    "test": "nx test",
    "dev": "nx serve",
    ...
  },
  "dependencies": {
    "@angular/animations": "15.1.1",
    "@angular/cdk": "15.1.2",
    "@angular/common": "15.1.1",
    "@angular/compiler": "15.1.1",
    "@angular/core": "15.1.1",
    "@angular/flex-layout": "^14.0.0-beta.41",
    "@angular/forms": "15.1.1",
    "@angular/localize": "15.1.1",
    "@angular/material": "15.1.2",
    "@angular/platform-browser": "15.1.1",
    "@angular/platform-browser-dynamic": "15.1.1",
    "@angular/platform-server": "15.1.1",
    "@angular/router": "15.1.1",
    "@apollo/client": "^3.7.5",
    "@nguniversal/express-engine": "15.1.0",
    "@nrwl/angular": "15.7.2",
    "@nrwl/webpack": "15.7.2",
    "apollo-angular": "^4.2.0",
    "axios": "^1.1.3",
    "body-parser": "^1.20.1",
    "caniuse-lite": "^1.0.30001456",
    "compression": "^1.7.4", 
    "date-fns": "^2.29.3",
    "domino": "^2.1.6",
    "express": "^4.15.2",
    "http-proxy": "^1.18.1",
    "http-proxy-middleware": "^2.0.6",
    "rxjs": "~7.5.0",
    "tslib": "^2.3.0",
    "zone.js": "~0.11.4"
  },
  "devDependencies": {
    "@angular-devkit/architect": "^0.1502.1",
    "@angular-devkit/build-angular": "15.1.1",
    "@angular-devkit/core": "~15.1.0",
    "@angular-devkit/schematics": "~15.1.0",
    "@angular-eslint/eslint-plugin": "15.0.0",
    "@angular-eslint/eslint-plugin-template": "15.0.0",
    "@angular-eslint/template-parser": "15.0.0",
    "@angular/cli": "~15.1.0",
    "@angular/compiler-cli": "15.1.1",
    "@angular/language-service": "15.1.1",
    "@graphql-codegen/cli": "^2.13.11",
    "@graphql-codegen/fragment-matcher": "^3.3.1",
    "@graphql-codegen/introspection": "^2.2.1",
    "@graphql-codegen/near-operation-file-preset": "^2.4.4",
    "@graphql-codegen/schema-ast": "^2.5.1",
    "@graphql-codegen/typescript": "^2.8.1",
    "@graphql-codegen/typescript-apollo-angular": "^3.5.6",
    "@graphql-codegen/typescript-graphql-files-modules": "^2.2.1",
    "@graphql-codegen/typescript-graphql-request": "^4.5.8",
    "@graphql-codegen/typescript-operations": "^2.5.6",
    "@nguniversal/builders": "15.1.0",
    "@nrwl/cli": "15.7.2",
    "@nrwl/cypress": "15.7.2",
    "@nrwl/eslint-plugin-nx": "15.7.2",
    "@nrwl/jest": "15.7.2",
    "@nrwl/linter": "15.7.2",
    "@nrwl/storybook": "15.7.2",
    "@nrwl/workspace": "15.7.2",
    "@schematics/angular": "~15.1.0",
    "@storybook/addon-essentials": "6.5.15",
    "@storybook/angular": "6.5.15",
    "@storybook/builder-webpack5": "6.5.15",
    "@storybook/core-server": "6.5.15",
    "@storybook/manager-webpack5": "6.5.15",
    "@types/apollo-upload-client": "^17.0.2",
    "@types/body-parser": "^1.19.2",
    "@types/compression": "^1.7.2",
    "@types/cookie-parser": "^1.4.3",
    "@types/express": "^4.17.0",
    "@types/jest": "28.1.1",
    "@types/node": "16.11.7",
    "@typescript-eslint/eslint-plugin": "^5.36.1",
    "@typescript-eslint/parser": "^5.36.1",
    "cypress": "^12.7.0",
    "eslint": "~8.15.0",
    "eslint-config-prettier": "8.1.0",
    "eslint-plugin-cypress": "^2.10.3",
    "husky": "^8.0.1",
    "jest": "28.1.1",
    "jest-environment-jsdom": "28.1.1",
    "jest-preset-angular": "12.2.3",
    "lint-staged": "^13.0.3",
    "nx": "15.7.2",
    "prettier": "^2.6.2",
    "source-map-explorer": "^2.5.3",
    "ts-jest": "28.0.5",
    "ts-node": "10.9.1",
    "typescript": "~4.8.2",
    "webpack": "^5.64.0",
    "webpack-bundle-analyzer": "^4.8.0"
  }
}

@Cammisuli
Copy link
Member

Oh, thats the reason then.

Npm/yarn will install the specific native binary for your OS and arch. So if you run npm install on your darwin based os, you will only get @nrwl/nx-darwin-arm, but because your docker container is based on linux, it needs @nrwl/nx-linux-arm64-gnu. The only way to get that installed on a system is to execute npm install.

We can't specifically add those packages as deps because there's a hard requirement on the os and architecture of the host, and npm will fail the install if it detects a different os/arch than what is specified.

So I suggest to not mount the host node_modules, and do a npm install in the container.

@GeorgianStan
Copy link

GeorgianStan commented Mar 16, 2023

@Cammisuli I'm facing with the same issue. My docker file content is the following:

# Stage 1
FROM node:latest as build

WORKDIR /usr/local/app
COPY ./ /usr/local/app/

RUN npm ci

# ARG NX_NON_NATIVE_HASHER=true
RUN npm run build

# Stage 2
FROM nginx:latest

COPY --from=build /usr/local/app/dist/my-app /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf

EXPOSE 80

With this config I get Cannot find module '@nrwl/nx-linux-x64-gnu', but as it can be seen I'm running npm ci which will remove the copied version of node_modules folder and install them again on the container.
The only workaround for now is to uncomment the # ARG NX_NON_NATIVE_HASHER=true line.

@timonmasberg
Copy link

timonmasberg commented Mar 16, 2023

We are having a similar issue with '@nrwl/nx-linux-x64-gnu'. We are installing our dependencies with npm ci without any flags in GitHub actions on ubuntu-latest and node 18. https://github.com/kordis-leitstelle/kordis/actions/runs/4436908773/jobs/7785926966

@Cammisuli
Copy link
Member

Thank you @timonmasberg! Now I can get my hands dirtier and take your run apart 🙂

@Cammisuli
Copy link
Member

After looking at @timonmasberg flow, it looked like the package-lock.json removed the needed packages.

So for people having issues still, please take a look at your package-lock.json, pnpm lock, yarn lock, all your locks and make sure that the native packages are listed there. Specifically look for @nrwl/nx-linux-x64-gnu

@GeorgianStan
Copy link

@Cammisuli, since package-lock.json comes from running npm i, shouldn't this dependency be included with the nx / @nrwl libraries?

@Cammisuli
Copy link
Member

Yes they are. The binary packages are optional dependencies on nx. For some reason it looks like the package-lock.json doesn't include them.

@jaredtbates
Copy link

So for those of us who are running npm i on local workstations, committing package-lock.json to a repo, and having GitHub Actions run npm ci on the project, is it recommended to explicitly add @nrwl/nx-linux-x64-gnu to our package.jsons?

@Cammisuli
Copy link
Member

@jaredtbates no you should not explicitly add the native packages to your package.json files. Those packages are already part of the optional dependencies for the nx package.

Just make sure to run npm install without --no-optional and ensure that the generated package-lock.json has the optional dependencies listed.

@damiangreen
Copy link

@nrwl/nx-linux-x64-gnu is not in my pnpm lock. Am i meant to add it explicitly?

@intellix
Copy link

intellix commented Mar 21, 2023

sorry for jumping on this as well but it's causing me issues in Cloudflare Pages and I've no idea what's going on. They use npm install and I've even tried adding a preliminary npm ci step before building my project.

node: v16.19.1
npm: v8.19.3

I see these inside package-lock.json:

"optionalDependencies": {
  "@nrwl/nx-darwin-arm64": "15.8.7",
  "@nrwl/nx-darwin-x64": "15.8.7",
  "@nrwl/nx-linux-arm-gnueabihf": "15.8.7",
  "@nrwl/nx-linux-arm64-gnu": "15.8.7",
  "@nrwl/nx-linux-arm64-musl": "15.8.7",
  "@nrwl/nx-linux-x64-gnu": "15.8.7",
  "@nrwl/nx-linux-x64-musl": "15.8.7",
  "@nrwl/nx-win32-arm64-msvc": "15.8.7",
  "@nrwl/nx-win32-x64-msvc": "15.8.7"
},

Error when installing:

Error: Cannot find module '@nrwl/nx-linux-x64-gnu'

The only way I'm able to fix it is by hooking into the build command within package.json:

"build": "npm i @nrwl/nx-linux-x64-gnu && nx run build:frontend",

timonmasberg added a commit to kordis-leitstelle/kordis that referenced this issue Mar 22, 2023
@jahller
Copy link

jahller commented Mar 24, 2023

we have the same issue right now.
GitLab pipeline, where the yarn cache is build and stored on an Alpine based Docker image.
Tests are then executed on an official Cypress Docker image, which is based on Debian/bullseye.
Which leads to this error: Cannot find module '@nrwl/nx-linux-x64-gnu'

@Cammisuli
Copy link
Member

@jahller does the yarn.lock file have the native binaries listed?

It should be something similar to this (with whatever version you're using):

"@nrwl/nx-darwin-arm64@npm:15.9.0-beta.7":
  version: 15.9.0-beta.7
  resolution: "@nrwl/nx-darwin-arm64@npm:15.9.0-beta.7"
  conditions: os=darwin & cpu=arm64
  languageName: node
  linkType: hard

"@nrwl/nx-darwin-x64@npm:15.9.0-beta.7":
  version: 15.9.0-beta.7
  resolution: "@nrwl/nx-darwin-x64@npm:15.9.0-beta.7"
  conditions: os=darwin & cpu=x64
  languageName: node
  linkType: hard

"@nrwl/nx-linux-arm-gnueabihf@npm:15.9.0-beta.7":
  version: 15.9.0-beta.7
  resolution: "@nrwl/nx-linux-arm-gnueabihf@npm:15.9.0-beta.7"
  conditions: os=linux & cpu=arm
  languageName: node
  linkType: hard

"@nrwl/nx-linux-arm64-gnu@npm:15.9.0-beta.7":
  version: 15.9.0-beta.7
  resolution: "@nrwl/nx-linux-arm64-gnu@npm:15.9.0-beta.7"
  conditions: os=linux & cpu=arm64 & libc=glibc
  languageName: node
  linkType: hard

"@nrwl/nx-linux-arm64-musl@npm:15.9.0-beta.7":
  version: 15.9.0-beta.7
  resolution: "@nrwl/nx-linux-arm64-musl@npm:15.9.0-beta.7"
  conditions: os=linux & cpu=arm64 & libc=musl
  languageName: node
  linkType: hard

"@nrwl/nx-linux-x64-gnu@npm:15.9.0-beta.7":
  version: 15.9.0-beta.7
  resolution: "@nrwl/nx-linux-x64-gnu@npm:15.9.0-beta.7"
  conditions: os=linux & cpu=x64 & libc=glibc
  languageName: node
  linkType: hard

"@nrwl/nx-linux-x64-musl@npm:15.9.0-beta.7":
  version: 15.9.0-beta.7
  resolution: "@nrwl/nx-linux-x64-musl@npm:15.9.0-beta.7"
  conditions: os=linux & cpu=x64 & libc=musl
  languageName: node
  linkType: hard

"@nrwl/nx-win32-arm64-msvc@npm:15.9.0-beta.7":
  version: 15.9.0-beta.7
  resolution: "@nrwl/nx-win32-arm64-msvc@npm:15.9.0-beta.7"
  conditions: os=win32 & cpu=arm64
  languageName: node
  linkType: hard

"@nrwl/nx-win32-x64-msvc@npm:15.9.0-beta.7":
  version: 15.9.0-beta.7
  resolution: "@nrwl/nx-win32-x64-msvc@npm:15.9.0-beta.7"
  conditions: os=win32 & cpu=x64
  languageName: node
  linkType: hard

@jahller
Copy link

jahller commented Mar 27, 2023

@Cammisuli
yes, all of them

"@nrwl/nx-darwin-arm64@npm:15.8.7":
  version: 15.8.7
  resolution: "@nrwl/nx-darwin-arm64@npm:15.8.7"
  conditions: os=darwin & cpu=arm64
  languageName: node
  linkType: hard

"@nrwl/nx-darwin-x64@npm:15.8.7":
  version: 15.8.7
  resolution: "@nrwl/nx-darwin-x64@npm:15.8.7"
  conditions: os=darwin & cpu=x64
  languageName: node
  linkType: hard

"@nrwl/nx-linux-arm-gnueabihf@npm:15.8.7":
  version: 15.8.7
  resolution: "@nrwl/nx-linux-arm-gnueabihf@npm:15.8.7"
  conditions: os=linux & cpu=arm
  languageName: node
  linkType: hard

"@nrwl/nx-linux-arm64-gnu@npm:15.8.7":
  version: 15.8.7
  resolution: "@nrwl/nx-linux-arm64-gnu@npm:15.8.7"
  conditions: os=linux & cpu=arm64 & libc=glibc
  languageName: node
  linkType: hard

"@nrwl/nx-linux-arm64-musl@npm:15.8.7":
  version: 15.8.7
  resolution: "@nrwl/nx-linux-arm64-musl@npm:15.8.7"
  conditions: os=linux & cpu=arm64 & libc=musl
  languageName: node
  linkType: hard

"@nrwl/nx-linux-x64-gnu@npm:15.8.7":
  version: 15.8.7
  resolution: "@nrwl/nx-linux-x64-gnu@npm:15.8.7"
  conditions: os=linux & cpu=x64 & libc=glibc
  languageName: node
  linkType: hard

"@nrwl/nx-linux-x64-gnu@npm:15.8.8":
  version: 15.8.8
  resolution: "@nrwl/nx-linux-x64-gnu@npm:15.8.8"
  checksum: 22a7948e7dec7532145351020ee5753775330e3cc6c9477c67721febca459125fc5db746c1139672676cecffec23b17965ced804dc258704466cbec4f9150e23
  conditions: os=linux & cpu=x64 & libc=glibc
  languageName: node
  linkType: hard

"@nrwl/nx-linux-x64-musl@npm:15.8.7":
  version: 15.8.7
  resolution: "@nrwl/nx-linux-x64-musl@npm:15.8.7"
  conditions: os=linux & cpu=x64 & libc=musl
  languageName: node
  linkType: hard

"@nrwl/nx-linux-x64-musl@npm:15.8.8":
  version: 15.8.8
  resolution: "@nrwl/nx-linux-x64-musl@npm:15.8.8"
  checksum: d92f289f2aad5b97f91d07874e1c271dd5f0d1167fdb4bee438c34f9a70f14ef826dcc9cd60c9a69bfa8c35d9e1e3018b0d1b5c5fbb3407eb15a071b1de5620a
  conditions: os=linux & cpu=x64 & libc=musl
  languageName: node
  linkType: hard

"@nrwl/nx-win32-arm64-msvc@npm:15.8.7":
  version: 15.8.7
  resolution: "@nrwl/nx-win32-arm64-msvc@npm:15.8.7"
  conditions: os=win32 & cpu=arm64
  languageName: node
  linkType: hard

"@nrwl/nx-win32-x64-msvc@npm:15.8.7":
  version: 15.8.7
  resolution: "@nrwl/nx-win32-x64-msvc@npm:15.8.7"
  conditions: os=win32 & cpu=x64
  languageName: node
  linkType: hard

@Cammisuli
Copy link
Member

@jahller is there a way for me to see the CI run or is it internal only?

@jahller
Copy link

jahller commented Mar 27, 2023

@jahller is there a way for me to see the CI run or is it internal only?

it's a private CI unfortunately. But just ask me, what you would like to know and I'll try to provide the information.

@andyjessop
Copy link

andyjessop commented Mar 27, 2023

I faced a similar issue just now.

Problem: my package.lock was corrupted because I'd been doing some really stupid shenanigans locally. When I committed, I was getting failed CI runs (very basic CI action setup) here: andyjessop/pivot#15

Solution: Removed node_modules and package-lock.json, ran npm install and committed the changes. CI then passed.

@localpcguy
Copy link

Running into this as well, and 2 things recommended in this thread fixed it:

remove --no-optional from the npm install command for CI (recommend in a couple of these issue threads)
remove node_modules and package-lock.json locally, then run npm install and commit the changes (ala @Cammisuli recommendation to check the lock file(s) and @andyjessop last post)

@damiangreen
Copy link

blowing away package lock files can be a little dangerous as we end up getting new and potentially broken versions

@andyjessop
Copy link

@damiangreen yes, that's correct. I should have pulled the package-lock.json from the remote. The only reason I wrote the above is because those were the actual steps I took to fix the issue.

@localpcguy
Copy link

localpcguy commented Mar 29, 2023

blowing away package lock files can be a little dangerous as we end up getting new and potentially broken versions

I would argue rebuilding the package lock files should not be considered a dangerous procedure, and something you could do at any time. Using the appropriate version range descriptors in the package.json should make it so you only get updates that you are comfortable that are compatible and not breaking.

As an example of when I'd regenerate it, since the lock file is a generated file, I never try to resolve merge conflicts in it manually. I just delete it and regenerate it, then commit it.

@Cammisuli
Copy link
Member

I've updated the docs site to include some information on how to trouble shoot this issue.

In summary, it's usually because of npm not including optional dependencies when node_modules already exist while doing an Nx update.

https://nx.dev/recipes/ci/troubleshoot-nx-install-issues

I'll need to add another point of running npm install without using --no-optional.

@github-actions
Copy link

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.