nx migrate randomly fails

[urugator]

Current Behavior

yarn nx migrate randomly fails, most likely due to some race conditions. It's always some FS error like ENOENT on a random file in yarn cache. I am absolutely positive it's random. You can run it over and over until it eventually succeeds, without making any changes whatsoever. Deleting caches or node_modules has no effect on this. The problem is it takes quite long before it fails and also it tends to fail more often than not.
It's a long standing issue, which me and my colleagues have been experiencing for as long as we are using nx - versions 14,15,16.

Examples of my last 2 attemps (see below for full log):

Command failed: yarn add -W @nx/[email protected]
...
error An unexpected error occurred: "https://redacted/repository/npmjs-all/@babel/core/-/core-7.22.5.tgz: ENOENT: no such file or directory, open 'C:\\Users\\urug\\AppData\\Local\\Yarn\\Cache\\v6\\npm-@babel-core-7.22.5-d67d9747ecf26ee7ecd3ebae1ee22225fe902a89-integrity\\node_modules\\@babel\\core\\.yarn-tarball.tgz'".`

Command failed: yarn add -W @nx/[email protected]
...
error https://redacted/repository/npmjs-all/@nx/node/-/node-16.3.2.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat 'C:\\Users\\urug\\AppData\\Local\\Yarn\\Cache\\v6\\npm-@nx-node-16.3.2-912d5906bd8dc3b26e6d9f3b0101231e4ee04e15-integrity\\node_modules\\@nx\\node\\src\\generators\\e2e-project\\files\\cli\\tsconfig.spec.json__tmpl__'"

Expected Behavior

yarn nx migrate doesn't fail randomly.

GitHub Repo

No response

Steps to Reproduce

1. yarn nx migrate latest

Nx Report

Node   : 19.2.0
   OS     : win32 x64
   yarn   : 1.22.19
   Hasher : Native

   nx                 : 16.1.4
   @nx/js             : 16.1.4
   @nx/jest           : 16.1.4
   @nx/linter         : 16.1.4
   @nx/workspace      : 16.1.4
   @nx/angular        : 16.1.4
   @nx/cypress        : 16.1.4
   @nx/devkit         : 16.1.4
   @nx/eslint-plugin  : 16.1.4
   @nx/nest           : 16.1.4
   @nx/node           : 16.1.4
   @nrwl/tao          : 16.1.4
   @nx/web            : 16.1.4
   @nx/webpack        : 16.1.4
   typescript         : 5.0.4
   ---------------------------------------
   Community plugins:
   @ngneat/transloco : 4.2.7

Failure Logs

redacted>yarn nx migrate latest --interactive
yarn run v1.22.19
$ redacted\node_modules\.bin\nx migrate latest --interactive
Fetching meta data about packages.
It may take a few minutes.
Fetching [email protected]
Fetching [email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]

 >  NX   The migrate command failed.


 >  NX   Command failed: yarn add -W @nx/[email protected]

   warning package.json: No license field
   warning No license field
   error An unexpected error occurred: "https://redacted/repository/npmjs-all/@babel/core/-/core-7.22.5.tgz: ENOENT: no such file or directory, open 'C:\\Users\\urug\\AppData\\Local\\Yarn\\Cache\\v6\\npm-@babel-core-7.22.5-d67d9747ecf26ee7ecd3ebae1ee22225fe902a89-integrity\\node_modules\\@babel\\core\\.yarn-tarball.tgz'".

   Pass --verbose to see the stacktrace.

Command failed: C:\Users\urug\AppData\Local\Temp\tmp-10448-FiEnlw5f1Okb\node_modules\.bin\nx _migrate latest --interactive


yarn run v1.22.19
$ redacted\node_modules\.bin\nx migrate latest --interactive
Fetching meta data about packages.
It may take a few minutes.
Fetching [email protected]
Fetching [email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]

 >  NX   The migrate command failed.


 >  NX   Command failed: yarn add -W @nx/[email protected]

   warning package.json: No license field
   warning No license field
   warning @nx/node > @nx/jest > jest-config > jest-environment-node > @jest/fake-timers > @sinonjs/[email protected]: Use version 10.1.0. Version 10.2.0 has potential breaking issues
   error https://redacted/repository/npmjs-all/@nx/node/-/node-16.3.2.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat 'C:\\Users\\urug\\AppData\\Local\\Yarn\\Cache\\v6\\npm-@nx-node-16.3.2-912d5906bd8dc3b26e6d9f3b0101231e4ee04e15-integrity\\node_modules\\@nx\\node\\src\\generators\\e2e-project\\files\\cli\\tsconfig.spec.json__tmpl__'"

   Pass --verbose to see the stacktrace.

Command failed: C:\Users\urug\AppData\Local\Temp\tmp-16096-wPD331wrq6KE\node_modules\.bin\nx _migrate latest --interactive
error Command failed with exit code 1.

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

No response

[elizww]

same to me in recent updates,

My guess that it relates to windows locking some files during a parallel fetching

alternatively can be since it fetches the same packages several times

Fetching [email protected]
Fetching [email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]
Fetching @nx/[email protected]

To workaround I migrate all @nx/ dependencies one by one and run the migration file once at the end.
I think it will be nice to have an option to run fetch serially instead

[xiongemi]

how about set env variable NX_MIGRATE_SKIP_INSTALL=true when running the migration command?
from the error message, it seems like the fetch failed when trying to get a package from your local proxy npm registry https://redacted/repository/npmjs-all.
Maybe skill the install command when running the migration script, then explicitly run the install command after the migration is done?

[urugator]

Afaict NX_MIGRATE_SKIP_INSTALL=true is only relevant for --run-migrations

it seems like the fetch failed when trying to get a package from your local proxy npm registry

It fails because of some race condition in the nx command logic. It has nothing to do with the registry. It probably tries to run some commands concurrently and it doesn't play nice (could be just windows thing, dunno).

Why closing the issue? Even if NX_MIGRATE_SKIP_INSTALL would be relevant, you just provided workaround, not a solution.

Btw NX_MIGRATE_SKIP_INSTALL is nowhere to be found in the documentation.

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.