yarn install --production doesn't install correct dependencies

[donovan-graham]

When running yarn install --production it does not install the required dependencies of forever. This seem to be related to havingnodemon in devDependencies.

Error response:

> forever app.js
module.js:457
    throw err;
    ^
Error: Cannot find module 'minimatch'

I've created a test application here:
https://github.com/donovan-graham/yarn-example-app

#  Steps to reproduce error
git clone https://github.com/donovan-graham/yarn-example-app.git
cd yarn-example-app
yarn install --production
npm start

#  temporary step to bypass error
rm -rf node_modules
yarn remove nodemon
yarn install --production
npm start

[sendilkumarn]

@Daniel15 I guess this is because of nodemon is having latest version of minimatch.

The linker function currently takes in both deps and dev deps into it. For argument production, this should be prevented.

Even on normal yarn install with out production argument. Only latest version is installed in the actual path. This has to checked too.

[liorbrauer]

I'm getting a similar problem when running yarn install --production and then trying to run my build using webpack (running yarn install works fine).

> NODE_ENV=production webpack -p --config webpack/production.config.js

module.js:457
    throw err;
    ^

Error: Cannot find module 'graceful-fs'
    at Function.Module._resolveFilename (module.js:455:15)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)

and if I recall correctly, previous attempts showed similar errors with another package (not only graceful-fs)

[comfroels]

I'm getting very similar as well... yarn install works just fine. but with the --production flag I get this:

> yarn install --production

yarn install v0.15.1
error npm-shrinkwrap.json found. This will not be updated or respected. See [TODO] for more information.
[1/4] Resolving packages...
[2/4] Fetching packages...
warning [email protected]: The engine "rhino" appears to be invalid.
warning [email protected]: The platform "win32" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
warning [email protected]: The engine "rhino" appears to be invalid.
warning [email protected]: The engine "rhino" appears to be invalid.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[1/1] ⠐ node-sass:     at Module.require (module.js:367:17)
[-/1] ⠐ waiting...
[-/1] ⠐ waiting...
[-/1] ⠐ waiting...
error C:\vagrant\ebroker-quoteengine\node_modules\node-sass: Command failed.
Exit code: 1
Command: C:\WINDOWS\system32\cmd.exe
Arguments: /d /s /c node scripts/install.js
Directory: C:\vagrant\ebroker-quoteengine\node_modules\node-sass
Output:
module.js:341
    throw err;
    ^

Error: Cannot find module 'tough-cookie'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at Object.<anonymous> (C:\vagrant\ebroker-quoteengine\node_modules\node-sass\node_modules\request\lib\cookies.js:3:13)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Module.require (module.js:367:17)
    at SpawnError (C:\Users\nathan.white\AppData\Roaming\npm\node_modules\yarnpkg\lib\errors.js:18:1)
    at ChildProcess.<anonymous> (C:\Users\nathan.white\AppData\Roaming\npm\node_modules\yarnpkg\lib\util\child.js:107:15)
    at emitTwo (events.js:100:13)
    at ChildProcess.emit (events.js:185:7)
    at maybeClose (internal/child_process.js:827:16)
    at Socket.<anonymous> (internal/child_process.js:319:11)
    at emitOne (events.js:90:13)
    at Socket.emit (events.js:182:7)
    at Pipe._onclose (net.js:471:12)

[jkrems]

Can reproduce a similar issue with:

npm init --yes
yarn add --dev nodemon
yarn add gulp
rm -rf node_modules
yarn install --production

This will install is-glob but not its dependency is-extglob:

> yarn why is-glob
yarn why v0.16.0
# ...
info Reasons this module exists
   - "nodemon#chokidar" depends on it
   - "gulp#liftoff#findup-sync" depends on it

> yarn why is-extglob
yarn why v0.16.0
#  ...
info This module exists because "nodemon#chokidar#is-glob" depends on it.

It seems to "forget" the gulp#liftoff dependency path while traversing..?

EDIT: Smaller example:

npm init --yes
yarn add --dev [email protected]
yarn add [email protected]
rm -rf node_modules
yarn --prod
node -e "require('is-glob')"

Also confirmed that removing devDependencies before running yarn --prod installs the correct dependency tree.

[ArkadyDR]

My software team ran into this issue, specifically with the package prr which is a dependency of both less and pouchdb. Many other packages were also missing from the --production build but prr was the first to cause a failure in our product. This issue has been a showstopper for us as the size of our installer would significantly increase if we included the dev packages, so we've returned to using npm.

[gihrig]

FWIW: I can work around the issue by deleting devDependencies section from package.json prior to running yarn in production.

[royriojas]

as @gihrig said running npm prune --production can remove the devDependencies which helps to workaround this issue.

[tanx]

as @gihrig said running npm prune --production can remove the devDependencies which helps to workaround this issue.

Yarn's main advantage over npm is a deterministic node_modules dir i.e. same in dev, CI, and production out of the box. Does running npm prune --production yield that same behavior?

My current workaround is to just install devDependencies in production as well. Disk is cheap (especially on AWS) and a deterministic install is much more important to me than disk space. So my "workaround" is to just act like yarn --production does not exist right now .

[royriojas]

@tanx npm prune --production just remove the devDependencies. And in my tests always the same modules were removed. On the other hand, yes disk space is cheap, so maybe acting as if yarn --production does not exits is a better workaround :)

[tanx]

@tanx npm prune --production just remove the devDependencies. And in my tests always the same modules were removed.

This is precisely the "works on my machine" mentality described in the Yarn blog post. The problem is you're letting npm change the state of node_modules without yarn's integrity check via theyarn.lock file.

[kylecordes]

Hopefully the workarounds discussed will be rendered moot soon by an yarn update to respect dev vs production deps. In the meantime, indeed there is much to groan about with the "npm prune" post-processing hack.

[jkrems]

The yarn why thing described above is unrelated afaict. It looks to be just a side effect of how the why code is searching for the packages.

[jkrems]

Tried to find a nice way of doing this without adding an additional pass to propagate the visibility after the graph was walked once. Not sure if it would be acceptable to just split out the visibility into a separate step..?

[jkrems]

There's some interesting edge cases as well, it's not just about properly resolving visibility:

• A is an optional dependency of a production dependency
• B is a non-optional dependency of a dev dependency
• C is a non-optional dependency of both

In that case the optional flag of C depends on dev vs. prod. In dev it would be non-optional, in prod it would be optional. Just inheriting the optional flag from one of the parents (or always inheriting it from the prod parent) could lead to weirdness.

[SimenB]

This is still not fixed in 0.17.2 😢

Repro: https://gist.github.com/SimenB/2b179f3b6bca73ba824e1273ea38aed3

yarn

node index.js # works

yarn --prod

node index.js # explodes

/cc @jkrems

[beheh]

Doesn't seem fixed for me either in 0.17.2 (HearthSim/Joust#169).

[dashmug]

It is related. Probably the same cause. Just a different symptom. I would put that as a different issue since it's not exactly the same as what the title says.

[bestander]

@SimenB, thanks, I'll have a look

[adamreisnz]

Ok will do guys.

[SimenB]

@bestander Making a package wth just those 3 in the output that fails makes it reproducible on master for me with just public deps. It's not a minimal reproduce, but still

{
  "name": "app",
  "version": "1.0.0",
  "dependencies": {
    "brakes": "^2.5.1",
    "compression": "^1.6.2",
    "envalid": "^2.4.0",
    "express": "^4.14.0",
    "object.entries": "^1.0.4",
    "prom-client": "^7.0.0",
    "response-time": "^2.3.2",
    "spaden": "^7.13.1",
    "yarn-issue-repro-package": "^1.0.0"
  },
  "devDependencies": {
    "babel-preset-es2015": "^6.18.0",
    "browserify": "^13.1.1",
    "browserify-middleware": "^7.1.0",
    "cheerio": "^0.22.0",
    "codeceptjs": "^0.4.13",
    "del-cli": "^0.2.1",
    "eslint": "^3.12.2",
    "eslint-config-finn": "^1.0.1",
    "espower-loader": "^1.0.1",
    "hashmark": "^4.1.0",
    "interfake": "^1.19.0",
    "mocha": "^3.2.0",
    "nightmare": "^2.9.0",
    "nightmare-upload": "^0.1.1",
    "nock": "^9.0.2",
    "nodemon": "^1.11.0",
    "nyc": "^10.0.0",
    "power-assert": "^1.4.1",
    "sinon": "^1.17.6",
    "supertest": "^2.0.1",
    "uglify-js": "^2.7.5",
    "uglifyify": "^3.0.4"
  }
}

package.json of yarn-issue-repro-package

{
  "name": "yarn-issue-repro-package",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "nunjucks": "^2.5.2",
    "pretty-error": "^2.0.2",
    "unleash-client": "^1.0.0-beta.7"
  }
}

Interestingly, it produces 4 errors instead of 3...

$ yarn check v0.20.0-0                                                                                                   │├─ [email protected]
error "yarn-issue-repro-package#nunjucks#chokidar#anymatch" not installed                                              │└─ [email protected]
error "yarn-issue-repro-package#unleash-client#request#json-stringify-safe" not installed                              │✨  Done in 2.65s.
error "yarn-issue-repro-package#nunjucks#yargs#string-width#code-point-at" not installed                               │ ~/repos/yarn-issue-repro-package  vim package.json
error "yarn-issue-repro-package#pretty-error#renderkid#css-select#domutils#dom-serializer#entities" not installed      │ ~/repos/yarn-issue-repro-package  npm publish
error Found 4 errors.

[havenchyk]

I'm not sure is this the same issue or not, but here is the output (tested with 0.19.1)

[blexrob]

I think I have found the root cause for the installation issue.
The failure to install a package can be reproduced by the following package.json:

{ "dependencies": {
    "bcrypt": "^1.0.1",
    "gamepad": "1.4.2"
  },
  "devDependencies": {
     "istanbul": "^1.0.0-alpha.2"
  }
}

And then the commands

rm -rf node_modules yarn.lock
yarn install
rm -rf node_modules
yarn install --production
npm ls abbrev

In this configuration, abbrev is not installed.

abbrev is used by istanbul and by nopt (as can be seen from yarn why abbrev). nopt is used by istanbul and node-pre-gyp (which is used by bcrypt and gamepad).

When deduping abbrev in the package hoister, the following code is used to determine the new isIgnored function of the hoisting record:

          // switch to non ignored if earlier deduped version was ignored
          if (existing.isIgnored() && !info.isIgnored()) {
            existing.isIgnored = info.isIgnored;
          }

abbrev is one of the first hoisting records to be processed. At that moment, the existing record is istanbul#abbrev (ignored because istanbul is ignored), and the duplicate record is istanbul#nopt#abbrev, which is also ignored at the time for the same reason.

Because both records are ignored at the time, the ignore function isn't adjusted as it was supposed to - because nopt will become non-ignored in a later dedup because of the dependency by node-pre-gyp. The ignore status of both records can change at any time, so the new ignore function should mix them.

And indeed, the installation problem goes away when we replace those lines with

          // switch to non ignored if earlier deduped version was ignored
          if (existing.isIgnored()) {
            if (info.isIgnored()) {
              // both are ignored now, but any one could become non ignored later on.
              let oldIsIgnored = existing.isIgnored;
              existing.isIgnored = () => oldIsIgnored() && info.isIgnored();
            } else {
              existing.isIgnored = info.isIgnored;
            }
          }

[bestander]

@blexrob, great find!
Would you send a PR?
There is a disabled test for "should not lose dependencies when installing with --production" in integration.js that should be fixed now

[blexrob]

@bestander, just tested it, and this fix causes a stack overflow in the test you mentioned, so it can't be applied. The following cycle pops up:

d#es5-ext -> es6-symbol#es5-ext -> es6-set#es5-ext -> es6-iterator#es5-ext -> es6-map#es5-ext -> es5-ext#es6-iterator -> es6-set#es6-iterator -> es6-weak-map#es6-iterator -> event-emitter#es5-ext -> d#es5-ext

So, the naive recursive call approach is out...

[bestander]

Yeah, I believe it needs a bit tweaking but the idea seems correct

[Dzenly]

I have such a problem with phantomjs-prebuilt module (as dependence of dependence) with yarn 0.27.5-1.
So now I do dummy yarn add phantomjs-prebuilt, before yarn install --production.

[adamreisnz]

I regret to say that this still seems to be a problem in Yarn 1.3.2.
My builds on Netlify are failing when I use Yarn 1.3.2 but succeeding with Yarn 0.18.2.
The build errors with a cannot find module 'are-we-there-yet' and only with production flag.

[bestander]

@adamreisnz, this thread is too big to track all issues.
Could you raise a new one with repro script?

[adamreisnz]

@bestander done, thanks.

[olastor]

For someone who still can't get it work and doesn't want to install jq can use

$ python -c "import json; p = json.loads(open('package.json').read()); del p['devDependencies']; open('package.json', 'w').write(json.dumps(p, indent=2));"

[hannadrehman]

i am on yarn 1.17.3 and node v10.16.2 in lerna monorepo. still facing same issue.

[TAnas0]

I can confirm it too.
I have a lot of dependencies, but when I use yarn install --production, only two modules are installed.
Notable though, I am on a Lerna monorepo similar to @hannadrehman with Yarn workspaces, which may explain the extreme behavior.

Yarn version: 1.22.0
Node: v12.16.1

npm install --production works perfectly though.

@hannadrehman is the project in question a package of your monorepo?

[nolazybits]

Same problem as @TAnas0

[nocive]

define the following yarn script

"scripts": {
  (...) 
  "prune-dev-deps": "node -e \"p = JSON.parse(fs.readFileSync('package.json')); console.log(Object.keys(p.devDependencies).join('\\n'));\" | xargs -r yarn remove",
  (...)
}

run after building your app when you no longer need build dependencies

yarn install ...
yarn build
yarn run prune-dev-deps