doesn't support `$HOME/.node_modules` or `$HOME/.node_libraries` #163

mk-pmb · 2018-06-02T23:15:06Z

Hi, thanks for this module!
It seems to have trouble finding my packages, though:

$ readlink -m ~/.node_modules
/mnt/…/nodejs/modules
$ nodejs -e 'console.log(require.resolve("lodash")); require("resolve")("lodash", console.log);'
/mnt/…/nodejs/modules/lodash/lodash.js
{ Error: Cannot find module 'lodash' from '.'
    at /mnt/…/nodejs/modules/resolve/lib/async.js:50:31
    at processDirs (/mnt/…/nodejs/modules/resolve/lib/async.js:184:39)
    at ondir (/mnt/…/nodejs/modules/resolve/lib/async.js:199:13)
    at load (/mnt/…/nodejs/modules/resolve/lib/async.js:82:43)
    at onex (/mnt/…/nodejs/modules/resolve/lib/async.js:107:17)
    at /mnt/…/nodejs/modules/resolve/lib/async.js:12:69
    at FSReqWrap.oncomplete (fs.js:152:21) code: 'MODULE_NOT_FOUND' }

So require.resolve was able to find it, but resolve wasn't. However, it works when I create a symlink /mnt/…/nodejs/node_modules to modules.

Update: Work-around: It also works when I pass option { paths: [process.env.HOME + "/.node_modules"] }
Update 2: However, in that case I have to then explicitly resolve the returned path in order to get the exact same result.

The text was updated successfully, but these errors were encountered:

ljharb · 2018-06-02T23:37:14Z

I think you'll need to set the preserveSymlinks option to true.

mk-pmb · 2018-06-03T00:15:08Z

Nope, doesn't seem to help:

$ nodejs -e 'console.log(require.resolve("lodash")); require("resolve")("lodash", { preserveSymlinks: true }, console.log);' 
/mnt/…/nodejs/modules/lodash/lodash.js
{ Error: Cannot find module 'lodash' from '.'
[…]

mk-pmb · 2018-06-03T00:52:43Z

To clarify, with "explicitly resolve the returned path" I meant I have to fs.realpath() it myself.

ljharb · 2018-06-17T07:07:24Z

(Possibly related to #122)

safizn · 2018-08-26T08:55:22Z

preserveSymlinks doesn't seem to work on Windows Symbolic link or Junction. Is there a problem with the implementation in the module ?
It never resolves to the real path, which what I needed from this module anyway. But the options preserveSymlinks: false, should work.
There seems to be a bug.

ljharb · 2018-08-26T17:05:02Z

@myuseringithub does node itself work with windows symlinks?

mk-pmb · 2018-08-26T18:46:41Z

I reproduced the problem and took an strace capture. The diff shows what really happened vs. what I'd expect to happen:

-stat64("/mnt/…/nodejs/modules/node_modules/lodash/package.json", …
+stat64("/mnt/…/nodejs/modules/lodash/package.json", …
 stat64("/mnt/…/nodejs/node_modules/lodash/package.json", …
 stat64("/mnt/…/node_modules/lodash/package.json", …
 stat64("/mnt/node_modules/lodash/package.json", …
 stat64("/node_modules/lodash/package.json", …

It seems resolve searches for modules in module directories inside of module search path directories, but not inside the module search path directories themselves.

safizn · 2018-08-27T10:37:44Z

@ljharb Yes, I use symbolic link of type Junction, as it allows me to create symlinks that are supported between the host machine (Windows OS), and the Docker container (Linux OS) which I'm running the app on. In my setup both the host machine and container use JS scripts to run the project.
Node's require.resolve('<a symlink>') returns the real path of the symlink as Node's command option --reserve-symlink is set to false by default.

ljharb · 2022-01-03T17:08:38Z

I'm unclear on what needs to be done here. Could anyone provide a repro repo, or better, a failing test case?

mk-pmb · 2022-01-09T04:20:54Z

Bug demo in GitLab Actions: https://github.com/mk-pmb/browserify-resolve-symlink-163

ljharb · 2022-01-11T18:16:12Z

@mk-pmb thanks - is there any way to repro it on the command line? When i run npm test I get:

> [email protected] test
> bash demo.sh

readlink: illegal option -- m
usage: readlink [-n] [file ...]
demo.sh: line 11: cd: /prepare: No such file or directory

ljharb · 2022-01-11T18:19:12Z

@mk-pmb ah! resolve 1 has preserveSymlinks: true set by default. resolve 2 has preserveSymlinks: false set by default.

Try setting preserveSymlinks: false explicitly (or try a resolve 2 prerelease).

mk-pmb · 2022-01-11T20:37:27Z

is there any way to repro it on the command line?

Yes, if you run demo.sh in an environment similar enough to the GitHub Actions env.

readlink: illegal option -- m

You'll need GNU coreutils 8.21 or later installed. Ubuntu ships it by default, so I use that in the GitHub Actions.

Try setting preserveSymlinks: false explicitly (or try a resolve 2 prerelease).

Done in the no-preserve-symlinks branch, results are in.

ljharb · 2022-01-11T21:42:53Z

pify?
41    node.js resolver: /…/browserify-resolve-symlink-163/mnt/projects/webdev/js/modules/pify/index.js
42    "resolve" module: /…/browserify-resolve-symlink-163/mnt/projects/webdev/js/modules/pify/index.js

Seems like it's good! Does that mean this can be closed?

mk-pmb · 2022-01-11T23:18:24Z

The first two tests are the part that had been easy to fix ever since. The bug in my opening post is reproduced in the 3rd test case, and it still fails:

global-module-163?
    node.js resolver: /…/browserify-resolve-symlink-163/mnt/projects/webdev/js/secret_modules/global-module-163/dummy.js
    "resolve" module: Error: Cannot find module 'global-module-163' from '/home/runner/work/browserify-resolve-symlink-163/browserify-resolve-symlink-163/home/demo/my-cool-app'

mk-pmb · 2022-01-11T23:19:42Z

I see now why you thought it was about the pify path. I'll clarify the tutorial in the demo readme.

ljharb · 2022-01-12T00:34:05Z

the words "global module" concern me a bit; global modules aren't requireable or resolveable (unless you use the deprecated NODE_PATHS env var, which this package intentionally doesn't support and likely never will). Can you elaborate on that one?

mk-pmb · 2022-01-12T01:45:05Z

I'm not sure if my naming of "global" in the example is correct. I'm also not up-to-date on the deprecation. If you can determine the cause and conclude that the difference in the 3rd example is out of project scope, we shall instead check whether the readme could use better info about why.

mk-pmb · 2022-01-12T01:48:50Z

However, what I can provide, is to say I didn't write anything about NODE_PATHS in the example repo. Also git grep -nFe NODE_PATH came back empty, so it hasn't crept in.

ljharb · 2022-01-12T02:26:21Z

In the no-preserve-symlinks, I'm able to cd into home/demo/my-cool-app and run node app.js, and require.resolve itself can't find global-module-163 - which makes sense, since there's no node_modules/global-module-163 present, in that directory or any above it. What do you expect will happen?

mk-pmb · 2022-01-15T02:40:39Z

I expect it will consider ~/.node_modules at some stage, probably after ~/node_modules.

ljharb · 2022-01-15T04:58:04Z

@mk-pmb it doesn't and shouldn't, no - .node_modules is never a thing.

Either way, in your repro repo there's no global-module-163 anywhere, so it's supposed to fail to resolve, best I can tell.

mk-pmb · 2022-01-15T06:23:18Z

What do you mean by ".node_modules is never a thing."? It's still in current node master: lib/internal/modules/cjs/loader.js

  if (homeDir) {
    ArrayPrototypeUnshift(paths, path.resolve(homeDir, '.node_libraries'));
    ArrayPrototypeUnshift(paths, path.resolve(homeDir, '.node_modules'));
  }

Which explains why on GitHub Actions, node.js can successfully resolve all examples. (See logs, or example output in readme).

global-module-163?
    node.js resolver: /…/browserify-resolve-symlink-163/mnt/projects/webdev/js/secret_modules/global-module-163/dummy.js

Either way, in your repro repo there's no global-module-163 anywhere,

The resolving chain is (all inside repo):

~/.node_modules = home/demo/.node_modules = symlink to ../../mnt/projects/webdev/js/modules
→ mnt/projects/webdev/js/modules contains symlink global-module-163 to ../secret_modules/global-module-163
→ mnt/projects/webdev/js/secret_modules/global-module-163 contains package.json

ljharb · 2022-01-15T16:24:42Z

whoa, that was not a thing I was aware of.

resolve has no knowledge of those paths and isn't set up to use them, and I'm not sure why anybody else would. $HOME shouldn't have any node_modules.

ljharb · 2022-01-15T16:33:59Z

In other words, this issue has nothing to do with symlinks - it's about some kind of special $HOME/.node_modules path that nobody's ever mentioned in a decade of this package not supporting it :-) I'll update the title to reflect that.

(note that "global" means npm root -g; this home dir thing is something else)

ljharb · 2022-01-15T16:36:28Z

For history's sake: nodejs/node@5a49f96#diff-cdd77182e73349d6c83fbbce1cccf6f15972fbece4dbd9c52eececf1ca3bfcd5R307-R308 is in every node version since basically ever.

mk-pmb · 2022-01-16T18:38:01Z

I updated my test project;

Fix the naming of the module to be resolved via ~/.node_modules
Add test for ~/.node_libraries
~~Add test for an actual global module~~

Update: Today I learned that my global modules being require()-able system-wide is just accidential. The node design concept seems to understand npm --global as related only to modules' CLI commands, not the modules themselves.

ljharb · 2022-01-16T21:10:43Z

indeed; global things aren’t supposed to be requireable.

ljharb · 2022-01-18T21:42:14Z

The current workaround for this is to pass [$HOME/.node_libraries, $HOME/.node_modules] as the paths option.

…s,libraries}` Note, this is a rarely used feature that should be aggressively avoided, but it‘s important to minimize gaps between node and this package. Fixes #163

ljharb · 2022-01-21T20:33:13Z

@mk-pmb can you confirm that #273 works for you? (or the equivalent change on 1.x)

…s,libraries}` Note, this is a rarely used feature that should be aggressively avoided, but it‘s important to minimize gaps between node and this package. Fixes #163

…s,libraries}` (#273) Note, this is a rarely used feature that should be aggressively avoided, but it‘s important to minimize gaps between node and this package. Fixes #163

malash mentioned this issue Nov 27, 2018

Resolve doesn't resolve real path for basedir if it is relative path #177

Closed

ljharb changed the title ~~Doesn't find modules behind symlinks?~~ doesn't support $HOME/.node_modules Jan 15, 2022

ljharb changed the title ~~doesn't support $HOME/.node_modules~~ doesn't support $HOME/.node_modules or $HOME/.node_libraries Jan 15, 2022

ljharb mentioned this issue Jan 21, 2022

[New] add default support for paths to include $HOME/.node_{modules,libraries} #273

Merged

ljharb closed this as completed in 66066fd Jan 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doesn't support `$HOME/.node_modules` or `$HOME/.node_libraries` #163

doesn't support `$HOME/.node_modules` or `$HOME/.node_libraries` #163

mk-pmb commented Jun 2, 2018 •

edited

Loading

ljharb commented Jun 2, 2018

mk-pmb commented Jun 3, 2018 •

edited

Loading

mk-pmb commented Jun 3, 2018

ljharb commented Jun 17, 2018

safizn commented Aug 26, 2018 •

edited

Loading

ljharb commented Aug 26, 2018

mk-pmb commented Aug 26, 2018

safizn commented Aug 27, 2018 •

edited

Loading

ljharb commented Jan 3, 2022

mk-pmb commented Jan 9, 2022

ljharb commented Jan 11, 2022 •

edited

Loading

ljharb commented Jan 11, 2022

mk-pmb commented Jan 11, 2022 •

edited

Loading

ljharb commented Jan 11, 2022 •

edited

Loading

mk-pmb commented Jan 11, 2022

mk-pmb commented Jan 11, 2022

ljharb commented Jan 12, 2022

mk-pmb commented Jan 12, 2022 •

edited

Loading

mk-pmb commented Jan 12, 2022

ljharb commented Jan 12, 2022 •

edited

Loading

mk-pmb commented Jan 15, 2022

ljharb commented Jan 15, 2022

mk-pmb commented Jan 15, 2022

ljharb commented Jan 15, 2022

ljharb commented Jan 15, 2022

ljharb commented Jan 15, 2022

mk-pmb commented Jan 16, 2022 •

edited

Loading

ljharb commented Jan 16, 2022

ljharb commented Jan 18, 2022

ljharb commented Jan 21, 2022

doesn't support $HOME/.node_modules or $HOME/.node_libraries #163

doesn't support $HOME/.node_modules or $HOME/.node_libraries #163

Comments

mk-pmb commented Jun 2, 2018 • edited Loading

ljharb commented Jun 2, 2018

mk-pmb commented Jun 3, 2018 • edited Loading

mk-pmb commented Jun 3, 2018

ljharb commented Jun 17, 2018

safizn commented Aug 26, 2018 • edited Loading

ljharb commented Aug 26, 2018

mk-pmb commented Aug 26, 2018

safizn commented Aug 27, 2018 • edited Loading

ljharb commented Jan 3, 2022

mk-pmb commented Jan 9, 2022

ljharb commented Jan 11, 2022 • edited Loading

ljharb commented Jan 11, 2022

mk-pmb commented Jan 11, 2022 • edited Loading

ljharb commented Jan 11, 2022 • edited Loading

mk-pmb commented Jan 11, 2022

mk-pmb commented Jan 11, 2022

ljharb commented Jan 12, 2022

mk-pmb commented Jan 12, 2022 • edited Loading

mk-pmb commented Jan 12, 2022

ljharb commented Jan 12, 2022 • edited Loading

mk-pmb commented Jan 15, 2022

ljharb commented Jan 15, 2022

mk-pmb commented Jan 15, 2022

ljharb commented Jan 15, 2022

ljharb commented Jan 15, 2022

ljharb commented Jan 15, 2022

mk-pmb commented Jan 16, 2022 • edited Loading

ljharb commented Jan 16, 2022

ljharb commented Jan 18, 2022

ljharb commented Jan 21, 2022

doesn't support `$HOME/.node_modules` or `$HOME/.node_libraries` #163

doesn't support `$HOME/.node_modules` or `$HOME/.node_libraries` #163

mk-pmb commented Jun 2, 2018 •

edited

Loading

mk-pmb commented Jun 3, 2018 •

edited

Loading

safizn commented Aug 26, 2018 •

edited

Loading

safizn commented Aug 27, 2018 •

edited

Loading

ljharb commented Jan 11, 2022 •

edited

Loading

mk-pmb commented Jan 11, 2022 •

edited

Loading

ljharb commented Jan 11, 2022 •

edited

Loading

mk-pmb commented Jan 12, 2022 •

edited

Loading

ljharb commented Jan 12, 2022 •

edited

Loading

mk-pmb commented Jan 16, 2022 •

edited

Loading