Module loading implementation does not match API docs

[yortus]

The API Documentation for modules states:

If the module identifier passed to require() is not a native module, and does not begin with '/', '../', or './', then Node.js starts at the parent directory of the current module, and adds /node_modules, and attempts to load the module from that location.

If it is not found there, then it moves to the parent directory, and so on, until the root of the file system is reached.

In the given example, a call to require('bar.js') inside a file at '/home/ry/projects/foo.js' would search the following locations:

/home/ry/projects/node_modules/bar.js
/home/ry/node_modules/bar.js
/home/node_modules/bar.js
/node_modules/bar.js

The documented lookup procedure makes no qualifications about the names of parent directories, so the example's ry and projects directory names are presumably arbitrary. But if we change ry to 'node_modules' then the lookup is materially different:

/home/node_modules/projects/node_modules/bar.js
/home/node_modules/bar.js
/node_modules/bar.js

Note that the location '/home/node_modules/node_modules/bar.js' is not considered.

The API documentation seems to diverge slightly from the implementation here. Perhaps one should be amended to match the other?

Example

This is just for the curious; the point of this issue is that the module loading documentation and implementation diverge in this case.

This came up in a discussion about an undesirable consequence of npm3's flattening behaviour, namely that it makes it easy to ship buggy code that forgets to declare some dependencies in its package.json, but still works most of the time because one or more other dependencies require the missing dependency so it is installed at the top level anyway (due to flat npm install).

For example, suppose a module at /home/foo declares a dependency on fs-extra. Later on, it starts using rimraf as well but forgets to add it as a dependency. However it just works (usually) because fs-extra depends on rimraf and so npm3 (usually) installs it at '/home/foo/node_modules/rimraf'. Note the usually because the behaviour is also a function of other factors.

A suggestion to catch this bug early was to put indirect dependencies one level deeper. So the example would look like:

/home/foo/package.json
/home/foo/index.js
/home/foo/node_modules/fs-extra
/home/foo/node_modules/node_modules/rimraf

According to the modules API documentation, the expected behaviour is that fs-extra can successfully require('rimraf'), but if foo tries to require('rimraf') it fails.

The actual behaviour is that require('rimraf') fails both for fs-extra and for foo.

[targos]

The documentation was correct at the beginning see this commit but changed at some point.

[yortus]

@targos good find.

The older documentation describes this case as an 'optimization', although that's a questionable label given that it materially changes the API's contract.

[Fishrock123]

We'll probably need to update the docs. Whatever changed, we probably can't change back.

[cjihrig]

Let's just add back the original:

Node will not append node_modules to a path already ending in node_modules.

[cjihrig]

PR: #3920