Watchpack 2.0.0-beta.13 fails on large codebase with "too many open files" error

[marcins]

With Webpack 4 and Watchpack 1.x we avoided this error by applying a patch to Watchpack to use sane instead of chokidar, which in combination with Facebook's watchman worked for us.

However Webpack 5 + Watchpack 2 uses Node native fs.watch which seems to end up in the same "too many open files" issue as we had before with chokidar. After a build with Webpack dev server the console just starts repeating:

Watchpack Error (watcher): Error: EMFILE: too many open files, watch

There are known existing bugs with Node's fs.watch and many files: nodejs/node#29460

I can work around this by using poll instead for testing - but that's not sustainable due to the CPU overhead.

Would there be any interest in supporting watchman as a watcher "natively" to avoid another dependency on something like sane? If not, what would your recommendation be for large projects?

[sokra]

Which OS? If you are on MacOS, the max file limit it very low by default and we recommend to increase it (watchman does it automatically).

ulimit -n prints the current limit.

There are known existing bugs with Node's fs.watch and many files: nodejs/node#29460

All the linked issues are closed. Which bugs are you talking about exactly?

[marcins]

Yep, aware of the ulimit issue. With Webpack 4 / Watchpack 1 we tried a bunch of stuff to get it working reliably (we had the (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21) error on our codebase). Eventually switching over to sane/watchman resolved the issue for us.

I've dug in further by adding additional logging to watchpack and it seems that fs.watch (and doScan(true)) is getting called individually for every directory in the project - some 15,953 times each. That might explain the issue - fs.watch on OSX uses FSEvents so should be able to work recursively from the top level instead of needing to watch individual directories. I will investigate further.

[marcins]

It's definitely an issue with the way Watchpack installs watchers. When I created a simple test script that'll call fs.watch for every directory in our src folder without the recursive option (like Webpack/Watchpack is doing) I get the same "too many open files" issue:

Installed 16862 watchers
Error: EMFILE: too many open files, watch
    at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:123:28)
Emitted 'error' event on FSWatcher instance at:
    at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:129:12) {
  errno: -24,
  syscall: 'watch',
  code: 'EMFILE',
  filename: null
}

This is with a ulimit -n of 200,000 - but only ~16,000 directories (and ~40,000 files). For what it's worth, if you use fs.watch(path.resolve('.'), { recursive: true } it works without any issues, even with a default ulimit - but the recursive option is only supported on Windows and macOS, not Linux.

I will try and put together a minimal reproduction today with a similar setup if that helps. I am also trying to find out from a co-worker if they get the same issue on Linux with the same repro script I'm using.

Edit: interestingly some further digging shows that it falls over if more that 4,096 fs.watch calls are executed. This doesn't match any of the limits in ulimit -a, but seems like a nice even number that must be a limit somewhere, so is worth digging into. Also running the same script on Linux does work, seems it's OK to run fs.watch 16,000+ times on that platform (even with a smaller max open files limit - my colleague had 1024).

[marcins]

OK, here's a repro which fails on macOS (not just my machine, I've tried it with some colleagues) regardless of how high (or low) you set the max open files limit: https://github.com/marcins/webpack-fs-watch-error-repro (FYI @sokra)

(I had a colleague with the default limit of 256 try it - works with 4000 dirs, fails with 4096, same as with 10k or 200k max open files limit)

[sokra]

Seems like it's using some "other" limit we are not aware of. I also found launchctl limit maxfiles, maybe you can try printing that limit.

I think a good solution is to merge watchers to recursive watchers when supported until we stay within a limit of watchers (I think 500 would be fine, so you can technically run 8 processes with watchpack). As it seems like only osx is the problem and it supports recursive watchers that should work fine.

I want to avoid recursive watchers by default as they 1. are not always supported, 2. watch too much and we don't know how deep your directories are, which could have performance implications.

[marcins]

launchctl limit maxfiles is definitely related to the ulimit -n limit.. you can't set ulimit -n xxx to something greater than that value - so before setting it to 200k I had to update the maxfiles limit. It doesn't seem to have any effect on whatever limit the registration of multiple watchers is hitting though - as you say, there's some other limit at play here. I had a quick look at the FSEventStream docs from Apple and the NodeJS source for fs.watch but could not find any references to what this limit might be. I did however find the underlying error in the Console which matches errors folks have seen in the past (pretty much every search result for that error is related to the JS ecosystem - but there's no solutions, usually people end up doing something that as a side-effect presumably reduces the number of watchers):

FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-22)

(I think 500 would be fine, so you can technically run 8 processes with watchpack

That might be something worth testing, whether this is a global or per-process limit.

I want to avoid recursive watchers by default as they 1. are not always supported, 2. watch too much and we don't know how deep your directories are, which could have performance implications.

Yep, definitely wouldn't be a default - but they are supported on Windows & macOS (according to the fs.watch documentation) which probably cover a fair amount of users. I'm guessing the main performance implication of watching too much is potentially saturating the event loop with change events if you end up watching a directory that's subject to churn and not necessarily part of the codebase. For what it's worth, @parcel/watcher implements the cross-platform file watching part in C++ native code and normalises / aggregates / throttles events back to JavaScript - so an approach like that might also avoid this.

Without only a surface understanding of how Webpack / Watchpack code interacts (though I have dug through it a little bit), I can imagine that it should be possible to process the list of files to be watched and narrow down to a set of common roots that will cover everything needed - though I guess the problem in the real world is all the different and wonderful way people set up their projects might mean that common root is higher in the filesystem than you might want it. As you suggest though, potentially you don't need to go all the way to common roots, just enough to stay under the limit.

This would definitely be a blocker for adoption of Webpack 5 for us (though at the moment we're just evaluating the work required to migrate). I will likely unblock myself for now by hacking in a recursive directory watch for the project root which should be enough for my evaluation.

[marcins]

This is interesting, it looks like there was a proposed changed many years ago to implement a single FSEventStream and filter in libuv however it never got merged: joyent/libuv#894

It also refers to a total limit of FSEventStreams on macOS (with not much other detail)

[sokra]

I think I fixed the issue.

I added a merging algorithm which merges watchers into recursive watchers when reaching the watcher limit.

I wrote a test and got the same results as you, only OSX has this limit of watchers. The test passes now with the new algorithm.

[oalexdoda]

Hey @sokra, I think the issue is still not fixed in Webpack 5. We described it here: martpie/next-transpile-modules#236

The only thing that helps is setting this in our Next config file:

webpack: (config) => {
        config.watchOptions = {
            poll: 10000,
        };

        return config;
},

The project has thousands of files, possibly crossing 10.000 watched files. Using macOS, Webpack 5, latest Next.js, Next Transpile.

Thank you

[php4fan]

I'm facing this issue on linux when using Angular and ng serve

Steps to reproduce:

$ npm install -g @angular/cli
$ ng new my-project
$ cd my-project
$ ng serve

and the terminal gets inundated with these "too many open files" errors.

I have tried running

ulimit -m 100000

before it, but the error persists

[php4fan]

Why is this closed??

[GeeWee]

This does not seem resolved to me either.

[gkfirst8]

We still bump into this issue using Angular 13 & 14. So I hope it can be re-opened?

[stevenlafl]

This shouldn't be closed. @php4fan identical steps, same issue.

Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/app/seven8/frontend/node_modules/typescript/lib'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/app/seven8/frontend/node_modules/typescript'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/app/seven8/frontend/node_modules/typescript/lib'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/app/seven8/frontend/node_modules/typescript'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/app/seven8/frontend/node_modules/typescript/lib'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/app/seven8/frontend/node_modules/typescript'
...