fs.watch() doesn't notify content changes starting v10.16.0

[d4vidi]

• Version: v10.16.0
• Platform: Darwin abc-d-mac15-111.local 18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64
• Subsystem: fs

In versions 10.16.0 and newer, fs.watch() does not notify change events when watching a specific file -- at least not when fs.write() is used in either the running or a different process. The change event is triggered in version 10.15.3 and older in such a use case, nevertheless.

Simple reproduction code:

watch.js

Watches a file called test.txt using fs.watch():

const fs = require('fs');

const filename = 'test.txt';

const fd = fs.openSync(filename, 'w');

fs.watch(filename, {}, (event) => {
	console.log(event);
});

setTimeout(() => {
	fs.closeSync(fd);
	process.exit(0);
}, 300000);

write.js

Repeatedly writes text buffers onto test.txt:

const fs = require('fs');

const filename = 'test.txt';

async function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function run() {
	const fd = fs.openSync(filename, 'a');
	fs.writeSync(fd, '123\n')
	await sleep(1000)
	fs.writeSync(fd, '456\n')
	await sleep(1000)
	fs.writeSync(fd, '789\n')
	await sleep(1000)
	fs.writeSync(fd, 'abc\n')
	await sleep(1000)
	fs.writeSync(fd, 'def\n')
	await sleep(1000)
	fs.writeSync(fd, 'ghi\n')
	await sleep(1000)
	fs.closeSync(fd);
	
	process.exit(0);
}

run();

Final output using node 10.16.0:

$ node watch.js
change
change

(i.e. change is only reported when test.txt is created and upon fd closing).

Final output using node 10.15.3:

$ node watch.js
change
change
change
change
change
change

(i.e. reported per each write)

Notes:

• A tail -f test.txt running simultaneously vividly shows that the writes do take place properly even when the events are not reported by fs.watch().
• Appending content to the watched file through the shell (i.e. using echo something >> test.txt) does trigger a change event in node 10.16.0 or higher.

[bnoordhuis]

Thanks for the report. It's a duplicate of #27869 and one or two other issues so I'll go ahead and close this out.

I'm afraid a fix is hampered by the fact that it swaps one regression for another, see the linked issues for details.

[d4vidi]

@bnoordhuis thanks for the quick response. Nevertheless, I do not understand how this could be a duplication of #27869, as the latter refers to events reported by watch due to file deletion / renaming; Here I was focussing on files' content change.

[bnoordhuis]

Sorry, I should've explained. The reason the behavior changed is because Node.js / libuv switched to a completely different mechanism for watching files (kqueue vs. fsevents.)

[d4vidi]

I see - same root cause for both issues. Nevertheless to me the use case seems different and I'd rather keep it open, but suite yourself: I'll set myself as a watcher on the other issue as well 🙏

[bpasero]

@bnoordhuis as author of #27869 I am not sure I would agree this issue is related. Yes, the underlying source of the bug is the change of libuv to use fsevents, but then this bug should move to libuv? I think it is a perfectly valid regression that is easy to reproduce.

Btw when changing the code to use fs.writeFileSync without reusing the same file descriptor, I can no longer reproduce, so it seems related.

[d4vidi]

As a side note - which could possibly help readers out: sharing that I've switched to fs.watchFile() with a 1.5sec polling interval (reasonable compared to the rate of content changes I expect) -- things are now less efficient but stable throughout node.js versions.