FSEvents-related SIGBUS crashes on macOS 10.15.7 #37697
Comments
|
Additional data point that may be helpful. I'm seeing 24,680 bytes on the stack from Most of this is Interestingly 24,680 is still less than |
Hm. The 25 KB figure I gave in the last comment seems to be correct. |
|
Just discovered that ____chkstk_darwin: // %rax == alloca size
pushq %rcx
leaq 0x10(%rsp), %rcx
// validate that the frame pointer is on our stack (no alt stack)
cmpq %rcx, %gs:_PTHREAD_STRUCT_DIRECT_STACKADDR_OFFSET
jb Lprobe
cmpq %rcx, %gs:_PTHREAD_STRUCT_DIRECT_STACKBOTTOM_OFFSET
jae Lprobe
// validate alloca size
subq %rax, %rcx
jb Lcrash
cmpq %rcx, %gs:_PTHREAD_STRUCT_DIRECT_STACKBOTTOM_OFFSET
ja Lcrash
popq %rcx
retqThe process above seems to be crashing on an allocation size of
Going to look into what the |
|
I think I've gotten to the bottom of this crash. The Psuedo-code for the problematic block looks like this: void implementation_callback_rpc(char** eventPaths, long nEventPaths) {
// ...
char** strippedEventPaths = alloca(nEventPaths)
for (int i = 0; i < nEventPaths; i++) {
strippedEventPaths[i] = caller_path(eventPaths[i])
}
// ...
}The So what does char* caller_path(char* eventPath) {
if (startsWith(eventPath, "/System/Volumes/Data/") {
// ...
return eventPath + strlen("/System/Volumes/Data/")
}
// ...
return eventPath;
}So it basically just strips a constant prefix path. I was curious if there's any significance to
Which is fascinating. Multiple coworkers reported that this Node.js crash only started happening after upgrading to 10.15/Catalina. I bet this stack allocation loop isn't present in 10.14 since * Technically Variable-Length Arrays (VLAs) and |
Fixes nodejs#37697. This fixes `SIGBUS` crashes on macOS 10.15 due to FSEvents code attempting to allocate large arrays on the stack. The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once.
Fixes nodejs#37697. This fixes `SIGBUS` crashes on macOS 10.15 due to FSEvents code attempting to allocate large arrays on the stack. The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once.
Fixes nodejs#37697. This fixes `SIGBUS` crashes on macOS 10.15 due to FSEvents code attempting to allocate large arrays on the stack. The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once.
This fixes `SIGBUS` crashes on macOS 10.15 due to FSEvents code attempting to allocate large arrays on the stack. The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once. Fixes: nodejs#37697 Refs: joyent/libuv#964
This fixes `SIGBUS` crashes on macOS 10.15 due to FSEvents code attempting to allocate large arrays on the stack. The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once. Removing the manual `pthread_attr_setstacksize` call here defaults to a stack size of 512KB. Fixes: nodejs/node#37697 Refs: joyent/libuv#964
This fixes `SIGBUS` crashes on macOS 10.15 due to FSEvents code attempting to allocate large arrays on the stack. The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once. Removing the manual `pthread_attr_setstacksize` call here defaults to a stack size of 512KB. Fixes: nodejs/node#37697 Refs: joyent/libuv#964
This fixes `SIGBUS` crashes on macOS 10.15 due to FSEvents code attempting to allocate large arrays on the stack. The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once. Removing the manual `pthread_attr_setstacksize` call here defaults to a stack size of 512KB. Fixes: nodejs/node#37697 Refs: joyent/libuv#964
This fixes `SIGBUS` crashes on macOS 10.15 due to FSEvents code attempting to allocate large arrays on the stack. The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once. Removing the manual `pthread_attr_setstacksize` call here defaults to a stack size of 512KB. Fixes: nodejs/node#37697 Refs: joyent/libuv#964
This fixes `SIGBUS` crashes on macOS 10.15 due to an FSEvents change that causes it to allocate large arrays on the stack. The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once. Removing the manual `pthread_attr_setstacksize` call here defaults to a stack size of 512KB. Fixes: nodejs/node#37697 Refs: joyent/libuv#964
This fixes `SIGBUS` crashes on macOS 10.15 due a new change in `FSEvents.framework` that makes it allocate a large stack array for event paths. (See the linked nodejs/node issue for details on the `FSEvents.framework` memory requirements change itself.) The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once. Setting this to `uv__thread_stack_size()` increases it to 8192KB (by default) on 64-bit machines. This value can be configured at runtime on macOS with `ulimit -s <size-kb>`. The 32KB limit was originally added to reduce virtual memory fragmentation on 32-bit systems, which is not a concern on 64-bit systems. Fixes: nodejs/node#37697 Refs: joyent/libuv#964
This fixes `SIGBUS` crashes on macOS 10.15 due a new change in `FSEvents.framework` that makes it allocate a large stack array for event paths. (See the linked nodejs/node issue for details on the `FSEvents.framework` memory requirements change itself.) The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once. Setting this to `uv__thread_stack_size()` increases it to 8192KB (by default) on 64-bit machines. This value can be configured at runtime on macOS with `ulimit -s <size-kb>`. The 32KB limit was originally added to reduce virtual memory fragmentation on 32-bit systems, which is not a concern on 64-bit systems. Fixes: nodejs/node#37697 Refs: joyent/libuv#964
This fixes `SIGBUS` crashes on macOS 10.15 due a new change in `FSEvents.framework` that makes it allocate a large stack array for event paths. (See the linked nodejs/node issue for details on the `FSEvents.framework` memory requirements change itself.) The existing size (`4 * PTHREAD_STACK_MIN` or 32KB) causes a stack overflow when more than ~1000 events are received at once. Setting this to `uv__thread_stack_size()` increases it to 8192KB (by default) on 64-bit machines. This value can be configured at runtime on macOS with `ulimit -s <size-kb>`. The 32KB limit was originally added to reduce virtual memory fragmentation on 32-bit systems, which is not a concern on 64-bit systems. Fixes: nodejs/node#37697 Refs: joyent/libuv#964 PR-URL: libuv#3132 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Santiago Gimeno <[email protected]>
fs: https://github.com/nodejs/node/blob/227ad25/deps/uv/src/unix/fsevents.cWhat steps will reproduce the bug?
I'm running the TypeScript compiler in watch mode against a large proprietary monorepo. TypeScript is configured with
TSC_NONPOLLING_WATCHER, so it uses FSEvents to watch for source code changes.Unfortunately I haven't reproduced this in a more isolated setting yet without proprietary source code. I'm currently working on that and will update this issue as soon as one is available. I opted to create this issue early so others have a place to comment if they see similar problems.
How often does it reproduce? Is there a required condition?
I can consistently reproduce this in
tscwith the aforementioned proprietary repo:After TypeScript sets up FSEvent watchers, making a change to any source code file (with
touch) triggers:I apologize for the non-actionable repro for now. Although this is primarily observable through TypeScript, I do believe this is a Node.js bug. See Additional information below.
What is the expected behavior?
Node.js does not crash.
What do you see instead?
Here's a crash report summary from
Console.app.Additional information
From the stack trace, it looks like the
pthreadresponsible for handling FSEvents notifications from macOS is overflowing its stack, which triggerskernel-level stack canary checks. (Edit: This was actually a more simple stack bounds check. See comments below.) I noticed that this is similar to an issue previously seen on Node.js 0.10.18 and OS X 10.9.nodejs/node-v0.x-archive#6251
The fix from back then was to simply increase the stack size.
joyent/libuv#964
The limit hasn't changed since:
node/deps/uv/src/unix/fsevents.c
Lines 652 to 654 in 4947ce5
I recompiled Node.js with
5 * PTHREAD_STACK_MINand still saw crashes.8 * PTHREAD_STACK_MINeliminated the problem though. Removing thepthread_attr_setstacksizecall entirely (so the default macOSpthreadstack size is used) also fix the issue.I'm hesitant to suggest either of those changes as a real fix since this appears to be dependent on how many
fs.watchcalls are made.Other related issues
nodejs/help#3168 looks similar, but was due to an issue with the
fseventsnpm module. This issue is happening in Node.js core. (Specificallyfs.)I found libuv/libuv#669 interesting as well, which changed
libuvworker threads on macOS to useRLIMIT_STACK.The text was updated successfully, but these errors were encountered: