From fbf3759dabf21c478d1fc97bb2c6f63824cb5689 Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 14:32:19 +0100 Subject: [PATCH 01/11] feat(native): add backend trade-offs to Advanced Usage --- .../backend-tradeoffs/index.mdx | 68 +++++++++++++++++++ .../native/configuration/backends/index.mdx | 15 ++-- 2 files changed, 77 insertions(+), 6 deletions(-) create mode 100644 docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx new file mode 100644 index 0000000000000..524f5c13ff3be --- /dev/null +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -0,0 +1,68 @@ +--- +title: Backend Tradeoffs +description: "How to choose the right backend in the Native SDK." +sidebar_order: 1000 +--- +The Native SDK lets users decide at compile-time between three backends: + +* `crashpad` +* `breakpad` +* `inproc` + +Currently, `crashpad` is the default on all platforms because it + +* has an external `handler` process that allows for external snapshots and immediate sending of crash reports (instead of the next successful start of your application) +* is the primary target for extension compared to upstream, including + * client-side stack traces + * attachment handling + * HTTP proxy support + * CMake build scripts + * GCC and MinGW support + * `FirstChanceHandler` on Windows and extension of its synchronization to support sentry hooks + * cooperation with Epic's Easy Anti Cheat +* supports more error types on [Linux](/platforms/native/advanced-usage/signal-handling/#signals-of-interest) and Windows (`abort()` and other `fast-fail` crashes, handling of heap corruptions) +* is more maintained upstream (although most changes affect new platforms like Fuchsia) + +### When shouldn't I use the `crashpad` backend? + +Sentry decided on `crashpad` as the default on all platforms because there are a lot of upsides. However, there are use cases where `crashpad` cannot be used or makes distribution or deployment much harder. We provide other backends for situations when + +* you cannot package or deploy an additional executable (the `crashpad_handler`) +* you cannot allow a secondary process to connect via `ptrace` to your application (AWS Lambda, Flatpak-, Snap-Sandboxes) +* IPC between your process and the `crashpad_handler` is inhibited by security settings or not available in your deployment target +* your deployment scenario cannot wait for the `crashpad_handler` to finish its work before a shutdown-after-crash (systemd, Docker) +* you want to distribute your application via the macOS App Store + +In the above cases, if you cannot loosen the requirements of your environment, you have to choose an in-process backend (meaning either `breakpad` or `inproc`). + +### How do I decide between `breakpad` or `inproc`? + +Both backends are comparable in how they differ from `crashpad`. However, there are also considerable differences between the two: + +* `inproc` only provides the backtrace of the crashing thread. `breakpad` records all threads in the minidump. +* similar to `crashpad`, `breakpad` uses the lowest level error handling mechanism on each platform (macOS: mach exception ports, Windows: `UnhandledExceptionFilter`, Linux: signal handlers), it does cover a smaller range of errors though as mentioned above. +* `inproc`, on the other hand, uses signal handling on Linux and macOS (meaning you only get a `POSIX` compatibility layer over mach exception ports) and `UnhandledExceptionFilter` on Windows (solely errors registered by SEH) +* as a result of choosing signal handling on macOS, `inproc` is currently broken on macOS since Apple eliminated unwinding from signal handlers +* `inproc` is exceptionally lightweight and written entirely in C, meaning it does not rely on a weighty C++ runtime library, which is also more often affected by ABI incompatibilities +* `breakpad` generates minidumps (like `crashpad` does), whereas `inproc` follows the sentry event structure and primarily defers to the OS-provided unwinder and symbolication capabilities. Sentry can potentially extract more information from the provided minidump than simply a stack trace and registers. However, it also means that the crash context inside the minidump will be opaque until processed in the backend, whereas a local `relay` instance could process an entire `inproc`-event if required. + +### So when do I choose `inproc`? + +`inproc` is currently the backend of choice for `Android` because it allows us to couple it with our own fork of a powerful platform unwinder `libunwindstack` (rather than relying on a user-space interface). This allows us to support very old Android versions. + +`inproc` is the right choice if you want + +* want minimal dependencies +* want the smallest footprint for the resulting artifact +* don't need to support the latest macOS versions +* find the minimal featureset compared to `breakpad` and `crashpad` sufficient for your scenario + +### Summary + +There are many trade-offs in the selection of backends if you dive into the details. The above merely scratches the surface. Sentry suggests a sequence of evaluations like + +* `crashpad` (default) +* `breakpad` +* `inproc` + +from most feature-complete to least, where a step down should only be triggered by environmental inhibitors. With the above you now have exemplary decision points for your error reporting scenario. \ No newline at end of file diff --git a/docs/platforms/native/configuration/backends/index.mdx b/docs/platforms/native/configuration/backends/index.mdx index 81aec822e645e..66be0d08f3c63 100644 --- a/docs/platforms/native/configuration/backends/index.mdx +++ b/docs/platforms/native/configuration/backends/index.mdx @@ -9,19 +9,22 @@ and libraries. Similar to plugins, they extend the functionality of the Sentry SDK. The Native SDK can use different backends that are responsible for capturing -crashes. The backend is configured at build-time, using the `SENTRY_BACKEND` +crashes. The backend is configured at build-time using the `SENTRY_BACKEND` CMake option, with support for the following options: - [`crashpad`](crashpad/): This uses the out-of-process crashpad handler. - It is used as the default on Windows, macOS and Linux. + It is used as the default on Windows, macOS, and Linux. - `breakpad`: This uses the in-process breakpad handler. -- `inproc`: A small in-process handler which is supported on all platforms, - and is used as default on Android. +- `inproc`: A small in-process handler supported on all platforms +and used as a default on Android. - `none`: This builds `sentry-native` without a backend, so it does not handle - crashes at all. It is primarily used for tests. + crashes. It is primarily used for tests. `Breakpad` and `inproc` both run "in-process", so they run in the same process as the application they capture crashes for. This means these backends can't send a crash to Sentry immediately after it happens. Instead, the crash report is written to disk and sent the next time the application is run. Since `Crashpad` -runs in a different process it doesn't have this limitation. +runs in a different process, it doesn't have this limitation. + +The [Advanced Usage](/platforms/native/advanced-usage/backend-tradeoffs) section +explains the trade-offs in choosing the correct backend for your use case. \ No newline at end of file From d447fb27fc7ef47c24da2653b8fd19f12e731c0d Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 15:02:34 +0100 Subject: [PATCH 02/11] Add hook on macOS exception --- docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx index 524f5c13ff3be..cae7e710bb55a 100644 --- a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -32,6 +32,7 @@ Sentry decided on `crashpad` as the default on all platforms because there are a * IPC between your process and the `crashpad_handler` is inhibited by security settings or not available in your deployment target * your deployment scenario cannot wait for the `crashpad_handler` to finish its work before a shutdown-after-crash (systemd, Docker) * you want to distribute your application via the macOS App Store +* you want to define crash hooks on macOS, because there, error handling happens entirely in the `crashpad_handler` whereas on Linux and Windows at least the initial handling happens in your process after which `crashpad_handler` takes over and snapshots the process to send a crash report In the above cases, if you cannot loosen the requirements of your environment, you have to choose an in-process backend (meaning either `breakpad` or `inproc`). From 260265a5a07ff1ba0d6b248dba2391e06aba664c Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 15:06:27 +0100 Subject: [PATCH 03/11] Update docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx Co-authored-by: Karl Heinz Struggl --- .../platforms/native/advanced-usage/backend-tradeoffs/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx index cae7e710bb55a..72435fb70d81c 100644 --- a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -45,7 +45,7 @@ Both backends are comparable in how they differ from `crashpad`. However, there * `inproc`, on the other hand, uses signal handling on Linux and macOS (meaning you only get a `POSIX` compatibility layer over mach exception ports) and `UnhandledExceptionFilter` on Windows (solely errors registered by SEH) * as a result of choosing signal handling on macOS, `inproc` is currently broken on macOS since Apple eliminated unwinding from signal handlers * `inproc` is exceptionally lightweight and written entirely in C, meaning it does not rely on a weighty C++ runtime library, which is also more often affected by ABI incompatibilities -* `breakpad` generates minidumps (like `crashpad` does), whereas `inproc` follows the sentry event structure and primarily defers to the OS-provided unwinder and symbolication capabilities. Sentry can potentially extract more information from the provided minidump than simply a stack trace and registers. However, it also means that the crash context inside the minidump will be opaque until processed in the backend, whereas a local `relay` instance could process an entire `inproc`-event if required. +* `breakpad` generates minidumps (like `crashpad` does), whereas `inproc` follows the Sentry event structure and primarily defers to the OS-provided unwinder and symbolication capabilities. Sentry can potentially extract more information from the provided minidump than simply a stack trace and registers. However, it also means that the crash context inside the minidump will be opaque until processed in the backend, whereas a local `relay` instance could process an entire `inproc`-event if required. ### So when do I choose `inproc`? From a6396e2d98f8a26bc061cdf9c0b9d143bbcaba21 Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 15:06:49 +0100 Subject: [PATCH 04/11] Update docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx Co-authored-by: Karl Heinz Struggl --- .../platforms/native/advanced-usage/backend-tradeoffs/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx index 72435fb70d81c..f93be0d7aa24a 100644 --- a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -3,7 +3,7 @@ title: Backend Tradeoffs description: "How to choose the right backend in the Native SDK." sidebar_order: 1000 --- -The Native SDK lets users decide at compile-time between three backends: +The Native SDK lets users decide at compile-time between three crash backends: * `crashpad` * `breakpad` From 812149ef95b78f3db0d7fe48ed66459982c538a3 Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 15:07:13 +0100 Subject: [PATCH 05/11] Update docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx Co-authored-by: Karl Heinz Struggl --- .../platforms/native/advanced-usage/backend-tradeoffs/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx index f93be0d7aa24a..c26091a00cf21 100644 --- a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -1,6 +1,6 @@ --- title: Backend Tradeoffs -description: "How to choose the right backend in the Native SDK." +description: "How to choose the right crash backend in the Native SDK." sidebar_order: 1000 --- The Native SDK lets users decide at compile-time between three crash backends: From 2c50cd1228c812281182a7db8e7fead43aecf74b Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 15:07:31 +0100 Subject: [PATCH 06/11] Update docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx Co-authored-by: Karl Heinz Struggl --- .../platforms/native/advanced-usage/backend-tradeoffs/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx index c26091a00cf21..6b6f8689c7f8c 100644 --- a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -18,7 +18,7 @@ Currently, `crashpad` is the default on all platforms because it * HTTP proxy support * CMake build scripts * GCC and MinGW support - * `FirstChanceHandler` on Windows and extension of its synchronization to support sentry hooks + * `FirstChanceHandler` on Windows and extension of its synchronization to support Sentry hooks * cooperation with Epic's Easy Anti Cheat * supports more error types on [Linux](/platforms/native/advanced-usage/signal-handling/#signals-of-interest) and Windows (`abort()` and other `fast-fail` crashes, handling of heap corruptions) * is more maintained upstream (although most changes affect new platforms like Fuchsia) From 7d5b8bd80be94ba2cfabec81a036efa23d2af5e5 Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 15:07:50 +0100 Subject: [PATCH 07/11] Update docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx Co-authored-by: Ivan Dlugos <6349682+vaind@users.noreply.github.com> --- .../platforms/native/advanced-usage/backend-tradeoffs/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx index 6b6f8689c7f8c..5c6b3bd29f14e 100644 --- a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -9,7 +9,7 @@ The Native SDK lets users decide at compile-time between three crash backends: * `breakpad` * `inproc` -Currently, `crashpad` is the default on all platforms because it +Currently, `crashpad` is the default on all desktop platforms because it * has an external `handler` process that allows for external snapshots and immediate sending of crash reports (instead of the next successful start of your application) * is the primary target for extension compared to upstream, including From a92cf86e3c9fad73026cb979fbce7fe20c49474f Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 15:08:14 +0100 Subject: [PATCH 08/11] Update docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx Co-authored-by: Ivan Dlugos <6349682+vaind@users.noreply.github.com> --- .../platforms/native/advanced-usage/backend-tradeoffs/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx index 5c6b3bd29f14e..f74f50df0f74c 100644 --- a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -11,7 +11,7 @@ The Native SDK lets users decide at compile-time between three crash backends: Currently, `crashpad` is the default on all desktop platforms because it -* has an external `handler` process that allows for external snapshots and immediate sending of crash reports (instead of the next successful start of your application) +* has an external `handler` process that allows for external snapshots and sending crash reports immediately (instead of on the next successful start of your application) * is the primary target for extension compared to upstream, including * client-side stack traces * attachment handling From 1b641e135067334a9c2e4276fdd258e879bc966b Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 15:09:12 +0100 Subject: [PATCH 09/11] Update docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx Co-authored-by: Karl Heinz Struggl --- .../platforms/native/advanced-usage/backend-tradeoffs/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx index f74f50df0f74c..309796ec2a62d 100644 --- a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -51,7 +51,7 @@ Both backends are comparable in how they differ from `crashpad`. However, there `inproc` is currently the backend of choice for `Android` because it allows us to couple it with our own fork of a powerful platform unwinder `libunwindstack` (rather than relying on a user-space interface). This allows us to support very old Android versions. -`inproc` is the right choice if you want +`inproc` is the right choice if you * want minimal dependencies * want the smallest footprint for the resulting artifact From 48eb26a5266f00de6504ce94428219a941b6b730 Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 15:35:06 +0100 Subject: [PATCH 10/11] Highlight inproc macOS breakage --- .../platforms/native/advanced-usage/backend-tradeoffs/index.mdx | 2 +- docs/platforms/native/configuration/backends/index.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx index 309796ec2a62d..a5931db871764 100644 --- a/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx +++ b/docs/platforms/native/advanced-usage/backend-tradeoffs/index.mdx @@ -45,7 +45,7 @@ Both backends are comparable in how they differ from `crashpad`. However, there * `inproc`, on the other hand, uses signal handling on Linux and macOS (meaning you only get a `POSIX` compatibility layer over mach exception ports) and `UnhandledExceptionFilter` on Windows (solely errors registered by SEH) * as a result of choosing signal handling on macOS, `inproc` is currently broken on macOS since Apple eliminated unwinding from signal handlers * `inproc` is exceptionally lightweight and written entirely in C, meaning it does not rely on a weighty C++ runtime library, which is also more often affected by ABI incompatibilities -* `breakpad` generates minidumps (like `crashpad` does), whereas `inproc` follows the Sentry event structure and primarily defers to the OS-provided unwinder and symbolication capabilities. Sentry can potentially extract more information from the provided minidump than simply a stack trace and registers. However, it also means that the crash context inside the minidump will be opaque until processed in the backend, whereas a local `relay` instance could process an entire `inproc`-event if required. +* `breakpad` generates minidumps (like `crashpad` does), whereas `inproc` follows the Sentry event structure and primarily defers to the OS-provided unwinder and symbolication capabilities. Sentry can potentially extract more information from the provided minidump than simply a stack trace and registers. However, it also means that the crash context inside the minidump will be opaque until processed in the backend, whereas a local `relay` instance could process an entire `inproc` event if required. ### So when do I choose `inproc`? diff --git a/docs/platforms/native/configuration/backends/index.mdx b/docs/platforms/native/configuration/backends/index.mdx index 66be0d08f3c63..812a53213b368 100644 --- a/docs/platforms/native/configuration/backends/index.mdx +++ b/docs/platforms/native/configuration/backends/index.mdx @@ -16,7 +16,7 @@ CMake option, with support for the following options: It is used as the default on Windows, macOS, and Linux. - `breakpad`: This uses the in-process breakpad handler. - `inproc`: A small in-process handler supported on all platforms -and used as a default on Android. +and used as a default on Android. It does no longer work on macOS since version 13 ("Ventura"). - `none`: This builds `sentry-native` without a backend, so it does not handle crashes. It is primarily used for tests. From 8b260aaf2d72e5d64d531c3aa542035f8c3de2a7 Mon Sep 17 00:00:00 2001 From: Mischan Toosarani-Hausberger Date: Mon, 4 Nov 2024 16:29:58 +0100 Subject: [PATCH 11/11] grammar inproc macOS 13 breakage --- docs/platforms/native/configuration/backends/index.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/platforms/native/configuration/backends/index.mdx b/docs/platforms/native/configuration/backends/index.mdx index 812a53213b368..21f6f56a62f93 100644 --- a/docs/platforms/native/configuration/backends/index.mdx +++ b/docs/platforms/native/configuration/backends/index.mdx @@ -16,7 +16,7 @@ CMake option, with support for the following options: It is used as the default on Windows, macOS, and Linux. - `breakpad`: This uses the in-process breakpad handler. - `inproc`: A small in-process handler supported on all platforms -and used as a default on Android. It does no longer work on macOS since version 13 ("Ventura"). +and used as a default on Android. It has no longer worked on macOS since version 13 ("Ventura"). - `none`: This builds `sentry-native` without a backend, so it does not handle crashes. It is primarily used for tests. @@ -27,4 +27,4 @@ written to disk and sent the next time the application is run. Since `Crashpad` runs in a different process, it doesn't have this limitation. The [Advanced Usage](/platforms/native/advanced-usage/backend-tradeoffs) section -explains the trade-offs in choosing the correct backend for your use case. \ No newline at end of file +explains the trade-offs in choosing the correct backend for your use case.