Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault while collecting stacktrace with PIKA_WITH_THREAD_STACK_MMAP=OFF #1316

Open
msimberg opened this issue Nov 4, 2024 · 0 comments
Open
Labels
effort: 3 A few days of work. priority: low Nice to have, but nobody is going to be sad if this is never done. type: bug type: feature

Comments

@msimberg
Copy link
Contributor

msimberg commented Nov 4, 2024

When PIKA_WITH_THREAD_STACK_MMAP is disabled, the stack trace collection may segfault in a few of the resource partitioner tests (shutdown_suspended_pus, suspend_thread, suspend_pool). This is reproducible in CI with at least clang 15 and 17, and GCC 12 when they use PIKA_WITH_THREAD_STACK_MMAP=OFF. It's also reproducible locally for me with GCC 14. The failure does not seem to reproduce in debug mode (requires -O1 or higher for failure) or with custom allocators (tested mimalloc, need system for failure).

The quickest way to reproduce the failure is with the shutdown_suspended_pus_test and running it in a loop or multiple times through ctest:

ctest --repeat-until-fail 100 --output-on-failure -R shutdown_suspended_pus

The test should segfault most of the time within a few tens of repetitions.

The backtrace is always:

#0  0x00007ffff7e6898f in uw_frame_state_for () from /nix/store/zw9mmk47hg1cghq34ih0dff8q6ngnlr8-gcc-14.2.0-lib/lib/libgcc_s.so.1
#1  0x00007ffff7e6a85e in _Unwind_Backtrace () from /nix/store/zw9mmk47hg1cghq34ih0dff8q6ngnlr8-gcc-14.2.0-lib/lib/libgcc_s.so.1
#2  0x00007ffff7cb55d7 in pika::debug::detail::stack_trace::trace (array=array@entry=0x7fffec01b120, n=n@entry=22) at /home/mjs/src/pika/libs/pika/debugging/src/backtrace.cpp:117
#3  0x00007ffff7d3c0df in pika::debug::detail::backtrace::backtrace (this=0x7fffec262150, frames_no=22) at /nix/store/zzffrygpx8l7wk3falwrj206yyv6akgy-gcc-14.2.0/include/c++/14.2.0/bits/stl_iterator.h:1067
#4  pika::debug::detail::trace[abi:cxx11](unsigned long) (frames_no=20) at /home/mjs/src/pika/libs/pika/debugging/include/pika/debugging/detail/backtrace.hpp:109
#5  pika::detail::custom_exception_info (func=..., file=..., line=1420, auxinfo=...) at /home/mjs/src/pika/libs/pika/runtime/src/custom_exception_info.cpp:335
#6  0x00007ffff7cf3e77 in std::__invoke_impl<pika::exception_info, pika::exception_info (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, st
d::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<c
har, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (__f=<optimized out>)
    at /nix/store/zzffrygpx8l7wk3falwrj206yyv6akgy-gcc-14.2.0/include/c++/14.2.0/bits/invoke.h:61
#7  std::__invoke_r<pika::exception_info, pika::exception_info (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > cons
t&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<cha
r>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (__fn=<optimized out>)
    at /nix/store/zzffrygpx8l7wk3falwrj206yyv6akgy-gcc-14.2.0/include/c++/14.2.0/bits/invoke.h:116
#8  std::_Function_handler<pika::exception_info(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx
11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), pika::exception_info (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_trait
s<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::_M_invoke (__functor=..., __args#0=..., __args#1=..., __args#2=<optimized out>, __args#3=...)
    at /nix/store/zzffrygpx8l7wk3falwrj206yyv6akgy-gcc-14.2.0/include/c++/14.2.0/bits/std_function.h:291
#9  0x00007ffff7cc5a95 in std::function<pika::exception_info(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, lon
g, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator() (this=0x7ffff7e31360 <pika::detail::custom_exception_info_handler>, __args#0=..., __args#1=..., __args#2=<optimized out>, __args#3=...)
    at /nix/store/zzffrygpx8l7wk3falwrj206yyv6akgy-gcc-14.2.0/include/c++/14.2.0/bits/std_function.h:587
#10 pika::detail::construct_custom_exception<pika::exception> (e=..., func=..., file=..., line=1420, auxinfo=...) at /home/mjs/src/pika/libs/pika/errors/src/exception.cpp:202
#11 pika::detail::get_exception<pika::exception> (e=..., func=..., file=..., line=line@entry=1420, auxinfo=...) at /home/mjs/src/pika/libs/pika/errors/src/exception.cpp:238
#12 0x00007ffff7cc5bae in pika::detail::throw_exception<pika::exception> (e=..., func=..., file=..., line=line@entry=1420) at /home/mjs/src/pika/libs/pika/errors/src/exception.cpp:247
#13 0x00007ffff7cbdf6a in pika::detail::throw_exception (errcode=<optimized out>, msg=..., func=..., file=..., line=1420) at /home/mjs/src/pika/libs/pika/errors/src/throw_exception.cpp:24
#14 0x00007ffff7cbfefd in pika::detail::throws_if (ec=..., errcode=pika::error::invalid_status, msg=..., func=..., file=..., line=1420) at /home/mjs/src/pika/libs/pika/errors/src/throw_exception.cpp:54
#15 0x00007ffff7d59ec3 in pika::threads::detail::scheduled_thread_pool<pika::threads::detail::static_priority_queue_scheduler<std::mutex, pika::threads::detail::lockfree_fifo, pika::threads::detail::lockfree_fifo, pika::threads::detail::loc
kfree_fifo> >::suspend_processing_unit_direct (this=0x44dad0, virt_core=0, ec=...) at /home/mjs/src/pika/libs/pika/thread_pools/include/pika/thread_pools/scheduled_thread_pool_impl.hpp:1420
#16 0x0000000000403c45 in pika_main () at /home/mjs/src/pika/libs/pika/resource_partitioner/tests/unit/shutdown_suspended_pus.cpp:48

i.e. comes from libstdc++'s unwind functionality.

A quick test with libunwind (https://www.nongnu.org/libunwind/man/libunwind(3).html, see "Local Unwinding") does not seem to have the same problem, so we could consider adding libunwind support (optional?) for stacktraces.

@msimberg msimberg added effort: 3 A few days of work. priority: low Nice to have, but nobody is going to be sad if this is never done. type: feature type: bug labels Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
effort: 3 A few days of work. priority: low Nice to have, but nobody is going to be sad if this is never done. type: bug type: feature
Projects
Status: Backlog
Development

No branches or pull requests

1 participant