Fixing the website title #1

AhmedSoliman · 2018-09-12T15:51:39Z

Fixes the tagline of the website.

Summary: (gdb) bt #0 ld_event_add_nolock_ (ev=0x7fffb50050f8, tv=0x7fffb65f0a68, tv_is_absolute=0) at logdevice/external/libevent-2.1.3-alpha/minheap-internal.h:141 #1 0x00000000029c7fb5 in facebook::logdevice::LibeventTimer::activate (this=0x7fffb50050e8, delay=0x7fffb65f0a68) at logdevice/common/LibeventTimer.cpp:76 #2 0x00000000029c7bd2 in facebook::logdevice::LibeventTimer::activate (this=0x7fffb50050e8, delay=..., timeout_map=<optimized out>) at logdevice/common/LibeventTimer.cpp:65 #3 0x0000000002a73829 in facebook::logdevice::Worker::reportLoad (this=0x7ffff3aa8200) at logdevice/common/Worker.cpp:828 #4 0x0000000002a728d3 in facebook::logdevice::Worker::onThreadStarted (this=0x7ffff3aa8200) at logdevice/common/Worker.cpp:468 #5 0x000000000284b7e5 in facebook::logdevice::ClientWorker::onThreadStarted (this=0x7ffff3aa8200) at logdevice/lib/ClientWorker.cpp:41 #6 0x00000000036b0c29 in facebook::logdevice::EventLoop::run (this=0x7ffff3aa8200) at logdevice/common/EventLoop.cpp:155 #7 0x00000000036b0699 in facebook::logdevice::EventLoop::enter (self=0x7fffb50050f8) at logdevice/common/EventLoop.cpp:134 #8 0x00007ffff583f66e in start_thread (arg=0x7fffb65f5700) at pthread_create.c:465 #9 0x00007ffff5154e2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 (gdb) frame 1 #1 0x00000000029c7fb5 in facebook::logdevice::LibeventTimer::activate (this=0x7fffb50050e8, delay=0x7fffb65f0a68) at logdevice/common/LibeventTimer.cpp:76 76 logdevice/common/LibeventTimer.cpp: No such file or directory. (gdb) p timer_ $1 = {ev_evcallback = {evcb_active_next = {tqe_next = 0x0, tqe_prev = 0x0}, evcb_flags = 0, evcb_pri = 0 '\000', evcb_closure = 0 '\000', evcb_cb_union = {evcb_callback = 0x0, evcb_selfcb = 0x0, evcb_evfinalize = 0x0, evcb_cbfinalize = 0x0}, evcb_arg = 0x0}, ev_timeout_pos = {ev_next_with_common_timeout = {tqe_next = 0x0, tqe_prev = 0x0}, min_heap_idx = 0}, ev_fd = 0, ev_base = 0x0, ev_ = {ev_io = {ev_io_next = {le_next = 0x0, le_prev = 0x0}, ev_timeout = {tv_sec = 0, tv_usec = 0}}, ev_signal = {ev_signal_next = {le_next = 0x0, le_prev = 0x0}, ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 0, ev_res = 0, ev_timeout = {tv_sec = 0, tv_usec = 0}} Reviewed By: tau0 Differential Revision: D13891599 fbshipit-source-id: 940c60a939a16f6a30373fc0c8c6569f77ea4fca

Summary: Got a segfault with this stack trace: (gdb) bt #0 0x00007ff56c2919cf in raise () from /usr/local/fbcode/platform007/lib/libpthread.so.0 #1 0x0000000002a9656a in facebook::logdevice::handle_fatal_signal (sig=<optimized out>) at logdevice/server/fatalsignal.cpp:58 #2 <signal handler called> #3 facebook::logdevice::configuration::Node::getNumShards (this=0x7fe60c71c610) at logdevice/common/configuration/Node.h:258 #4 facebook::logdevice::AllServerReadStreams::sendShardStatusToClient (this=<optimized out>, cid=...) at logdevice/server/read_path/AllServerReadStreams.cpp:769 #5 0x0000000002a9ec3b in facebook::logdevice::AllServerReadStreams::onShardStatusChanged (this=0x7ff4fcbd5c00) at logdevice/server/read_path/AllServerReadStreams.cpp:740 #6 0x0000000002f9b2d1 in facebook::logdevice::ShardAuthoritativeStatusManager::notifySubscribers (this=<optimized out>) at logdevice/common/ShardAuthoritativeStatusMap.cpp:197 #7 0x0000000002f9b21e in facebook::logdevice::UpdateShardAuthoritativeMapRequest::execute (this=<optimized out>) at logdevice/common/ShardAuthoritativeStatusMap.cpp:138 #8 0x0000000002fd4a0e in facebook::logdevice::Worker::processRequest (this=0x7ff56953d600, rq=std::unique_ptr<facebook::logdevice::Request> containing 0x7ff480557e90) at logdevice/common/Worker.cpp:1282 #9 0x0000000002fdb424 in facebook::logdevice::Worker::forcePost(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >&, signed char)::$_9::operator()() (this=<optimized out>) at logdevice/common/Worker.cpp:1405 #10 folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::Worker::forcePost(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >&, signed char)::$_9>(folly::detail::function::Data&) (p=...) at folly/Function.h:360 #11 0x0000000002fd581b in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x0) at folly/Function.h:376 #12 facebook::logdevice::Worker::pickAndExecuteTask (this=<optimized out>, priority_hint=<optimized out>) at logdevice/common/Worker.cpp:1342 #13 0x0000000002fdaa62 in facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_8::operator()() const (this=<optimized out>) at logdevice/common/Worker.cpp:1388 #14 folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_8>(folly::detail::function::Data&) (p=...) at folly/Function.h:360 #15 0x00000000032de717 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=<optimized out>) at folly/Function.h:376 #16 facebook::logdevice::EventLoop::add(folly::Function<void ()>)::$_0::operator()() (this=0x7fee0ba685a0) at logdevice/common/EventLoop.cpp:159 #17 folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::EventLoop::add(folly::Function<void ()>)::$_0>(folly::detail::function::Data&) (p=...) at folly/Function.h:367 #18 0x0000000000606428 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7fee0ba685a0) at folly/Function.h:376 #19 facebook::logdevice::EventLoopTaskQueue::executeTasks (this=<optimized out>, num_tasks_to_dequeue=1) at logdevice/common/EventLoopTaskQueue.cpp:151 #20 0x0000000000605801 in facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::$_0::operator()(unsigned long) const (n=1, this=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:119 #21 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::$_0&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::$_0&, unsigned long) (this=0x7ff56986a4f0, func=..., maxBatchSize=<optimized out>) at logdevice/common/LifoEventSem.h:205 #22 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler (arg=<optimized out>, what=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:124 #23 0x0000000000603f52 in facebook::logdevice::EventHandler<&facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler, &facebook::logdevice::(anonymous namespace)::preflight_noop, &facebook::logdevice::(anonymous namespace)::postflight_noop> (what=<optimized out>, arg=0x20) at logdevice/common/EventHandler.h:92 #24 0x0000000000603e08 in event_process_active_single_queue (base=0x7ff561bddd00, activeq=0x7ff56983c230, max_to_process=2147483647, endtime=0x0) at logdevice/external/libevent-2.1.3-alpha/event.c:1449 #25 0x000000000548641c in event_process_active (base=<optimized out>) at logdevice/external/libevent-2.1.3-alpha/event.c:1596 #26 ld_event_base_loop (base=0x7ff561bddd00, flags=<optimized out>) at logdevice/external/libevent-2.1.3-alpha/event.c:1819 #27 0x00000000032dda80 in facebook::logdevice::EventLoop::run (this=0x7ff56953d600) at logdevice/common/EventLoop.cpp:222 #28 0x00000000032dd279 in facebook::logdevice::EventLoop::enter (self=0x20) at logdevice/common/EventLoop.cpp:173 #29 0x00007ff56c2866b6 in start_thread () from /usr/local/fbcode/platform007/lib/libpthread.so.0 #30 0x00007ff56b967ebf in clone () from /usr/local/fbcode/platform007/lib/libc.so.6 This diff fixes a bug that could cause this. Reviewed By: gdrane Differential Revision: D15887025 fbshipit-source-id: eb6b214ae33a0f36506d0d65d95f33469a6cbae1

Summary: Stack trace: 0x00007fd80a03f9cf raise unknown -1 0x00000000027eb82a facebook::logdevice::handle_fatal_signal logdevice/server/fatalsignal.cpp 58 (unknown) unknown -1 0x00007fd80963b14f raise unknown -1 0x00007fd809624935 abort unknown -1 0x00000000057e488d facebook::logdevice::dbg::ld_check_fail_impl logdevice/common/debug.cpp 643 0x0000000000912c02 facebook::logdevice::SealStorageTask::SealStorageTask logdevice/server/storage/SealStorageTask.cpp 55 0x0000000000912785 std::make_unique<facebook::logdevice::SealStorageTask, facebook::logdevice::logid_t const&, facebook::logdevice::epoch_t&, facebook::logdevice::Seal&, facebook::logdevice::WeakRefHolder<facebook::logdevice::PurgeUncleanEpochs>::Ref> ./third-party-buck/platform007/build/libgcc/include/c++/7.3.0/bits/unique_ptr.h 825 facebook::logdevice::PurgeUncleanEpochs::getPurgeEpochs logdevice/server/storage/PurgeUncleanEpochs.cpp 168 0x0000000000911d20 facebook::logdevice::PurgeUncleanEpochs::start logdevice/server/storage/PurgeUncleanEpochs.cpp 92 0x0000000000910818 facebook::logdevice::PurgeCoordinator::startPurge logdevice/server/storage/PurgeCoordinator.cpp 781 0x0000000000910044 facebook::logdevice::PurgeCoordinator::onCleanMessage logdevice/server/storage/PurgeCoordinator.cpp 575 0x0000000000ae582e facebook::logdevice::PurgeCoordinator::onReceived logdevice/server/storage/PurgeCoordinator.cpp 221 0x0000000000609a4d facebook::logdevice::ServerMessageDispatch::onReceivedImpl logdevice/server/ServerMessageDispatch.cpp 103 0x000000000060789a std::__invoke_impl<facebook::logdevice::Message::Disposition, facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*&)(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&), facebook::logdevice::MessageDispatch*&, facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&> ./third-party-buck/platform007/build/libgcc/include/c++/7.3.0/bits/ invoke.h 73 std::__invoke<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*&)(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&), facebook::logdevice::MessageDispatch*&, facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&> ./third-party-buck/platform007/build/libgcc/include/c++/7.3.0/bits/invoke.h 95 std::_Bind<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*(facebook::logdevice::MessageDispatch*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&)>::__call<facebook::logdevice::Message::Disposition, facebook::logdevice::Message*&&, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&, 0ul, 1ul, 2ul, 3ul>(std::tuple<facebook::logdevice::Message*&&, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) ./third-party-buck/platform007/build/libgcc/include/c++/7.3.0/functional 467 std::_Bind<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*(facebook::logdevice::MessageDispatch*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&)>::operator()<facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&, facebook::logdevice::Message::Disposition>(facebook::logdevice::Message*&&, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&) ./third-party-buck/platform007/build/libgcc/include/c++/7.3.0/functional 549 facebook::logdevice::(anonymous namespace)::executeOnWorker<std::_Bind<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*(facebook::logdevice::MessageDispatch*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&)>, facebook::logdevice::Message*>(facebook::logdevice::Worker*, std::_Bind<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*(facebook::logdevice::MessageDispatch*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&)>, facebook::logdevice::Message*, facebook::logdevice::Address const&, std::shared_ptr<facebook::logdevice::PrincipalIdentity>, signed char, facebook::logdevice::ResourceBudget::Token)::{lambda()#1}::operator()() logdevice/common/ Socket.cpp 2808 folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::(anonymous namespace)::executeOnWorker<std::_Bind<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*(facebook::logdevice::MessageDispatch*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&)>, facebook::logdevice::Message*>(facebook::logdevice::Worker*, std::_Bind<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*(facebook::logdevice::MessageDispatch*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&)>, facebook::logdevice::Message*, facebook::logdevice::Address const&, std::shared_ptr<facebook::logdevice::PrincipalIdentity>, signed char, facebook::logdevice::ResourceBudget::Token)::{lambda()#1}>(folly::detail::function::Data&) folly/Function.h 367 0x00000000006075a2 folly::detail::function::FunctionTraits<void ()>::operator()() folly/Function.h 376 facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9::operator()() logdevice/common/Worker.cpp 1359 folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9>(folly::detail::function::Data&) folly/Function.h 367 0x00000000035265af folly::detail::function::FunctionTraits<void ()>::operator()() folly/Function.h 376 facebook::logdevice::EventLoopTaskQueue::executeTasks logdevice/common/EventLoopTaskQueue.cpp 163 0x0000000003525d75 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1::operator()(unsigned int) logdevice/common/EventLoopTaskQueue.cpp 110 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&, unsigned int) logdevice/common/LifoEventSem.h 338 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler logdevice/common/EventLoopTaskQueue.cpp 115 0x0000000000605588 event_process_active_single_queue logdevice/external/libevent-2.1.3-alpha/event.c 1449 0x0000000005bcbb0c event_process_active logdevice/external/libevent-2.1.3-alpha/event.c 1596 ld_event_base_loop logdevice/external/libevent-2.1.3-alpha/event.c 1819 0x0000000005958a54 facebook::logdevice::EvBase::loop logdevice/common/libevent/EvBase.cpp 58 0x00000000035248e4 facebook::logdevice::EventLoop::run logdevice/common/EventLoop.cpp 175 0x00007fd80a93e680 execute_native_thread_routine unknown -1 0x00007fd80a0346b6 start_thread unknown -1 0x00007fd809715ebf clone unknown -1 The Seal::node_ is an invalid NodeID (0x80000000). Reviewed By: gdrane Differential Revision: D17332350 fbshipit-source-id: 202196527c5d2783d347e61ef847ba7e9e5236d6

…om maintenance and to check if a maintenance is empty Summary: Two new helper functions to: 1- Given a maintenance, remove all the non existent shards/sequencers out of it. 2- Given a maintenance, checks whether the maintenance is empty or not (usually used after #1). Reviewed By: AhmedSoliman Differential Revision: D17684399 fbshipit-source-id: 27eb1324eddb4f214f2ac86303edd1367682cac2

Summary: ASAN error: ==373949==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700007d938 at pc 0x0000005be44b bp 0x7f71060ab440 sp 0x7f71060ab438 READ of size 2 at 0x60700007d938 thread T7 (IOThreadPool2) SCARINESS: 42 (2-byte-read-heap-use-after-free) I1008 20:44:07.669364 373949 [not-set] TestUtil.cpp:292] wait_until() Still waiting until: node 0 starts (157.375s) I1008 20:44:07.811503 373949 [not-set] TestUtil.cpp:292] wait_until() Still waiting until: node 0 starts (187.389s) I1008 20:44:38.524135 373949 [not-set] TestUtil.cpp:292] wait_until() Still waiting until: node 0 starts (218.206s) #0 0x5be44a in folly::SocketAddress::SocketAddress(folly::SocketAddress const&) folly/SocketAddress.h:83 #1 0x2ba49a0 in facebook::logdevice::AdminCommandClient::Request::Request(facebook::logdevice::AdminCommandClient::Request const&) logdevice/ops/admin_command_client/AdminCommandClient.h:32 #2 0x2ba49a0 in facebook::logdevice::AdminClientConnection::AdminClientConnection(folly::EventBase*, facebook::logdevice::AdminCommandClient::Request const&, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::shared_ptr<folly::SSLContext>) logdevice/ops/admin_command_client/AdminCommandClient.cpp:39 #3 0x2ba40c1 in std::_MakeUniq<facebook::logdevice::AdminClientConnection>::__single_object std::make_unique<facebook::logdevice::AdminClientConnection, folly::EventBase*, facebook::logdevice::AdminCommandClient::Request const&, std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::shared_ptr<folly::SSLContext> >(folly::EventBase*&&, facebook::logdevice::AdminCommandClient::Request const&, std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::shared_ptr<folly::SSLContext>&&) third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:825 #4 0x2ba20b7 (/data/users/kolmike/fbsource/fbcode/buck-out/dbgo/gen/logdevice/test/tests+0x2ba20b7) #5 0x13e1204 in folly::futures::detail::Core<folly::Unit>::doCallback(folly::Executor::KeepAlive<folly::Executor>&&, folly::futures::detail::State)::'lambda'(folly::Executor::KeepAlive<folly::Executor>&&)::operator()(folly::Executor::KeepAlive<folly::Executor>&&) folly/futures/detail/Core.h:940 #6 0x2d05a3b in folly::detail::function::FunctionTraits<void ()>::operator()() folly/Function.h:376 #7 0x2d05a3b in folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) folly/executors/ThreadPoolExecutor.cpp:84 #8 0x2caaec7 in folly::IOThreadPoolExecutor::add(folly::Function<void ()>, std::chrono::duration<long, std::ratio<1l, 1000l> >, folly::Function<void ()>)::$_0::operator()() folly/executors/IOThreadPoolExecutor.cpp:106 #9 0x2caaec7 in void folly::detail::function::FunctionTraits<void ()>::callBig<folly::IOThreadPoolExecutor::add(folly::Function<void ()>, std::chrono::duration<long, std::ratio<1l, 1000l> >, folly::Function<void ()>)::$_0>(folly::detail::function::Data&) folly/Function.h:367 #10 0x2cfb023 in folly::detail::function::FunctionTraits<void ()>::operator()() folly/Function.h:376 #11 0x2cfb023 in folly::EventBase::FunctionRunner::messageAvailable(folly::Function<void ()>&&) folly/io/async/EventBase.cpp:59 #12 0x2cf4b37 in folly::NotificationQueue<folly::Function<void ()> >::Consumer::consumeMessages(bool, unsigned long*) folly/io/async/NotificationQueue.h:729 #13 0x2cfb9fc in folly::EventHandler::libeventCallback(int, short, void*) folly/io/async/EventHandler.cpp:161 #14 0x3772f98 in event_process_active /home/engshare/third-party2/libevent/1.4.14b_hphp/src/libevent-1.4.14b-stable/event.c:390 #15 0x3773237 in event_base_loop /home/engshare/third-party2/libevent/1.4.14b_hphp/src/libevent-1.4.14b-stable/event.c:532 #16 0x2cec440 in folly::EventBase::loopBody(int, bool) folly/io/async/EventBase.cpp:338 #17 0x2ceec9b in folly::EventBase::loop() folly/io/async/EventBase.cpp:261 #18 0x2ceec9b in folly::EventBase::loopForever() folly/io/async/EventBase.cpp:477 #19 0x2ca7590 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) folly/executors/IOThreadPoolExecutor.cpp:179 #20 0x2d0fd70 in void std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>(std::__invoke_memfun_deref, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&) third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/invoke.h:73 #21 0x7f711342467f in execute_native_thread_routine /home/engshare/third-party2/libgcc/7.x/src/gcc-7.x/libstdc++-v3/src/c++11/thread.cc:83:15 #22 0x7f71129026b5 in start_thread /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/nptl/pthread_create.c:465:7 #23 0x7f711221bebe in __GI___clone /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/sysdeps/unix/sysv/linux/x86_64/clone.S:95 0x60700007d938 is located 24 bytes inside of 72-byte region [0x60700007d920,0x60700007d968) freed by thread T0 here: #0 0x34b2258 in operator delete(void*) (/data/users/kolmike/fbsource/fbcode/buck-out/dbgo/gen/logdevice/test/tests+0x34b2258) #1 0x2b07f64 in facebook::logdevice::test::nc(std::shared_ptr<facebook::logdevice::AdminCommandClient> const&, folly::SocketAddress const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >) logdevice/test/utils/nc.cpp:36 #2 0xe1d5b1 in facebook::logdevice::IntegrationTestUtils::Node::sendCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::chrono::duration<long, std::ratio<1l, 1000l> >) const logdevice/test/utils/IntegrationTestUtils.cpp:1538 #3 0xe20a03 in facebook::logdevice::IntegrationTestUtils::Node::getServerInfo() const logdevice/test/utils/IntegrationTestUtils.cpp:1619 #4 0xe72c70 in facebook::logdevice::IntegrationTestUtils::Node::waitUntilStarted(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_5::operator()() const logdevice/test/utils/IntegrationTestUtils.cpp:1648 #5 0xe72c70 in std::_Function_handler<bool (), facebook::logdevice::IntegrationTestUtils::Node::waitUntilStarted(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_5>::_M_invoke(std::_Any_data const&) third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:301 #6 0x177c3f5 in std::function<bool ()>::operator()() const third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 #7 0x177c3f5 in facebook::logdevice::wait_until(char const*, std::function<bool ()>, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >) logdevice/common/test/TestUtil.cpp:277 #8 0xe0f70d in facebook::logdevice::IntegrationTestUtils::Node::waitUntilStarted(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >) logdevice/test/utils/IntegrationTestUtils.cpp:1676 #9 0xe04345 in facebook::logdevice::IntegrationTestUtils::Cluster::start(std::vector<short, std::allocator<short> >) logdevice/test/utils/IntegrationTestUtils.cpp:1122 #10 0xdfd2dd in facebook::logdevice::IntegrationTestUtils::ClusterFactory::createOneTry(facebook::logdevice::Configuration const&) logdevice/test/utils/IntegrationTestUtils.cpp:887 #11 0xdf9ece in facebook::logdevice::IntegrationTestUtils::ClusterFactory::create(facebook::logdevice::Configuration const&) logdevice/test/utils/IntegrationTestUtils.cpp:1423 #12 0xdf68e3 in facebook::logdevice::IntegrationTestUtils::ClusterFactory::create(int) logdevice/test/utils/IntegrationTestUtils.cpp:694 #13 0x6bbe76 in (anonymous namespace)::DataSizeTest::init() logdevice/test/DataSizeTest.cpp:132 #14 0x6cab7d in (anonymous namespace)::DataSizeTest_RestartNode_Test::TestBody() logdevice/test/DataSizeTest.cpp:336 #15 0x366771c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2417:27 #16 0x366771c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2453:52 #17 0x3658ad1 in testing::Test::Run() (.part.595) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2490:50 #18 0x3658f51 in testing::Test::Run() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2481:3 #19 0x3658f51 in testing::TestInfo::Run() (.part.596) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2677:14 #20 0x36590e4 in testing::TestInfo::Run() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2801:21 #21 0x36590e4 in testing::TestCase::Run() (.part.597) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2802:31 #22 0x365a664 in testing::TestCase::Run() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:4737:45 #23 0x365a664 in testing::internal::UnitTestImpl::RunAllTests() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:4739:46 #24 0x365a904 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2417:27 #25 0x365a904 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2453:52 #26 0x365a904 in testing::UnitTest::Run() /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:4333:55 #27 0x2c74173 in RUN_ALL_TESTS() third-party-buck/platform007/build/googletest/include/gtest/gtest.h:2255 #28 0x2c74173 in main common/gtest/LightMain.cpp:19 #29 0x7f711212b1a5 in __libc_start_main /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/csu/libc-start.c:308:16 #30 0x4f0c99 in _start /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/sysdeps/x86_64/start.S:120 Reviewed By: MohamedBassem Differential Revision: D17825971 fbshipit-source-id: f68ee49d5b9a079e55c5bed6e2e6b28dbb161c9b

Summary: I feel bad for such a hacky fix, but (a) I don't have better simple ideas, (b) it's only used in tests. Stack trace: (gdb) bt #0 raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x000000000096052a in facebook::logdevice::handle_fatal_signal (sig=<optimized out>) at logdevice/server/fatalsignal.cpp:58 #2 <signal handler called> #3 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #4 0x00007f0b92e3b935 in __GI_abort () at abort.c:90 #5 0x00007f0b940fc395 in __gnu_cxx::__verbose_terminate_handler () at ../../.././libstdc++-v3/libsupc++/vterminate.cc:95 #6 0x00007f0b940fa016 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:47 #7 0x00007f0b940fa061 in std::terminate () at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:57 #8 0x00007f0b940fa33b in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x26e5800 <typeinfo for std::bad_function_call>, dest=0x415cf0 <std::bad_function_call::~bad_function_call()plt>) at ../../.././libstdc++-v3/libsupc++/eh_throw.cc:93 #9 0x00000000004b5f92 in folly::throw_exception<std::bad_function_call> (ex=...) at folly/lang/Exception.h:36 #10 0x00000000004b5f57 in folly::detail::throw_exception_<std::bad_function_call> () at folly/lang/Exception.h:64 #11 0x0000000000d70019 in folly::throw_exception<std::bad_function_call> () at folly/lang/Exception.h:82 #12 folly::detail::function::FunctionTraits<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>::uninitCall(folly::detail::function::Data&, facebook::logdevice::E&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) () at folly/Function.h:372 #13 0x0000000000f023f8 in folly::detail::function::FunctionTraits<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>::operator()(facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (this=<optimized out>, args=..., args=...) at folly/Function.h:377 #14 facebook::logdevice::FileBasedVersionedConfigStore::getConfig(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, folly::Function<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>, folly::Optional<facebook::logdevice::vcs_config_version_t>) const ( this=<optimized out>, key=..., cb=..., base_version=...) at logdevice/common/FileBasedVersionedConfigStore.cpp:258 #15 0x0000000000d76019 in facebook::logdevice::configuration::nodes::VersionedNodesConfigurationStore<facebook::logdevice::FileBasedVersionedConfigStore>::getConfig(folly::Function<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>, folly::Optional<facebook::logdevice::vcs_config_version_t>) const ( this=<optimized out>, cb=..., base_version=...) at logdevice/common/configuration/nodes/NodesConfigurationStore.h:88 #16 0x0000000000d6de2f in facebook::logdevice::configuration::nodes::ncm::Dependencies::readFromStore (this=0x7f0b91967fa0, should_do_consistent_config_fetch=false) at logdevice/common/configuration/nodes/NodesConfigurationManagerDependencies.cpp:342 #17 0x0000000000d5e64e in facebook::logdevice::configuration::nodes::NodesConfigurationManager::onHeartBeat (this=0x7f0b90d0df10) at logdevice/common/configuration/nodes/NodesConfigurationManager.cpp:538 #18 0x0000000000d73073 in facebook::logdevice::configuration::nodes::ncm::Dependencies::scheduleHeartBeat()::$_4::operator()() const (this=<optimized out>) at logdevice/common/configuration/nodes/NodesConfigurationManagerDependencies.cpp:365 #19 std::_Function_handler<void (), facebook::logdevice::configuration::nodes::ncm::Dependencies::scheduleHeartBeat()::$_4>::_M_invoke(std::_Any_data const&) (__functor=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:316 #20 0x0000000000e250a6 in std::function<void ()>::operator()() const (this=<optimized out>) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 #21 facebook::logdevice::(anonymous namespace)::WheelTimerDispatchImpl::makeWheelTimerInternalExecutor(facebook::logdevice::Worker*)::$_0::operator()()::{lambda()#1}::operator()() const (this=0x7f0b8bf34070) at logdevice/common/Timer.cpp:123 #22 folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::(anonymous namespace)::WheelTimerDispatchImpl::makeWheelTimerInternalExecutor(facebook::logdevice::Worker*)::$_0::operator()()::{lambda()#1}>(folly::detail::function::Data&) (p=...) at folly/Function.h:361 #23 0x0000000000e63602 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=<optimized out>) at folly/Function.h:377 #24 facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9::operator()() (this=0x7f0b8bf34060) at logdevice/common/Worker.cpp:1373 #25 folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9>(folly::detail::function::Data&) (p=...) at folly/Function.h:368 #26 0x00000000010d7e75 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7f0b8bf34060) at folly/Function.h:377 #27 facebook::logdevice::EventLoopTaskQueue::executeTasks (this=0x7f0b74023000, tokens=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:154 #28 0x00000000010d7699 in facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1::operator()(unsigned int) const (this=<optimized out>, n=1) at logdevice/common/EventLoopTaskQueue.cpp:101 #29 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&, unsigned int) (this=<optimized out>, func=..., maxBatchSize=<optimized out>) at logdevice/common/LifoEventSem.h:351 #30 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler (this=0x7f0b74023000) at logdevice/common/EventLoopTaskQueue.cpp:106 #31 0x0000000001a907f2 in event_process_active_single_queue (base=<optimized out>, activeq=0x7f0b74020010, max_to_process=2147483647, endtime=0x0) at logdevice/external/libevent-2.1.3-alpha/event.c:1449 #32 0x0000000001a8efae in event_process_active (base=<optimized out>) at logdevice/external/libevent-2.1.3-alpha/event.c:1599 #33 ld_event_base_loop (base=0x7f0b7401b000, flags=<optimized out>) at logdevice/external/libevent-2.1.3-alpha/event.c:1819 #34 0x000000000176b5f1 in facebook::logdevice::EvBaseLegacy::loop (this=0x7f0b74000018) at logdevice/common/libevent/EvBaseLegacy.cpp:58 #35 0x00000000010d5a79 in facebook::logdevice::EventLoop::run (this=<optimized out>) at logdevice/common/EventLoop.cpp:140 #36 0x00000000016fa223 in std::function<void ()>::operator()() const (this=0x7f0b90d0c520) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 #37 facebook::logdevice::thread_func (arg=0x7f0b90d0c520) at logdevice/common/PThread.cpp:9 #38 0x00007f0b936136b6 in start_thread (arg=0x7f0b74dff700) at pthread_create.c:465 #39 0x00007f0b92f2cebf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Reviewed By: gdrane Differential Revision: D17975422 fbshipit-source-id: 499c6aceb577f859e9d4e71006b86f73537221ab

Summary: TSAN report: ================== WARNING: ThreadSanitizer: data race (pid=3276373) Write of size 8 at 0x7ba000000330 by thread T10: #0 close at ??:? #1 facebook::logdevice::detail::SynchronizationFd::~SynchronizationFd() at ./logdevice/common/LifoEventSem.cpp:59 #2 facebook::logdevice::detail::FdBaton<std::atomic>::~FdBaton() at ./logdevice/common/LifoEventSem.h:68 #3 std::default_delete<facebook::logdevice::detail::FdBaton<std::atomic> >::operator()(facebook::logdevice::detail::FdBaton<std::atomic>*) const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:78 #4 std::unique_ptr<facebook::logdevice::detail::FdBaton<std::atomic>, std::default_delete<facebook::logdevice::detail::FdBaton<std::atomic> > >::~unique_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:268 #5 facebook::logdevice::detail::LifoFdBaton<std::atomic>::~LifoFdBaton() at ./logdevice/common/LifoEventSem.h:155 #6 folly::detail::LifoSemNode<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic>::destroy() at ./folly/synchronization/LifoSem.h:195 #7 folly::detail::LifoSemNodeRecycler<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic>::operator()(folly::detail::LifoSemNode<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic>*) const at ./folly/synchronization/LifoSem.h:213 #8 std::unique_ptr<folly::detail::LifoSemNode<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic>, folly::detail::LifoSemNodeRecycler<facebook::logdevice::detail::LifoFdBaton<std::atomic>, std::atomic> >::~unique_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:268 #9 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::~AsyncWaiter() at ./logdevice/common/LifoEventSem.h:380 #10 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::~AsyncWaiter() at ./logdevice/common/LifoEventSem.h:358 #11 folly::DelayedDestruction::onDelayedDestroy(bool) at ./folly/io/async/DelayedDestruction.h:115 #12 folly::DelayedDestruction::destroy() at ./folly/io/async/DelayedDestruction.h:55 #13 folly::DelayedDestruction::Destructor::operator()(folly::DelayedDestruction*) const at ./folly/io/async/DelayedDestruction.h:70 #14 std::unique_ptr<facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter, folly::DelayedDestruction::Destructor>::reset(facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:376 #15 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler() at ./logdevice/common/EventLoopTaskQueue.cpp:113 #16 facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0::operator()() const at ./logdevice/common/EventLoopTaskQueue.cpp:36 #17 void folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0>(folly::detail::function::Data&) at ./folly/Function.h:361 #18 folly::detail::function::FunctionTraits<void ()>::operator()() at ./folly/Function.h:377 #19 facebook::logdevice::EventLegacy::evCallback(int, short, void*) at ./logdevice/common/libevent/EventLegacy.cpp:41 #20 event_persist_closure at ./logdevice/external/libevent-2.1.3-alpha/event.c:1452 #21 event_process_active_single_queue at ./logdevice/external/libevent-2.1.3-alpha/event.c:1508 #22 event_process_active at ./logdevice/external/libevent-2.1.3-alpha/event.c:1596 #23 ld_event_base_loop at ./logdevice/external/libevent-2.1.3-alpha/event.c:1819 #24 facebook::logdevice::EvBaseLegacy::loop() at ./logdevice/common/libevent/EvBaseLegacy.cpp:58 #25 facebook::logdevice::EvBase::loop() at ./logdevice/common/libevent/LibEventCompatibility.cpp:52 #26 facebook::logdevice::EventLoop::run() at ./logdevice/common/EventLoop.cpp:140 #27 facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType)::$_0::operator()() const at ./logdevice/common/EventLoop.cpp:81 #28 std::_Function_handler<void (), facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType)::$_0>::_M_invoke(std::_Any_data const&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:316 #29 std::function<void ()>::operator()() const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 #30 facebook::logdevice::thread_func(void*) at ./logdevice/common/PThread.cpp:9 #31 __tsan_thread_start_func at tsan.c:? Previous read of size 8 at 0x7ba000000330 by main thread: #0 write at ??:? #1 facebook::logdevice::detail::SynchronizationFd::write() at ./logdevice/common/LifoEventSem.cpp:75 #2 post at ./logdevice/common/LifoEventSem.h:72 #3 facebook::logdevice::EventLoopTaskQueue::shutdown() at ./logdevice/common/EventLoopTaskQueue.cpp:68 #4 facebook::logdevice::Processor::shutdown() at ./logdevice/common/Processor.cpp:585 #5 facebook::logdevice::ClientImpl::~ClientImpl() at ./logdevice/lib/ClientImpl.cpp:314 #6 void __gnu_cxx::new_allocator<facebook::logdevice::ClientImpl>::destroy<facebook::logdevice::ClientImpl>(facebook::logdevice::ClientImpl*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/ext/new_allocator.h:157 #7 void std::allocator_traits<std::allocator<facebook::logdevice::ClientImpl> >::destroy<facebook::logdevice::ClientImpl>(std::allocator<facebook::logdevice::ClientImpl>&, facebook::logdevice::ClientImpl*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/alloc_traits.h:488 #8 std::_Sp_counted_ptr_inplace<facebook::logdevice::ClientImpl, std::allocator<facebook::logdevice::ClientImpl>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:537 #9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:156 #10 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:686 #11 std::__shared_ptr<facebook::logdevice::Client, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:1125 #12 std::shared_ptr<facebook::logdevice::Client>::~shared_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr.h:93 #13 SequencerIntegrationTest_ExpandShrinkReactivationDelayTest_Test::TestBody() at ./logdevice/test/SequencerIntegrationTest.cpp:580 #14 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2417 #15 main at ./common/gtest/LightMain.cpp:19 #16 ?? ??:0 During shutdown, a write() to fd triggers the worker thread to wake up and destroy the same fd. So, technically, fd might be destroyed before the write() returns, while write() is still accessing the fd. This diff adds some synchronization to make sure fd close() starts only after write() returns. #thanks Nick Sukhanov for figuring that out. Reviewed By: mcrnic, tau0 Differential Revision: D18018223 fbshipit-source-id: ddbff95257dd39c835ad6e8733ba0bf2ed432fa0

…eptions Summary: To turn this: terminate called after throwing an instance of 'std::bad_function_call' what(): bad_function_call handle_fatal_signal(): Caught coredump signal 6 @ 000000000095fb9f facebook::logdevice::handle_fatal_signal(int) ./logdevice/server/fatalsignal.cpp:106 @ 00007f366ef8bb0f (unknown) @ 00007f366e7bf14f __GI_raise @ 00007f366e7a8934 __GI_abort @ 00007f366fa69394 __gnu_cxx::__verbose_terminate_handler() @ 00007f366fa67015 __cxxabiv1::__terminate(void (*)()) @ 00007f366fa67060 std::terminate() @ 00007f366fa6738c __cxa_rethrow @ 00000000010d7aa4 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler() ./logdevice/common/LifoEventSem.h:341 -> ./logdevice/common/EventLoopTaskQueue.cpp @ 0000000001a907e1 event_process_active_single_queue ./logdevice/external/libevent-2.1.3-alpha/event.c:0 @ 0000000001a8ef9d ld_event_base_loop ./logdevice/external/libevent-2.1.3-alpha/event.c:0 @ 000000000176b5e0 facebook::logdevice::EvBaseLegacy::loop() ./logdevice/common/libevent/EvBaseLegacy.cpp:58 @ 00000000010d5a78 facebook::logdevice::EventLoop::run() ./logdevice/common/EventLoop.cpp:140 @ 00000000016fa212 facebook::logdevice::thread_func(void*) ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 -> ./logdevice/common/PThread.cpp @ 00007f366ef806b5 start_thread @ 00007f366e899ebe __GI___clone into this: terminate called after throwing an instance of 'std::bad_function_call' what(): bad_function_call handle_fatal_signal(): Caught coredump signal 6 @ 000000000095fb9f facebook::logdevice::handle_fatal_signal(int) ./logdevice/server/fatalsignal.cpp:106 @ 00007f4f03d07b0f (unknown) @ 00007f4f0353b14f __GI_raise @ 00007f4f03524934 __GI_abort @ 00007f4f047e5394 __gnu_cxx::__verbose_terminate_handler() @ 00007f4f047e3015 __cxxabiv1::__terminate(void (*)()) @ 00007f4f047e3060 std::terminate() @ 00007f4f047e333a __cxa_throw @ 00000000004b5f91 void folly::throw_exception<std::bad_function_call>(std::bad_function_call&&) ./folly/lang/Exception.h:36 -> ./logdevice/server/Server.cpp @ 00000000004b5f56 void folly::detail::throw_exception_<std::bad_function_call>() ./folly/lang/Exception.h:64 -> ./logdevice/server/Server.cpp @ 0000000000d70018 folly::detail::function::FunctionTraits<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>::uninitCall(folly::detail::function::Data&, facebook::logdevice::E&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) ./folly/lang/Exception.h:82 -> ./logdevice/common/configuration/nodes/NodesConfigurationManagerDependencies.cpp @ 0000000000f023f7 facebook::logdevice::FileBasedVersionedConfigStore::getConfig(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, folly::Function<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>, folly::Optional<facebook::logdevice::vcs_config_version_t>) const ./folly/Function.h:377 -> ./logdevice/common/FileBasedVersionedConfigStore.cpp @ 0000000000d76018 facebook::logdevice::configuration::nodes::VersionedNodesConfigurationStore<facebook::logdevice::FileBasedVersionedConfigStore>::getConfig(folly::Function<void (facebook::logdevice::E, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>, folly::Optional<facebook::logdevice::vcs_config_version_t>) const ./logdevice/common/configuration/nodes/NodesConfigurationStore.h:88 -> ./logdevice/common/configuration/nodes/NodesConfigurationManagerFactory.cpp @ 0000000000d6de2e facebook::logdevice::configuration::nodes::ncm::Dependencies::readFromStore(bool) ./logdevice/common/configuration/nodes/NodesConfigurationManagerDependencies.cpp:342 @ 0000000000d5e64d facebook::logdevice::configuration::nodes::NodesConfigurationManager::onHeartBeat() ./logdevice/common/configuration/nodes/NodesConfigurationManager.cpp:538 @ 0000000000d73072 std::_Function_handler<void (), facebook::logdevice::configuration::nodes::ncm::Dependencies::scheduleHeartBeat()::$_4>::_M_invoke(std::_Any_data const&) ./logdevice/common/configuration/nodes/NodesConfigurationManagerDependencies.cpp:365 @ 0000000000e250a5 void folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::(anonymous namespace)::WheelTimerDispatchImpl::makeWheelTimerInternalExecutor(facebook::logdevice::Worker*)::$_0::operator()()::{lambda()#1}>(folly::detail::function::Data&) ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 -> ./logdevice/common/Timer.cpp @ 0000000000e63601 void folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9>(folly::detail::function::Data&) ./folly/Function.h:377 -> ./logdevice/common/Worker.cpp @ 00000000010d7e74 facebook::logdevice::EventLoopTaskQueue::executeTasks(unsigned int) ./folly/Function.h:377 -> ./logdevice/common/EventLoopTaskQueue.cpp @ 00000000010d7698 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler() ./logdevice/common/EventLoopTaskQueue.cpp:101 @ 0000000001a907f1 event_process_active_single_queue ./logdevice/external/libevent-2.1.3-alpha/event.c:0 @ 0000000001a8efad ld_event_base_loop ./logdevice/external/libevent-2.1.3-alpha/event.c:0 @ 000000000176b5f0 facebook::logdevice::EvBaseLegacy::loop() ./logdevice/common/libevent/EvBaseLegacy.cpp:58 @ 00000000010d5a78 facebook::logdevice::EventLoop::run() ./logdevice/common/EventLoop.cpp:140 @ 00000000016fa222 facebook::logdevice::thread_func(void*) ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 -> ./logdevice/common/PThread.cpp @ 00007f4f03cfc6b5 start_thread @ 00007f4f03615ebe __GI___clone Reviewed By: tau0 Differential Revision: D17974875 fbshipit-source-id: 48dbcae2ddd9ebe17bb5c005318c9598e8fd1a0b

Summary: A bug very similar to D18018223: 1. ClientReadStream detects the last gap of a log (up to until_lsn_), calls ReaderBridgeImpl's gap callback. The ClientReadStream will self-destruct right after callback returns. 2. The callback puts the gap on ReaderImpl's queue and begins to consider whether or not it needs to also post to a condition variable. 3. ReaderImpl pops from the queue, sees that lsn is equal to until_lsn, and forgets the stream (i.e. destructor won't wait for this stream to stop) because it's going to self-destruct anyway. 4. The gap is returned to the user, who also notices that until lsn was reached and calls ~ReaderImpl() right away. 5. ReaderImpl is deallocated, along with its ReaderBridgeImpl. 6. ReaderBridgeImpl's gap callback from step 1 is still running, and now it finally got around to posting on ReaderImpl's condition variable, which is freed by now. This diff fixes it in the same simple way as D18018223: by having a counter of in-flight callbacks and making ~ReaderImpl() wait for the counter to go to zero. It can probably be done more efficiently, but I couldn't come up with anything simple in a few minutes of thinking. TSAN report: ================== [122/429]WARNING: ThreadSanitizer: data race (pid=309956) Write of size 8 at 0x7b0400014248 by main thread: #0 operator delete(void*) at ??:? #1 facebook::logdevice::ReaderBridgeImpl::~ReaderBridgeImpl() at ./logdevice/common/ReaderImpl.cpp:33 #2 std::default_delete<facebook::logdevice::ReaderBridge>::operator()(facebook::logdevice::ReaderBridge*) const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:78 #3 std::unique_ptr<facebook::logdevice::ReaderBridge, std::default_delete<facebook::logdevice::ReaderBridge> >::~unique_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:268 #4 facebook::logdevice::ReaderImpl::~ReaderImpl() at ./logdevice/common/ReaderImpl.cpp:165 #5 facebook::logdevice::ReaderImpl::~ReaderImpl() at ./logdevice/common/ReaderImpl.cpp:110 #6 std::default_delete<facebook::logdevice::Reader>::operator()(facebook::logdevice::Reader*) const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:78 #7 std::unique_ptr<facebook::logdevice::Reader, std::default_delete<facebook::logdevice::Reader> >::~unique_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:268 #8 RebuildingTest_RebuildMetaDataLogsOfDeletedLogs_Test::TestBody()::$_8::operator()(facebook::logdevice::logid_t) const at ./logdevice/test/RebuildingTest.cpp:2134 #9 RebuildingTest_RebuildMetaDataLogsOfDeletedLogs_Test::TestBody() at ./logdevice/test/RebuildingTest.cpp:2136 #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2473 (discriminator 4) #11 main at ./common/gtest/LightMain.cpp:19 #12 ?? ??:0 Previous read of size 8 at 0x7b0400014248 by thread T50: #0 facebook::logdevice::ReaderBridgeImpl::maybeWakeConsumer() at ./logdevice/common/ReaderImpl.cpp:785 #1 facebook::logdevice::ReaderBridgeImpl::onEntry(facebook::logdevice::ReaderImpl::QueueEntry&&, bool) at ./logdevice/common/ReaderImpl.cpp:747 #2 facebook::logdevice::ReaderBridgeImpl::onGapRecord(facebook::logdevice::read_stream_id_t, facebook::logdevice::GapRecord, bool) at ./logdevice/common/ReaderImpl.cpp:781 #3 facebook::logdevice::ClientReadStream::deliverGap(facebook::logdevice::GapType, unsigned long, unsigned long) at ./logdevice/common/client_read_stream/ClientReadStream.cpp:3398 #4 facebook::logdevice::ClientReadStream::handleBridgeRecord(unsigned long, facebook::logdevice::ClientReadStreamRecordState*) at ./logdevice/common/client_read_stream/ClientReadStream.cpp:3189 #5 facebook::logdevice::ClientReadStream::handleRecord(facebook::logdevice::ClientReadStreamRecordState*, bool, bool) at ./logdevice/common/client_read_stream/ClientReadStream.cpp:3116 #6 facebook::logdevice::ClientReadStream::findGapsAndRecords(bool, bool)::$_6::operator()() const at ./logdevice/common/client_read_stream/ClientReadStream.cpp:1475 #7 facebook::logdevice::ClientReadStream::findGapsAndRecords(bool, bool) at ./logdevice/common/client_read_stream/ClientReadStream.cpp:1525 #8 facebook::logdevice::ClientReadStream::onDataRecord(facebook::logdevice::ShardID, std::unique_ptr<facebook::logdevice::DataRecordOwnsPayload, std::default_delete<facebook::logdevice::DataRecordOwnsPayload> >) at ./logdevice/common/ client_read_stream/ClientReadStream.cpp:1122 #9 facebook::logdevice::AllClientReadStreams::onDataRecord(facebook::logdevice::ShardID, facebook::logdevice::logid_t, facebook::logdevice::read_stream_id_t, std::unique_ptr<facebook::logdevice::DataRecordOwnsPayload, std::default_del ete<facebook::logdevice::DataRecordOwnsPayload> >&&) at ./logdevice/common/client_read_stream/AllClientReadStreams.cpp:58 #10 facebook::logdevice::RECORD_Message::onReceived(facebook::logdevice::Address const&) at ./logdevice/common/protocol/RECORD_Message.cpp:273 #11 facebook::logdevice::ClientMessageDispatch::onReceivedImpl(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&) at ./logdevice/lib/ClientMessageDispatch.cpp:61 #12 facebook::logdevice::MessageDispatch::onReceived(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&) at ./logdevice/common/protocol/MessageDispatch.cpp:24 #13 facebook::logdevice::Message::Disposition std::__invoke_impl<facebook::logdevice::Message::Disposition, facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*&)(facebook::logdevice::Message*, facebook:: logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&), facebook::logdevice::MessageDispatch*&, facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&>(std::__invoke_ memfun_deref, facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*&)(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&), facebook::logdevice::Me ssageDispatch*&, facebook::logdevice::Message*&&, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/invoke.h:73 #14 std::__invoke_result<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*&)(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&), facebook ::logdevice::MessageDispatch*&, facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&>::type std::__invoke<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispat ch::*&)(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&), facebook::logdevice::MessageDispatch*&, facebook::logdevice::Message*, facebook::logdevice::Address const&, facebo ok::logdevice::PrincipalIdentity&>(facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*&)(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&), fa cebook::logdevice::MessageDispatch*&, facebook::logdevice::Message*&&, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/invoke.h:95 #15 facebook::logdevice::Message::Disposition std::_Bind<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*(facebook::logdevice::MessageDispatch*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Place holder<3>))(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&)>::__call<facebook::logdevice::Message::Disposition, facebook::logdevice::Message*&&, facebook::logdevice::Addre ss const&, facebook::logdevice::PrincipalIdentity&, 0ul, 1ul, 2ul, 3ul>(std::tuple<facebook::logdevice::Message*&&, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/functional:467 #16 facebook::logdevice::Message::Disposition std::_Bind<facebook::logdevice::Message::Disposition (facebook::logdevice::MessageDispatch::*(facebook::logdevice::MessageDispatch*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Place holder<3>))(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&)>::operator()<facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::Principal Identity&, facebook::logdevice::Message::Disposition>(facebook::logdevice::Message*&&, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/function al:549 ... Reviewed By: mcrnic Differential Revision: D18135318 fbshipit-source-id: 41deadbe12b60307bcbbe800ea1309477ca1c555

Summary: TSAN report: ================== WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=203663) Read of size 8 at 0x7b8800001810 by thread T58: #0 facebook::logdevice::UpdateShardAuthoritativeMapRequest::broadcastToAllWorkers(facebook::logdevice::ShardAuthoritativeStatusMap const&) at ./logdevice/common/ShardAuthoritativeStatusMap.cpp:170 #1 facebook::logdevice::SHARD_STATUS_UPDATE_Message::onReceived(facebook::logdevice::Address const&) at ./logdevice/common/protocol/SHARD_STATUS_UPDATE_Message.cpp:56 #2 facebook::logdevice::ClientMessageDispatch::onReceivedImpl(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&) at ./logdevice/lib/ClientMessageDispatch.cpp:61 #3 facebook::logdevice::MessageDispatch::onReceived(facebook::logdevice::Message*, facebook::logdevice::Address const&, facebook::logdevice::PrincipalIdentity const&) at ./logdevice/common/protocol/MessageDispatch.cpp:24 #4 facebook::logdevice::SocketDependencies::onReceived(facebook::logdevice::Message*, facebook::logdevice::Address const&, std::shared_ptr<facebook::logdevice::PrincipalIdentity>, facebook::logdevice::ResourceBudget::Token) at ./logdevice/common/SocketDependencies.cpp:334 #5 facebook::logdevice::Socket::dispatchMessageBody(facebook::logdevice::ProtocolHeader, std::unique_ptr<folly::IOBuf, std::default_delete<folly::IOBuf> >) at ./logdevice/common/Socket.cpp:2136 #6 facebook::logdevice::Connection::dispatchMessageBody(facebook::logdevice::ProtocolHeader, std::unique_ptr<folly::IOBuf, std::default_delete<folly::IOBuf> >) at ./logdevice/common/Connection.cpp:375 #7 facebook::logdevice::Socket::onBytesAvailable(bool) at ./logdevice/common/Socket.cpp:657 #8 facebook::logdevice::Socket::dataReadCallback(bufferevent*, void*, short) at ./logdevice/common/Socket.cpp:717 #9 void facebook::logdevice::BufferEventHandlerImpl<&facebook::logdevice::Socket::dataReadCallback, &facebook::logdevice::(anonymous namespace)::bufferevent_preflight_noop, &facebook::logdevice::(anonymous namespace)::bufferevent_postflight_noop>(bufferevent*, short, void*) at ./logdevice/common/EventHandler.h:103 #10 void facebook::logdevice::BufferEventHandler<&facebook::logdevice::Socket::dataReadCallback, &facebook::logdevice::(anonymous namespace)::bufferevent_preflight_noop, &facebook::logdevice::(anonymous namespace)::bufferevent_postflight_noop>(bufferevent*, void*) at ./logdevice/common/EventHandler.h:126 #11 ld_bufferevent_run_readcb_ at ./logdevice/external/libevent-2.1.3-alpha/bufferevent.c:233 #12 bufferevent_readcb at ./logdevice/external/libevent-2.1.3-alpha/bufferevent_sock.c:188 #13 event_persist_closure at ./logdevice/external/libevent-2.1.3-alpha/event.c:1452 #14 event_process_active_single_queue at ./logdevice/external/libevent-2.1.3-alpha/event.c:1508 #15 event_process_active at ./logdevice/external/libevent-2.1.3-alpha/event.c:1596 #16 ld_event_base_loop at ./logdevice/external/libevent-2.1.3-alpha/event.c:1819 #17 facebook::logdevice::EvBaseLegacy::loop() at ./logdevice/common/libevent/EvBaseLegacy.cpp:58 #18 facebook::logdevice::EvBase::loop() at ./logdevice/common/libevent/LibEventCompatibility.cpp:52 #19 facebook::logdevice::EventLoop::run() at ./logdevice/common/EventLoop.cpp:156 #20 facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType, bool)::$_0::operator()() const at ./logdevice/common/EventLoop.cpp:86 #21 std::_Function_handler<void (), facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType, bool)::$_0>::_M_invoke(std::_Any_data const&) at ./third-party-b uck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:316 #22 std::function<void ()>::operator()() const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 #23 facebook::logdevice::thread_func(void*) at ./logdevice/common/PThread.cpp:18 #24 __tsan_thread_start_func at tsan.c:? Previous write of size 8 at 0x7b8800001810 by main thread: #0 facebook::logdevice::Processor::~Processor() at ./logdevice/common/Processor.cpp:321 #1 facebook::logdevice::ClientProcessor::~ClientProcessor() at ./logdevice/lib/ClientProcessor.h:22 #2 void __gnu_cxx::new_allocator<facebook::logdevice::ClientProcessor>::destroy<facebook::logdevice::ClientProcessor>(facebook::logdevice::ClientProcessor*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/ext/new_allocator.h:157 #3 void std::allocator_traits<std::allocator<facebook::logdevice::ClientProcessor> >::destroy<facebook::logdevice::ClientProcessor>(std::allocator<facebook::logdevice::ClientProcessor>&, facebook::logdevice::ClientProcessor*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/alloc_traits.h:488 #4 std::_Sp_counted_ptr_inplace<facebook::logdevice::ClientProcessor, std::allocator<facebook::logdevice::ClientProcessor>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:537 #5 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:156 #6 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:686 #7 std::__shared_ptr<facebook::logdevice::ClientProcessor, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:1125 #8 std::shared_ptr<facebook::logdevice::ClientProcessor>::~shared_ptr() at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr.h:93 #9 facebook::logdevice::ClientImpl::ClientImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<facebook::logdevice::UpdateableConfig>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >&&, std::shared_ptr<facebook::logdevice::PluginRegistry>) at ./logdevice/lib/ClientImpl.cpp:307 #10 void __gnu_cxx::new_allocator<facebook::logdevice::ClientImpl>::construct<facebook::logdevice::ClientImpl, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >, std::shared_ptr<facebook::logdevice::PluginRegistry>&>(facebook::logdevice::ClientImpl*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >&&, std::shared_ptr<facebook::logdevice::PluginRegistry>&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/ext/new_allocator.h:153 #11 void std::allocator_traits<std::allocator<facebook::logdevice::ClientImpl> >::construct<facebook::logdevice::ClientImpl, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >, std::shared_ptr<facebook::logdevice::PluginRegistry>&>(std::allocator<facebook::logdevice::ClientImpl>&, facebook::logdevice::ClientImpl*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >&&, std::shared_ptr<facebook::logdevice::PluginRegistry>&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/alloc_traits.h:476 #12 std::_Sp_counted_ptr_inplace<facebook::logdevice::ClientImpl, std::allocator<facebook::logdevice::ClientImpl>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >, std::shared_ptr<facebook::logdevice::PluginRegistry>&>(std::allocator<facebook::logdevice::ClientImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >&&, std::shared_ptr<facebook::logdevice::PluginRegistry>&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:528 #13 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<facebook::logdevice::ClientImpl, std::allocator<facebook::logdevice::ClientImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >, std::shared_ptr<facebook::logdevice::PluginRegistry>&>(std::_Sp_make_shared_tag, facebook::logdevice::ClientImpl*, std::allocator<facebook::logdevice::ClientImpl> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >&&, std::shared_ptr<facebook::logdevice::PluginRegistry>&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:639 #14 std::__shared_ptr<facebook::logdevice::ClientImpl, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<facebook::logdevice::ClientImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >, std::shared_ptr<facebook::logdevice::PluginRegistry>&>(std::_Sp_make_shared_tag, std::allocator<facebook::logdevice::ClientImpl> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >&&, std::shared_ptr<facebook::logdevice::PluginRegistry>&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr_base.h:1296 #15 std::shared_ptr<facebook::logdevice::ClientImpl>::shared_ptr<std::allocator<facebook::logdevice::ClientImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >, std::shared_ptr<facebook::logdevice::PluginRegistry>&>(std::_Sp_make_shared_tag, std::allocator<facebook::logdevice::ClientImpl> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >&&, std::shared_ptr<facebook::logdevice::PluginRegistry>&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr.h:344 #16 std::shared_ptr<facebook::logdevice::ClientImpl> std::allocate_shared<facebook::logdevice::ClientImpl, std::allocator<facebook::logdevice::ClientImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >, std::shared_ptr<facebook::logdevice::PluginRegistry>&>(std::allocator<facebook::logdevice::ClientImpl> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >&&, std::shared_ptr<facebook::logdevice::PluginRegistry>&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr.h:690 #17 std::shared_ptr<facebook::logdevice::ClientImpl> std::make_shared<facebook::logdevice::ClientImpl, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >, std::shared_ptr<facebook::logdevice::PluginRegistry>&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<facebook::logdevice::UpdateableConfig>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [1], std::chrono::duration<long, std::ratio<1l, 1000l> >&, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >&&, std::shared_ptr<facebook::logdevice::PluginRegistry>&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/shared_ptr.h:706 #18 facebook::logdevice::IntegrationTestUtils::Cluster::createClient(std::chrono::duration<long, std::ratio<1l, 1000l> >, std::unique_ptr<facebook::logdevice::ClientSettings, std::default_delete<facebook::logdevice::ClientSettings> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) at ./logdevice/test/utils/IntegrationTestUtils.cpp:2461 #19 FailureDetectorIntegrationTest_StartingState_Test::TestBody() at ./logdevice/test/FailureDetectorIntegrationTest.cpp:563 #20 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2417 #21 main at ./common/gtest/LightMain.cpp:19 #22 ?? ??:0 Reviewed By: tau0 Differential Revision: D18319700 fbshipit-source-id: 2d1644b14ea97045fa7308f59908aaf160ebe216

Summary: When creating various components in Processor::init() (watchdog thread, health monitor, failure detector, etc), at first create all of them in a dormant state, then start all of them. In particular, pause all worker threads until later in the startup sequence. This prevents race conditions when the parts initialized early may try to access parts of the Processor that are not initialized yet. In particular, fixes this race when assigning Processor::failure_detector_ in a unit test (it's a test-only code path, but afaict the same thing can happen in server): ================== WARNING: ThreadSanitizer: data race (pid=886149) Write of size 8 at 0x7b88001dfe10 by main thread: #0 _ZSt4swapIPN8facebook9logdevice15FailureDetectorEENSt9enable_ifIXsr6__and_ISt6__not_ISt15__is_tuple_likeIT_EESt21is_move_constructibleIS7_ESt18is_move_assignableIS7_EEE5valueEvE4typeERS7_SG_ at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/move.h:199 #1 std::unique_ptr<facebook::logdevice::FailureDetector, std::default_delete<facebook::logdevice::FailureDetector> >::reset(facebook::logdevice::FailureDetector*) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:374 #2 _ZNSt10unique_ptrIN8facebook9logdevice15FailureDetectorESt14default_deleteIS2_EEaSINS1_19MockFailureDetectorES3_IS7_EEENSt9enable_ifIXsr6__and_ISt6__and_IJSt14is_convertibleINS_IT_T0_E7pointerEPS2_ESt6__not_ISt8is_arrayISC_EESt5__or_IJSA_IJSt12is_referenceIS4_ESt7is_sameIS4_SD_EEESA_IJSI_ISO_ESB_ISD_S4_EEEEEEESt13is_assignableIRS4_OSD_EEE5valueERS5_E4typeEOSE_ at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:303 #3 (anonymous namespace)::make_processor_with_detector(short, unsigned long, facebook::logdevice::GossipSettings const&) at ./logdevice/server/test/FailureDetectorTest.cpp:185 #4 (anonymous namespace)::create_processors_and_detectors(unsigned long, facebook::logdevice::GossipSettings const&) at ./logdevice/server/test/FailureDetectorTest.cpp:266 #5 (anonymous namespace)::simulate(unsigned long, facebook::logdevice::GossipSettings const&) at ./logdevice/server/test/FailureDetectorTest.cpp:296 #6 FailureDetectorTest_RandomGossip_Test::TestBody() at ./logdevice/server/test/FailureDetectorTest.cpp:313 #7 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /home/engshare/third-party2/googletest/master/src/googletest/googletest/src/gtest.cc:2417 #8 main at ./common/gtest/LightMain.cpp:19 #9 ?? ??:0 Previous read of size 8 at 0x7b88001dfe10 by thread T79: #0 std::__uniq_ptr_impl<facebook::logdevice::FailureDetector, std::default_delete<facebook::logdevice::FailureDetector> >::_M_ptr() const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:147 #1 std::unique_ptr<facebook::logdevice::FailureDetector, std::default_delete<facebook::logdevice::FailureDetector> >::get() const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:337 #2 facebook::logdevice::ServerWorker::onServerConfigUpdated() at ./logdevice/server/ServerWorker.cpp:190 #3 facebook::logdevice::Worker::onNodesConfigurationUpdated() at ./logdevice/common/Worker.cpp:393 #4 facebook::logdevice::Worker::initializeSubscriptions() at ./logdevice/common/Worker.cpp:541 #5 facebook::logdevice::Worker::setupWorker() at ./logdevice/common/Worker.cpp:582 #6 facebook::logdevice::ServerWorker::setupWorker() at ./logdevice/server/ServerWorker.cpp:201 #7 facebook::logdevice::ServerProcessor::createWorker(folly::Executor::KeepAlive<folly::Executor>, facebook::logdevice::worker_id_t, facebook::logdevice::WorkerType)::$_0::operator()() const at ./logdevice/server/ServerProcessor.cpp:26 #8 void folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::ServerProcessor::createWorker(folly::Executor::KeepAlive<folly::Executor>, facebook::logdevice::worker_id_t, facebook::logdevice::WorkerType)::$_0>(folly::detail::function::Data&) at ./folly/Function.h:376 #9 folly::detail::function::FunctionTraits<void ()>::operator()() at ./folly/Function.h:392 #10 facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9::operator()() at ./logdevice/common/Worker.cpp:1413 #11 void folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9>(folly::detail::function::Data&) at ./folly/Function.h:383 #12 folly::detail::function::FunctionTraits<void ()>::operator()() at ./folly/Function.h:392 #13 facebook::logdevice::EventLoopTaskQueue::executeTasks(unsigned int) at ./logdevice/common/EventLoopTaskQueue.cpp:154 #14 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1::operator()(unsigned int) const at ./logdevice/common/EventLoopTaskQueue.cpp:101 #15 void facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&, unsigned int) at ./logdevice/common/LifoEventSem.h:368 #16 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler() at ./logdevice/common/EventLoopTaskQueue.cpp:106 #17 facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0::operator()() const at ./logdevice/common/EventLoopTaskQueue.cpp:36 #18 void folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0>(folly::detail::function::Data&) at ./folly/Function.h:376 #19 folly::detail::function::FunctionTraits<void ()>::operator()() at ./folly/Function.h:392 #20 facebook::logdevice::EventLegacy::evCallback(int, short, void*) at ./logdevice/common/libevent/EventLegacy.cpp:41 #21 event_persist_closure at ./logdevice/external/libevent-2.1.3-alpha/event.c:1452 #22 event_process_active_single_queue at ./logdevice/external/libevent-2.1.3-alpha/event.c:1508 #23 event_process_active at ./logdevice/external/libevent-2.1.3-alpha/event.c:1596 #24 ld_event_base_loop at ./logdevice/external/libevent-2.1.3-alpha/event.c:1819 #25 facebook::logdevice::EvBaseLegacy::loop() at ./logdevice/common/libevent/EvBaseLegacy.cpp:58 #26 facebook::logdevice::EvBase::loop() at ./logdevice/common/libevent/LibEventCompatibility.cpp:52 W1101 14:22:40.569391 907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 433 msec: NODE_STATE_UPDATED (id: 4294979886), p :LO_PRI #27 facebook::logdevice::EventLoop::run() at ./logdevice/common/EventLoop.cpp:140 W1101 14:22:40.584447 907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 446 msec: NODE_STATE_UPDATED (id: 4294979887), p :LO_PRI #28 facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType)::$_0::operator()() const at ./logdevice/common/EventLoop.cpp:81 #29 std::_Function_handler<void (), facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType)::$_0>::_M_invoke(std::_Any_data const&) at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:316 W1101 14:22:40.603377 907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 468 msec: NODE_STATE_UPDATED (id: 4294979888), p :LO_PRI #30 std::function<void ()>::operator()() const at ./third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 W1101 14:22:40.621593 907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 491 msec: NODE_STATE_UPDATED (id: 4294979892), p :LO_PRI #31 facebook::logdevice::thread_func(void*) at ./logdevice/common/PThread.cpp:18 W1101 14:22:40.645430 907396 [logdevice:WG0] Worker.cpp:1319] processRequest() Request queued for 505 msec: NODE_STATE_UPDATED (id: 4294979893), p :LO_PRI #32 __tsan_thread_start_func at tsan.c:? Reviewed By: mcrnic Differential Revision: D18286463 fbshipit-source-id: 1ea365415f1e9dc8e907eabcbdd8a8249652a16e

Summary: It was a null socket pointer which I don't understand I thought the map should never have that. I checked the server_sockets_ map and did not find a null socket so I am not sure what is happening here. I am adding a critical to get more info ``` (gdb) bt #0 std::chrono::operator==<long, std::ratio<1l, 1000000000l>, long, std::ratio<1l, 1000000000l> > (__lhs=..., __rhs=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/chrono:547 #1 std::chrono::operator==<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__lhs=..., __rhs=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/chrono:754 #2 std::chrono::operator!=<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__lhs=..., __rhs=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/chrono:760 #3 facebook::logdevice::operator!=<std::chrono::_V2::steady_clock, facebook::logdevice::detail::Holder, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, facebook::logdevice::detail::Holder, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (lhs=..., rhs=...) at logdevice/common/Timestamp.h:578 #4 facebook::logdevice::Socket::checkSocketHealth (this=0x0) <========================= *******nullptr****** at logdevice/common/Socket.cpp:2463 #5 0x000000000547779a in facebook::logdevice::Sender::closeSlowSockets ( this=0x7fe209854250) at logdevice/common/Sender.cpp:1612 #6 0x00000000054a3329 in folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::(anonymous namespace)::WheelTimerDispatchImpl::makeWheelTimerInternalExecutor(facebook::logdevice::Worker*)::$_0::operator()()::{lambda()#1}>(folly::detail::function::Data&) (p=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 #7 0x00000000054c3c95 in folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9>(folly::detail::function::Data&) (p=...) at folly/Function.h:416 #8 0x0000000005aba867 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7fe173385c80) at folly/Function.h:416 #9 facebook::logdevice::EventLoopTaskQueue::executeTasks ( this=<optimized out>, tokens=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:154 #10 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1::operator()(unsigned int) const (n=<optimized out>, this=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:101 #11 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&, unsigned int) (this=<optimized out>, func=..., maxBatchSize=<optimized out>) at logdevice/common/LifoEventSem.h:368 #12 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler ( this=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:106 #13 facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0::operator()() const (this=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:36 #14 folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0>(folly::detail::function::Data&) (p=...) at folly/Function.h:387 #15 0x0000000007f8bfb0 in event_process_active_single_queue ( base=0x7fe1e7a1c000, activeq=0x7fe1e7a21010, max_to_process=2147483647, endtime=0x0) at rocketspeed/external/libevent-2.1.3-alpha/event.c:1411 #16 0x0000000007f8b5b8 in ld_event_base_loop (base=0x7fe1e7a1c000, flags=0) at rocketspeed/external/libevent-2.1.3-alpha/event.c:1596 #17 0x000000000784800e in facebook::logdevice::EvBaseLegacy::loop ( this=0x7fe1e7a01018) at logdevice/common/libevent/EvBaseLegacy.cpp:59 #18 0x0000000005ab8795 in std::_Function_handler<void (), facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType, bool)::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at logdevice/common/EventLoop.cpp:156 #19 0x000000000770bd23 in facebook::logdevice::thread_func ( arg=0x7fe21c8b38a0) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 #20 0x00007fe225bbe6b6 in start_thread () from /usr/local/fbcode/platform007/lib/libpthread.so.0 #21 0x00007fe22529febf in clone () from /usr/local/fbcode/platform007/lib/libc.so.6 ``` Reviewed By: al13n321 Differential Revision: D19294475 fbshipit-source-id: e705d61277d2ad08245a8cacbcfcd68c127f5c08

Summary: It was a null socket pointer which I don't understand I thought the map should never have that. I checked the server_sockets_ map and did not find a null socket so I am not sure what is happening here. I am adding a critical to get more info ``` (gdb) bt #0 std::chrono::operator==<long, std::ratio<1l, 1000000000l>, long, std::ratio<1l, 1000000000l> > (__lhs=..., __rhs=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/chrono:547 facebookarchive#1 std::chrono::operator==<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__lhs=..., __rhs=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/chrono:754 facebookarchive#2 std::chrono::operator!=<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__lhs=..., __rhs=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/chrono:760 facebookarchive#3 facebook::logdevice::operator!=<std::chrono::_V2::steady_clock, facebook::logdevice::detail::Holder, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, facebook::logdevice::detail::Holder, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (lhs=..., rhs=...) at logdevice/common/Timestamp.h:578 facebookarchive#4 facebook::logdevice::Socket::checkSocketHealth (this=0x0) <========================= *******nullptr****** at logdevice/common/Socket.cpp:2463 facebookarchive#5 0x000000000547779a in facebook::logdevice::Sender::closeSlowSockets ( this=0x7fe209854250) at logdevice/common/Sender.cpp:1612 facebookarchive#6 0x00000000054a3329 in folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::(anonymous namespace)::WheelTimerDispatchImpl::makeWheelTimerInternalExecutor(facebook::logdevice::Worker*)::$_0::operator()()::{lambda()facebookarchive#1}>(folly::detail::function::Data&) (p=...) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 facebookarchive#7 0x00000000054c3c95 in folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::$_9>(folly::detail::function::Data&) (p=...) at folly/Function.h:416 facebookarchive#8 0x0000000005aba867 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7fe173385c80) at folly/Function.h:416 facebookarchive#9 facebook::logdevice::EventLoopTaskQueue::executeTasks ( this=<optimized out>, tokens=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:154 facebookarchive#10 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1::operator()(unsigned int) const (n=<optimized out>, this=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:101 facebookarchive#11 facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler()::$_1&, unsigned int) (this=<optimized out>, func=..., maxBatchSize=<optimized out>) at logdevice/common/LifoEventSem.h:368 facebookarchive#12 facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler ( this=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:106 facebookarchive#13 facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0::operator()() const (this=<optimized out>) at logdevice/common/EventLoopTaskQueue.cpp:36 facebookarchive#14 folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::EventLoopTaskQueue::EventLoopTaskQueue(facebook::logdevice::EvBase&, unsigned long, std::array<unsigned int, 3ul> const&)::$_0>(folly::detail::function::Data&) (p=...) at folly/Function.h:387 facebookarchive#15 0x0000000007f8bfb0 in event_process_active_single_queue ( base=0x7fe1e7a1c000, activeq=0x7fe1e7a21010, max_to_process=2147483647, endtime=0x0) at rocketspeed/external/libevent-2.1.3-alpha/event.c:1411 facebookarchive#16 0x0000000007f8b5b8 in ld_event_base_loop (base=0x7fe1e7a1c000, flags=0) at rocketspeed/external/libevent-2.1.3-alpha/event.c:1596 facebookarchive#17 0x000000000784800e in facebook::logdevice::EvBaseLegacy::loop ( this=0x7fe1e7a01018) at logdevice/common/libevent/EvBaseLegacy.cpp:59 facebookarchive#18 0x0000000005ab8795 in std::_Function_handler<void (), facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&, facebook::logdevice::EvBase::EvBaseType, bool)::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at logdevice/common/EventLoop.cpp:156 facebookarchive#19 0x000000000770bd23 in facebook::logdevice::thread_func ( arg=0x7fe21c8b38a0) at third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/std_function.h:706 facebookarchive#20 0x00007fe225bbe6b6 in start_thread () from /usr/local/fbcode/platform007/lib/libpthread.so.0 facebookarchive#21 0x00007fe22529febf in clone () from /usr/local/fbcode/platform007/lib/libc.so.6 ``` Reviewed By: al13n321 Differential Revision: D19294475 fbshipit-source-id: e705d61277d2ad08245a8cacbcfcd68c127f5c08

Summary: There are multiple ways of calling Thrift RPC on the client side with respect to how result of the call is delivered: 1. sync: blocking call, client thread is waiting. 2. async: asynchronous, result delivered through callback 3. Future-based: caller gets Future that is fulfilled when response is received 4. SemiFuture: the same as above but with SemiFuture instead of Future 5. coroutine (if enabled): caller gets coro task which is completed with RPC result This abundance of APIs gives us a problem because on our side we need to integrate them somehow with our Worker-based threading model. ### How we will do it #1 is trivial (if you call it from Worker you will get result in the same Worker thread but nonetheless you should never use it in production because it consumes the whole Worker). #4 and #5 trivial as well because caller has to provide execution context for getting result anyway and can simply use Worker. But #2 is problematic because by default it will deliver response in Thrift's own IO thread even if client method has been called from Worker (as well as #3 but this one should never be used in practice because Futures are deprecated). ### How to solve async methods? Async methods require callback to passed. We will provide option to set callback executor when creating a new Thrift client and when async_ method is called we will wrap client-provided callback into one which simply delegates execution to this callback executor. As a nice side benefit It will work with Future methods as well. Reviewed By: MohamedBassem Differential Revision: D22866545 fbshipit-source-id: 781b36ba38506cd9a1c78162cd1127afab7ed39e

Summary: == Removing extras == This stack of diff is about removing extras in the code base. === What is extras? === "extras" is a feature that consists in speculatively selecting more than R (replication factor) recipients in the copyset for an APPEND. This allows more predictable latencies of appends as Appender just needs to wait for the first R copies to acknowledge the write before retiring. As an optimization to save storage resources, Appender would send DELETE messages to unused recipients once it's successfully replicated the record, without waiting for these DELETEs to be acknowledged. This feature is configured by the "extraCopies" attribute in logs config. This is not to be confused with another piece of metadata called "extras" which allows passing arbitrary KV metadata to a log group (such as oncall for example). For example, if replication=3 and extras=2, Appender sends 5 stores and will acknowledge the APPEND to the client once any 3 recipient has acknowledged. If there is notification of a failure to store a copy on any recipient, Appender checks whether there are enough remaining outstanding stores in flight such that we can hope to complete the write. If that is not the case, Appender starts another wave. Without extras, Appender basically retries a wave as soon as any recipient fails the store. === Is this used in production? === No. It cannot be used in production for several reasons. 1/ Extras make rebuilding and Single Copy Delivery unsafe. Indeed, if replication=3 and extras=1, effectively the serialized copyset in local log store will have size 4. When a storage node reads this copyset, it's unclear which recipients in the copyset actually have a copy or not. Both Rebuilding and Single Copy Delivery rely on the copyset to make decisions about which recipient should do work. Extras make this logic more complex and error prone. One could modify Appender to have a final stage that sends a final AMEND message to all 3 copies to remove the "extras" from their copyset, but again this is additional complexity and would not be 100% safe if the Appender crashes before this stage. 2/ It's harder to implement copyset selection. Since we stopped using extras in production because of 1/, we also implemented new copyset selectors that have to navigate many constraints such as replicating over failure domains. These selectors do not work with extras and actually silently discard your intent to use them if you were to configure a log with that feature. === How is extras adding complexity in the code base? === This is a setting that adds a fair amount of boilerplate code in copyset selectors, logs configuration and tests. It is part of our public documentation although unusable. It is one less thing to wonder/worry about. By the way, I saw we also have a "syncedCopies" option that I'm pretty sure we also don't use in production. === Is there a future where we do have something similar? === In a world where we build a digest-based read path (Read Path 2.0) and centralized rebuilding also based on digest, copysets will not be useful anymore. This would remove problem 1/. Problem 2/ would still exist. Some systems speculatively attempt to select an additional node if the current time that has passed is greater than say the 99th percentile of observed latencies in past requests. In a world where we don't have copysets (and therefore don't need to AMEND and resend 3 STORE messages in each wave), this process would be there for free thanks to the "wave mechanism" that we already have in Appender. == Unit tests for Appender == More specifically, this diffs removes any logic in the unit tests for Appender related to verifying the behavior when extras are enabled. Reviewed By: MohamedBassem Differential Revision: D23867744 fbshipit-source-id: ce188095cf6ac30e454649455675344471c4db5c

Fixing the website title

35b7b19

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Sep 12, 2018

AhmedSoliman closed this Sep 12, 2018

march-github mentioned this pull request Oct 24, 2019

logdeviced 2.46.5 crashes if run with no parameters #133

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing the website title #1

Fixing the website title #1

AhmedSoliman commented Sep 12, 2018

Fixing the website title #1

Fixing the website title #1

Conversation

AhmedSoliman commented Sep 12, 2018