Periodic crash in free_stale_connections() on Linux

[oleevg]

Hi!

We have C++ REST client application that uses cpprestsdk. We are observing periodic crash in cpprestsdk code. The crash happens not very often - once in several days.

Environment:
Client side - cpprestsdk 2.10.6 Linux static build with gcc4.8, boost 1.68.0 Linux static build with gcc4.8.
Server side - .NET Core REST server.

I know we have quite old cpprestsdk version but in our case it is not very easy to move to the new one. Looking at the strange backtrace I can only assume that the crash might be due to some race conditions problem. I tried to examine the all commits since 2.10.6 version but I haven't found any commit that might handle this problem.

According to the backtrace at some point we have invalid 'this' address:
#2 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x25) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:141

If this is some known and already fixed problem it would be great to have reference to the related commit.

Thanks!

Below is the problem thread's backtrace:
#0 0x00000000004408de in __gnu_cxx::__exchange_and_add (__val=-1, __mem=0x2d) at /usr/include/c++/4.8.2/ext/atomicity.h:49
#1 __gnu_cxx::__exchange_and_add_dispatch (__val=-1, __mem=0x2d) at /usr/include/c++/4.8.2/ext/atomicity.h:82
#2 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x25) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:141
#3 0x0000000000684c94 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count (this=0x7fce9c001bc8, __in_chrg=) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:546
#4 std::__shared_ptr<web::http::client::details::asio_connection, (__gnu_cxx::_Lock_policy)2>::__shared_ptr (this=0x7fce9c001bc0, __in_chrg=) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:781
#5 std::shared_ptrweb::http::client::details::asio_connection::~shared_ptr (this=0x7fce9c001bc0, __in_chrg=) at /usr/include/c++/4.8.2/bits/shared_ptr.h:93
#6 std::_Destroy<std::shared_ptrweb::http::client::details::asio_connection > (__pointer=0x7fce9c001bc0) at /usr/include/c++/4.8.2/bits/stl_construct.h:93
#7 std::_Destroy_aux::__destroy<std::shared_ptrweb::http::client::details::asio_connection> (__last=, __first=0x7fce9c001bc0) at /usr/include/c++/4.8.2/bits/stl_construct.h:103
#8 std::_Destroy<std::shared_ptrweb::http::client::details::asio_connection> (__last=, __first=) at /usr/include/c++/4.8.2/bits/stl_construct.h:126
#9 std::_Destroy<std::shared_ptrweb::http::client::details::asio_connection, std::shared_ptrweb::http::client::details::asio_connection > (__last=0x7fce9c001bd0, __first=0x7fce9c001bc0)
at /usr/include/c++/4.8.2/bits/stl_construct.h:151
#10 std::vector<std::shared_ptrweb::http::client::details::asio_connection, std::allocator<std::shared_ptrweb::http::client::details::asio_connection > >::_M_erase_at_end (this=, __pos=0x7fce9c001bc0)
at /usr/include/c++/4.8.2/bits/stl_vector.h:1352
#11 std::vector<std::shared_ptrweb::http::client::details::asio_connection, std::allocator<std::shared_ptrweb::http::client::details::asio_connection > >::erase (__last=..., __first=..., this=)
at /usr/include/c++/4.8.2/bits/vector.tcc:153
#12 web::http::client::details::connection_pool_stack::free_stale_connections (this=) at /srv/build/agent/work/21711c7319e2844b/cpprestsdk/cpprest/src/http/client/http_client_asio.cpp:378
#13 web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptrweb::http::client::details::asio_connection_pool const&)::{lambda(boost::system::error_code const&)#1}::operator()(boost::system::error_code const&) const (ec=..., __closure=0x7fcee1345d40) at /srv/build/agent/work/21711c7319e2844b/cpprestsdk/cpprest/src/http/client/http_client_asio.cpp:485
#14 boost::asio::detail::binder1<web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptrweb::http::client::details::asio_connection_pool const&)::{lambda(boost::system::error_code const&)#1}, boost::system::error_code>::operator()() (this=0x7fcee1345d40) at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/detail/bind_handler.hpp:65
#15 boost::asio::asio_handler_invoke<boost::asio::detail::binder1<web::http::client::details::asio_connection_pool::start_epoch_interval(const std::shared_ptrweb::http::client::details::asio_connection_pool&)::__lambda134, boost::system::error_code> > (function=...) at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/handler_invoke_hook.hpp:69
#16 boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder1<web::http::client::details::asio_connection_pool::start_epoch_interval(const std::shared_ptrweb::http::client::details::asio_connection_pool&)::__lambda134, boost::system::error_code>, web::http::client::details::asio_connection_pool::start_epoch_interval(const std::shared_ptrweb::http::client::details::asio_connection_pool&)::__lambda134> (context=..., function=...)
at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#17 boost::asio::detail::handler_work<web::http::client::details::asio_connection_pool::start_epoch_interval(const std::shared_ptrweb::http::client::details::asio_connection_pool&)::__lambda134, boost::asio::system_executor>::complete<boost::asio::detail::binder1<web::http::client::details::asio_connection_pool::start_epoch_interval(const std::shared_ptrweb::http::client::details::asio_connection_pool&)::__lambda134, boost::system::error_code> > (
this=, handler=..., function=...) at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/detail/handler_work.hpp:82
#18 boost::asio::detail::wait_handler<web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptrweb::http::client::details::asio_connection_pool const&)::{lambda(boost::system::error_code const&)#1}>::do_complete(void, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (owner=, base=)
at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/detail/wait_handler.hpp:72
#19 0x000000000061282b in boost::asio::detail::scheduler_operation::complete (bytes_transferred=, ec=..., owner=0x2bea7e0, this=)
at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/detail/scheduler_operation.hpp:40
#20 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x2bea7e0) at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/detail/impl/scheduler.ipp:401
#21 boost::asio::detail::scheduler::run (this=0x2bea7e0, ec=...) at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/detail/impl/scheduler.ipp:154
#22 0x00000000006a1327 in boost::asio::io_context::run (this=0xce32c8 crossplat::threadpool::shared_instance()::s_shared+8)
at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/impl/io_context.ipp:62
#23 (anonymous namespace)::threadpool_impl::thread_start (arg=0xce32c0 crossplat::threadpool::shared_instance()::s_shared) at /srv/build/agent/work/21711c7319e2844b/cpprestsdk/cpprest/src/pplx/threadpool.cpp:63
#24 (anonymous namespace)::threadpool_impl::__lambda134::operator() (__closure=) at /srv/build/agent/work/21711c7319e2844b/cpprestsdk/cpprest/src/pplx/threadpool.cpp:45
#25 boost::asio::detail::posix_thread::func<(anonymous namespace)::threadpool_impl::add_thread()::__lambda134>::run(void) (this=)
at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/detail/posix_thread.hpp:86
#26 0x00000000006060cf in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x2bec280)
at /srv/build/agent/work/21711c7319e2844b/boost.lin.gcc48.rt-static.x86_64.1.68.0.3/include/boost/asio/detail/impl/posix_thread.ipp:74
#27 0x00007fceeb49c6db in start_thread (arg=0x7fcee1346700) at pthread_create.c:463
#28 0x00007fceea47a88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95