Thread::try_create(): pthread_create failed with error 11

[jrse]

Describe the bug

The bug will occur, when we use imaptest to simulate user behavior, with the attached imaptest-profile and 2000 parallel clients we run into resources issues. The result is that pthreads could not be created temporarily. Pthreads are used by the librados library to connect / write/ disconnect to the rados cluster. Currently the plugin will not check for the specific error code -11 (EAGAIN) and has no strategy to recreate the pthread implemented.

To Reproduce
Steps to reproduce the behavior:

1. Use attached imaptest-profile with the following cmd line:
   imaptest user=t%d pass=t port=10143 clients=2000 error_quit profile=imap-profile.conf
2. check dovecot.log for errors.

Expected behavior
The expected behavior is, because it is only a temporary error, that the plugin retries to connect to the cluster. e.g. retries write. It can detect this specific error by checking the error code -11 (EAGAIN).

Logfiles, screenshots
Error message:

Aug 20 17:25:51 lmtp: Error: Thread::try_create(): pthread_create failed with error 11/build/ceph-roHgVA/ceph-12.2.4/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 150ce7800000 time 2018-08-20 17:25:50.971919
Aug 20 17:25:51 lmtp: Error: /build/ceph-roHgVA/ceph-12.2.4/src/common/Thread.cc: 152: FAILED assert(ret == 0)
Aug 20 17:25:51 lmtp: Error: ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
Aug 20 17:25:51 lmtp: Error: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xf5) [0x150cdcb92e95]
Aug 20 17:25:51 lmtp: Error: 2: (()+0x4c8288) [0x150cdcdf2288]
Aug 20 17:25:51 lmtp: Error: 3: (CephContext::start_service_thread()+0x12f) [0x150cdcde867f]
Aug 20 17:25:51 lmtp: Error: 4: (common_init_finish(CephContext*)+0xbc) [0x150cdcde4dec]
Aug 20 17:25:51 lmtp: Error: 5: (librados::RadosClient::connect()+0x30) [0x150ce5e2c410]
Aug 20 17:25:51 lmtp: Error: 6: (librmb::RadosClusterImpl::connect()+0x43) [0x150ce6101e1f]
Aug 20 17:25:51 lmtp: Error: 7: (librmb::RadosClusterImpl::io_ctx_create(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, librados::IoCtx*)+0x67) [0x150ce610209d]
Aug 20 17:25:51 lmtp: Error: 8: (librmb::RadosStorageImpl::create_connection(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)+0x4b) [0x150ce61033c5]
Aug 20 17:25:51 lmtp: Error: 9: (librmb::RadosStorageImpl::open_connection(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)+0xa6) [0x150ce6103326]
Aug 20 17:25:51 lmtp: Error: 10: (rbox_open_rados_connection()+0x11e) [0x150ce635fae2]
Aug 20 17:25:51 lmtp: Error: 11: (rbox_mail_storage_copy(mail_save_context*, mail*)+0xab) [0x150ce634fb06]
Aug 20 17:25:51 lmtp: Error: 12: (rbox_mail_copy(mail_save_context*, mail*)+0xcb) [0x150ce634e235]
Aug 20 17:25:51 lmtp: Error: 13: (()+0x5d49) [0x150ce7401d49]
Aug 20 17:25:51 lmtp: Error: 14: (()+0x4b4f1) [0x150ce70fc4f1]
Aug 20 17:25:51 lmtp: Error: 15: (mail_deliver_save()+0x190) [0x150ce7402710]
Aug 20 17:25:51 lmtp: Error: 16: (mail_deliver()+0x18b) [0x150ce7402bfb]
Aug 20 17:25:51 lmtp: Error: 17: (()+0x73ac) [0x55eb5a0233ac]
Aug 20 17:25:51 lmtp: Error: 18: (io_loop_call_io()+0x52) [0x150ce6e33d02]
Aug 20 17:25:51 lmtp: Error: 19: (io_loop_handler_run_internal()+0x12e) [0x150ce6e3541e]
Aug 20 17:25:51 lmtp: Error: 20: (io_loop_handler_run()+0x36) [0x150ce6e33d96]
Aug 20 17:25:51 lmtp: Error: 21: (io_loop_run()+0x38) [0x150ce6e33f48]
Aug 20 17:25:51 lmtp: Error: 22: (master_service_run()+0x13) [0x150ce6db77c3]
Aug 20 17:25:51 lmtp: Error: 23: (main()+0x1cc) [0x55eb5a02148c]
Aug 20 17:25:51 lmtp: Error: 24: (__libc_start_main()+0xe7) [0x150ce69acb97]
Aug 20 17:25:51 lmtp: Error: 25: (_start()+0x2a) [0x55eb5a02155a]
Aug 20 17:25:51 lmtp: Error: NOTE: a copy of the executable, or objdump -rdS <executable> is needed to interpret this.
Aug 20 17:25:51 lmtp: Error: terminate called after throwing an instance of 'std::system_error'
Aug 20 17:25:51 lmtp: Error: what(): Resource temporarily unavailable
Aug 20 17:25:51 lmtp: Error: terminate called after throwing an instance of 'std::system_error'
Aug 20 17:25:51 lmtp: Error: what(): Resource temporarily unavailable
Aug 20 17:25:51 lmtp: Error: terminate called after throwing an instance of 'std::system_error'
Aug 20 17:25:51 lmtp: Error: what(): Resource temporarily unavailable

Server (please complete the following information):

• OS: ubuntu 18.04
• Ceph Version 12.2.4
• Dovecot Version [2.2.21]
• plugin Version: > 0.0.12
• docker-container (ceph demo, influxdb and grafana)

Client (please complete the following information):

• used client imaptest
• Version https://github.com/ceph-dovecot/imaptest

Additional context
Imap Profile
imap-profile.conf.txt

[jrse]

-> the pthread_create failed problem is due to a misconfiguration of the test system.