Thread::try_create(): pthread_create failed with error 11

[JoeKCWong]

I am following the wiki to setup ceph-plugin, at last dovecot (2.3.1) lmtp get assertion error as follow:

May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: Thread::try_create(): pthread_create failed with error 11/home/jenkins-build/build/wo
rkspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/cep
h-12.2.5/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f1ab9f59740 time 2018-05-08 22:39:52.236250
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAI
LABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/common/Thread.cc: 152: FAILED assert(ret ==
0)
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7
f1aacf32d50]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 2: (Thread::create(char const*, unsigned long)+0xba) [0x7f1aad17702a]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 3: (CephContext::CephContext(unsigned int, code_environment_t, int)+0x27b) [0x7f1aad
16f19b]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 4: (common_preinit(CephInitParameters const&, code_environment_t, int)+0xa7) [0x7f1a
ad168ae7]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 5: (()+0x81e60) [0x7f1ab61e9e60]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 6: (rados_create2()+0x92) [0x7f1ab61ea442]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 7: (librmb::RadosClusterImpl::init(std::string const&, std::string const&)+0x82) [0x
7f1ab670823c]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 8: (librmb::RadosStorageImpl::open_connection(std::string const&, std::string const&
, std::string const&)+0x85) [0x7f1ab670974d]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 9: (rbox_open_rados_connection()+0x138) [0x7f1ab695291b]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 10: (rbox_mail_storage_copy(mail_save_context*, mail*)+0x9f) [0x7f1ab694650a]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 11: (rbox_mail_copy(mail_save_context*, mail*)+0xfa) [0x7f1ab69455fd]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 12: (()+0x373a) [0x7f1ab9b4273a]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 13: (()+0x4cf96) [0x7f1ab9838f96]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 14: (mail_deliver_save()+0x1a9) [0x7f1ab9b43a49]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 15: (mail_deliver()+0x216) [0x7f1ab9b440b6]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 16: (lmtp_local_data()+0x6eb) [0x56191027f7ab]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 17: (cmd_data_continue()+0x233) [0x56191027e293]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 18: (()+0x4f7e0) [0x7f1ab94c17e0]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 19: (io_loop_call_io()+0x65) [0x7f1ab9556015]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 20: (io_loop_handler_run_internal()+0x10f) [0x7f1ab955790f]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 21: (io_loop_handler_run()+0x52) [0x7f1ab9556112]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 22: (io_loop_run()+0x38) [0x7f1ab9556338]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 23: (master_service_run()+0x13) [0x7f1ab94d26c3]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 24: (main()+0x229) [0x56191027d319]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 25: (__libc_start_main()+0xf5) [0x7f1ab90d0c05]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: 26: (()+0x5445) [0x56191027d445]
May 8 22:39:52 ceph-all-in-one dovecot: lmtp: Error: NOTE: a copy of the executable, or objdump -rdS <executable> is needed to interpret this.

I then rerun with strace, found it reaches ENOMEM after mmap then pthread_create failed:

[pid 2935] 16:48:33.051508 mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0 <unfinished ...>
[pid 2935] 16:48:33.051549 <... mmap resumed> ) = -1 ENOMEM (Cannot allocate memory) <0.000036>
[pid 2935] 16:48:33.051615 rt_sigprocmask(SIG_SETMASK, [], <unfinished ...>
[pid 2935] 16:48:33.051787 <... rt_sigprocmask resumed> NULL, 8) = 0 <0.000164>
[pid 2935] 16:48:33.051841 write(2, "Thread::try_create(): pthread_create failed with error 11", 57 <unfinished ...>
[pid 2935] 16:48:33.051896 <... write resumed> ) = 57 <0.000049>

I am running ceph 12.2.5 on Centos 7 VM with 8G RAM.

Any idea how to solve this?

[jrse]

looks like lmtp process is running out of memory:
you can increase the default_vsz_limit in 10-master.conf:
e.g.; default_vsz_limit = 768M

Can you check, that you increased the system limits (stack_size and open files) for the dovecot process with:
ulimit -s unlimited
ulimit -n 3072

[JoeKCWong]

Thanks, the tuning solved that error but it is still failing afterward.

May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3417>: Debug: value is: mail_storage
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3417>: Debug: [END] rbox-storage.cpp: read_plugin_configuration at line 344
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Debug: Error = -2
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Debug: [END] rbox-copy.cpp: rbox_mail_storage_copy at line 255, ret == -1, connection to rados failed
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Debug: [END] rbox-copy.cpp: rbox_mail_copy at line 55
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Debug: [START] rbox-save.cpp: rbox_transaction_save_rollback at line 567
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Warning: clean_up_failed, index entry for seq 0, not removed r_ctx->seq <= 0
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Debug: [END] rbox-save.cpp: rbox_transaction_save_rollback at line 598
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Debug: Mailbox : Opened mail UID=1 because: header Message-ID (Cache file is unusable)
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Debug: Mailbox : Opened mail UID=1 because: virtual size (Cache file is unusable)
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: msgid=[email protected]: save failed to INBOX: BUG: Unknown internal error
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Debug: [START] rbox-storage.cpp: rbox_mailbox_close at line 583
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(joewong)<3415>: Debug: [END] rbox-storage.cpp: rbox_mailbox_close at line 604
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(3415): Debug: [START] rbox-storage.cpp: rbox_storage_destroy at line 164
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(3415): Debug: [END] rbox-storage.cpp: rbox_storage_destroy at line 192
May 9 18:00:34 ceph-all-in-one dovecot: lmtp(3415): Disconnect from local: Client has quit the connection (state = READY)

[jrse]

It looks like the plugin cannot create the connection to rados storage. This may be due to insufficient user rights. Can you check that your dovecot user can read the ceph.conf and the client keyring. e.g. if you are using the defaults: ceph.client.admin.keyring. Can you connect with the ceph admin client via rados or ceph command?

[JoeKCWong]

Thanks for the hints, after adjusting the setup, Dovecot can talk through the plugin with CEPH. There are missing configuration or typo in WIKI that I have to read Dovecot / Plugin Source code in order to put the correct settings in place:

20-imap.conf: mail_plugins = $mail_plugins quota storage_rbox
20-lmtp.conf: mail_plugins = $mail_plugins quota storage_rbox

90-plugin.conf: rbox_cluster_name = ceph
90-plugin.conf: rbox_user_name = client.admin
90-plugin.conf: rbox_pool_name = mail_storage
90-plugin.conf: rbox_cfg_object_name = rbox_cfg
90-plugin.conf: rbox_bugfix_cephfs_21652 = false

The bold values above is the adjustment I made which is differ from WIKI

[jrse]

thank you for your feedback. i will have a look at it.

[jrse]

I updated the WIKI, so that the documentation should now be up to date.