Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not close all file descriptors on parent process #2628

Closed
renecannao opened this issue Mar 31, 2020 · 1 comment
Closed

Do not close all file descriptors on parent process #2628

renecannao opened this issue Mar 31, 2020 · 1 comment

Comments

@renecannao
Copy link
Contributor

We got a report about proxysql crashing with the following backtrace:

Error: signal 6: 
proxysql(_Z13crash_handleri+0x2d)[0x4ae7fd] 
/lib/x86_64-linux-gnu/libc.so.6(+0x354b0)[0x7f17816804b0] 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f1781680428] 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f178168202a] 
/usr/lib/x86_64-linux-gnu/libgnutls.so.30(+0xe9f2e)[0x7f17829c8f2e] 
/usr/lib/x86_64-linux-gnu/libhogweed.so.4(nettle_mpz_random_size+0x36)[0x7f1780926156] 
/usr/lib/x86_64-linux-gnu/libhogweed.so.4(nettle_mpz_random+0x34)[0x7f17809261c4] 
/usr/lib/x86_64-linux-gnu/libhogweed.so.4(nettle_rsa_compute_root_tr+0x121)[0x7f17809277b1] 
/usr/lib/x86_64-linux-gnu/libhogweed.so.4(nettle_rsa_pkcs1_sign_tr+0x7e)[0x7f1780927a3e] 
/usr/lib/x86_64-linux-gnu/libgnutls.so.30(+0xec3b3)[0x7f17829cb3b3] 
/usr/lib/x86_64-linux-gnu/libgnutls.so.30(gnutls_privkey_sign_data+0xbc)[0x7f178293193c] 
/usr/lib/x86_64-linux-gnu/libgnutls.so.30(+0x4422d)[0x7f178292322d] 
/usr/lib/x86_64-linux-gnu/libgnutls.so.30(gnutls_certificate_set_x509_key_mem2+0x56)[0x7f178292dd56] 
proxysql(MHD_start_daemon_va+0x1371)[0x92d511] 
proxysql(MHD_start_daemon+0x8a)[0x92d75a] 
proxysql(_ZN14ProxySQL_Admin43flush_admin_variables___database_to_runtimeEP9SQLite3DBb+0x5d3)[0x536cb3] 
proxysql(_ZN14ProxySQL_Admin4initEv+0xb53)[0x55d433] 
proxysql(_Z31ProxySQL_Main_init_Admin_modulev+0x82)[0x4a9a52] 
proxysql(_Z39ProxySQL_Main_init_phase2___not_startedv+0x148)[0x4aa428] 
proxysql(main+0x5a3)[0x4a5713] 
2020-03-19 20:55:26 main.cpp:1538:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!

After a long investigation, we found some clue in these two issues:

Those gave us some hint to finally be able to reproduce it.
Indeed, the crash happens only when proxysql is executed in background, as a daemon.
If proxysql is running in foreground, the crash doesn't happen.

If we do not call daemon_close_all() , proxysql doesn't crash also when running in background.

These are the file-descriptors that the parent process has after the fork():

proxysql 30059 rcannao    0r   CHR    1,3      0t0       6 /dev/null
proxysql 30059 rcannao    3r   CHR    1,9      0t0      11 /dev/urandom
proxysql 30059 rcannao    4r   REG   0,51 39530704  867685 /home/rcannao/proxysql/src/proxysql

Therefore, the same FDs are present on the child process.
if daemon_close_all() is called, the child process will lose a file-descriptor to /dev/urandom , and apparently never opens it again.
That brings me back to this comment: charybdis-ircd/charybdis#71 (comment)
Probably gnutls isn't able to pass a good random function because /dev/urandom is closed!

Although the backtrace in #2607 is different, and at today I am still unable to reproduce it, I won't be surprised if the root cause is the same: file descriptor to /dev/urandom being closed.

@renecannao renecannao changed the title Do not close all file descriptors on parent processes Do not close all file descriptors on parent process Mar 31, 2020
renecannao added a commit that referenced this issue Apr 1, 2020
Do not close file descriptors on parent #2628
@renecannao
Copy link
Contributor Author

PR merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant