Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash in libssl when opening connections in parallel on Windows #595

Closed
larskanis opened this issue Oct 16, 2024 · 0 comments · Fixed by #598
Closed

Crash in libssl when opening connections in parallel on Windows #595

larskanis opened this issue Oct 16, 2024 · 0 comments · Fixed by #598

Comments

@larskanis
Copy link
Collaborator

The fat binary gem for Windows pg-1.5.8-x64-mingw-ucrt.gem crashes when PG.connect is simultaneously called from multiple threads.

require "pg"

CONNECT_DB_MUTEX = Mutex.new

4.times.map do |idx|
  Thread.new do
    # CONNECT_DB_MUTEX.synchronize do
      p PG.connect dbname: 'postgres' #, sslmode: 'disable'
    # end
  end
end.each(&:join)
GDB output looks like so: 
warning: HEAP[ruby.exe]:
warning: Invalid address specified to RtlFreeHeap( 00000000006A0000, 0000000006128670 )

Thread 8 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 26424.0x4c10]
0x00007fffebe5705f in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
(gdb) bt
#0  0x00007fffebe5705f in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007fffebe21eef in ntdll!EtwLogTraceEvent () from C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007fffebe553c8 in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
#3  0x00007fffebe0cd18 in ntdll!EtwLogTraceEvent () from C:\Windows\SYSTEM32\ntdll.dll
#4  0x00007fffebd8c324 in ntdll!RtlGetCurrentServiceSessionId () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x00007fffebd8aff1 in ntdll!RtlFreeHeap () from C:\Windows\SYSTEM32\ntdll.dll
#6  0x00007fffe994364b in ucrtbase!_free_base () from C:\Windows\System32\ucrtbase.dll
#7  0x000000006457ef13 in CRYPTO_free ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#8  0x000000006454c18f in err_clear ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#9  0x000000006454c322 in ERR_pop_to_mark ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#10 0x000000006444d461 in ssl_evp_cipher_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#11 0x00000000644374f9 in ssl_load_ciphers ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#12 0x000000006444593d in SSL_CTX_new_ex ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#13 0x0000000064445f93 in SSL_CTX_new ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#14 0x0000000064420a13 in initialize_SSL ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#15 0x000000006441fcd3 in pgtls_open_client ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#16 0x000000006441b28c in pqsecure_open_client ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#17 0x00000000644070df in PQconnectPoll ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#18 0x000000007038143b in ?? ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#19 0x00007fff97bebfb5 in x64-ucrt-ruby330!rb_nogvl () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#20 0x00007fff97bec241 in x64-ucrt-ruby330!rb_thread_call_without_gvl ()
   from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#21 0x0000000070381a88 in ?? ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#22 0x000000007038ab3a in pg_ext!Init_pg_ext ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#23 0x00007fff97c2b410 in x64-ucrt-ruby330!rb_error_arity () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#24 0x00007fff97c3d950 in x64-ucrt-ruby330!rb_vm_invoke_bmethod () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#25 0x00007fff97c3de6b in x64-ucrt-ruby330!rb_vm_invoke_bmethod () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#26 0x00007fff97c49742 in x64-ucrt-ruby330!rb_ec_obj_respond_to () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#27 0x00007fff97c3ae45 in x64-ucrt-ruby330!rb_vm_exec () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#28 0x00007fff97c3f9c5 in x64-ucrt-ruby330!rb_vm_invoke_proc () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#29 0x00007fff97be8deb in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#30 0x00007fff97be9472 in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#31 0x00007fff97be9bcf in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#32 0x00007fffea79257d in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#33 0x00007fffebdaaf28 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#34 0x0000000000000000 in ?? ()
(gdb)
or so: 
warning: HEAP[ruby.exe]:
warning: Invalid address specified to RtlReAllocateHeap( 0000000000720000, 00000000061648F0 )

Thread 11 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 27176.0x6790]
0x00007fffebe5705f in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
(gdb) bt
#0  0x00007fffebe5705f in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007fffebe21eef in ntdll!EtwLogTraceEvent () from C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007fffebe55e99 in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
#3  0x00007fffebe0eaa1 in ntdll!EtwLogTraceEvent () from C:\Windows\SYSTEM32\ntdll.dll
#4  0x00007fffebd922c7 in ntdll!RtlReAllocateHeap () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x00007fffebd9218a in ntdll!RtlReAllocateHeap () from C:\Windows\SYSTEM32\ntdll.dll
#6  0x00007fffe99418e9 in ucrtbase!_realloc_base () from C:\Windows\System32\ucrtbase.dll
#7  0x000000006457edc6 in CRYPTO_realloc ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#8  0x0000000064576eb9 in expand ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#9  0x0000000064576a67 in OPENSSL_LH_insert ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#10 0x00000000645798c2 in lh_NAMENUM_ENTRY_insert ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#11 0x000000006457a01a in namemap_add_name ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#12 0x000000006457a0ca in ossl_namemap_add_name ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#13 0x000000006457a4ea in get_legacy_evp_names ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#14 0x000000006457a56a in get_legacy_cipher_names ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#15 0x0000000064592a28 in do_all_fn ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#16 0x0000000064592a5d in lh_OBJ_NAME_doall_OBJ_DOALL_thunk ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#17 0x0000000064576d0c in doall_util_fn ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#18 0x0000000064576e4a in OPENSSL_LH_doall_arg_thunk ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#19 0x0000000064592a96 in lh_OBJ_NAME_doall_OBJ_DOALL ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#20 0x0000000064592ae3 in OBJ_NAME_do_all ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#21 0x000000006457a748 in ossl_namemap_stored ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#22 0x0000000064557e3d in inner_evp_generic_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#23 0x00000000645582bc in evp_generic_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#24 0x0000000064557705 in EVP_CIPHER_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#25 0x000000006444d458 in ssl_evp_cipher_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#26 0x00000000644374f9 in ssl_load_ciphers ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#27 0x000000006444593d in SSL_CTX_new_ex ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#28 0x0000000064445f93 in SSL_CTX_new ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#29 0x0000000064420a13 in initialize_SSL ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#30 0x000000006441fcd3 in pgtls_open_client ()
--Type <RET> for more, q to quit, c to continue without paging--
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#31 0x000000006441b28c in pqsecure_open_client ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#32 0x00000000644070df in PQconnectPoll ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#33 0x000000007038143b in ?? ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#34 0x00007fff97bebfb5 in x64-ucrt-ruby330!rb_nogvl () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#35 0x00007fff97bec241 in x64-ucrt-ruby330!rb_thread_call_without_gvl ()
   from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#36 0x0000000070381a88 in ?? ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#37 0x000000007038ab3a in pg_ext!Init_pg_ext ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#38 0x00007fff97c2b410 in x64-ucrt-ruby330!rb_error_arity () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#39 0x00007fff97c3d950 in x64-ucrt-ruby330!rb_vm_invoke_bmethod () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#40 0x00007fff97c3de6b in x64-ucrt-ruby330!rb_vm_invoke_bmethod () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#41 0x00007fff97c49742 in x64-ucrt-ruby330!rb_ec_obj_respond_to () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#42 0x00007fff97c3ae45 in x64-ucrt-ruby330!rb_vm_exec () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#43 0x00007fff97c3f9c5 in x64-ucrt-ruby330!rb_vm_invoke_proc () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#44 0x00007fff97be8deb in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#45 0x00007fff97be9472 in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#46 0x00007fff97be9bcf in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#47 0x00007fffea79257d in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#48 0x00007fffebdaaf28 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#49 0x0000000000000000 in ?? ()
(gdb)

It doesn't crash when sslmode: 'disable' is set, or when a Mutex avoids concurrent connections.

It also doesn't happen, when libpq and libssl is used from the MSYS2-MINGW packages like so:

$ pacman -S %MINGW_PACKAGE_PREFIX%-postgresql
$ gem uninst pg
$ gem inst pg --platform ruby
$ ruby test-crash-concurrent-ar.rb
#<PG::Connection:0x000001f05a279ab0 host=localhost port=5432 user=kanis>
#<PG::Connection:0x000001f05a27c710 host=localhost port=5432 user=kanis>
#<PG::Connection:0x000001f05a27b8b0 host=localhost port=5432 user=kanis>
#<PG::Connection:0x000001f05a27a9b0 host=localhost port=5432 user=kanis>

Possibly there's some relationship with the crash observed on Linux with binary gems in #551 .
larskanis added a commit to larskanis/ruby-pg that referenced this issue Oct 24, 2024
@larskanis
Enable thread safety in static OpenSSL build
4126827 
This is disabled by default due to the `-static` option.
The `thread` doesn't work with `-static`, that's why it is enabled by `-DOPENSSL_THREADS`.
The thread functions are used in libcrypto.a and libssl.a but only defined in libssl.a.

Fixes ged#595
@larskanis larskanis mentioned this issue Oct 24, 2024
Merged
larskanis added a commit to larskanis/ruby-pg that referenced this issue Oct 24, 2024
@larskanis
Enable thread safety in static OpenSSL build
abc13f4 
This is disabled by default due to the `-static` option.
The `thread` doesn't work with `-static`, that's why it is enabled by `-DOPENSSL_THREADS`.
The thread functions are used in libcrypto.a and libssl.a but only defined in libssl.a.

Fixes ged#595
larskanis added a commit to larskanis/ruby-pg that referenced this issue Oct 24, 2024
@larskanis
Enable thread safety in static OpenSSL build
716221a 
This is disabled by default due to the `-static` option.
The `thread` doesn't work with `-static`, that's why it is enabled by `-DOPENSSL_THREADS`.
The thread functions are used in libcrypto.a and libssl.a but only defined in libssl.a.

Fixes ged#595
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable thread safety in static OpenSSL build larskanis/ruby-pg
1 participant
@larskanis