Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Immediate segfault on Linux (Manjaro) #337

Closed
1 of 3 tasks
Shamazo opened this issue Jul 14, 2020 · 18 comments
Closed
1 of 3 tasks

Immediate segfault on Linux (Manjaro) #337

Shamazo opened this issue Jul 14, 2020 · 18 comments

Comments

@Shamazo
Copy link

Shamazo commented Jul 14, 2020 •
edited
Loading

Description

The sentry SDK segfaults immediately when using the getting started code. I have a screenshot of the backtrace:
image
It appears to segfault somewhere in curl, which is being called from curl_easy_perform in sentry_transport_curl.c

  • During build
  • During run-time
  • When capturing a hard crash

Environment

  • OS: Manjaro (arch derivative)
  • Compiler: GNU 10.1.0
  • CMake config: -DCMAKE_BUILD_TYPE=Debug
  • CURL: 7.70.0

Steps To Reproduce

Place this snippet in main.

sentry_options_t *options = sentry_options_new();
    sentry_options_set_debug(options, true);
    sentry_options_set_dsn(options, "<dsn>");
    sentry_init(options);
    sentry_capture_event(sentry_value_new_message_event(
            /*   level */ SENTRY_LEVEL_INFO,
            /*  logger */ "custom",
            /* message */ "It works!"
    ));

Log output

[sentry] INFO using database path "<my build dir>/Linux/.sentry-native"
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG background worker thread started
[sentry] DEBUG starting backend
[sentry] INFO capturing event
[sentry] DEBUG merging scope into event
[sentry] DEBUG trying to read modules from /proc/self/maps
[sentry] DEBUG inspecting module "<my main function>"
[sentry] DEBUG inspecting module "/usr/lib/libgpg-error.so.0.29.0"
[sentry] DEBUG inspecting module "/usr/lib/libpcre.so.1.2.12"
[sentry] DEBUG inspecting module "/usr/lib/libgcrypt.so.20.2.5"
[sentry] DEBUG inspecting module "/usr/lib/liblz4.so.1.9.2"
[sentry] DEBUG inspecting module "/usr/lib/libFLAC.so.8.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libresolv-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libkeyutils.so.1.9"
[sentry] DEBUG inspecting module "/usr/lib/libkrb5support.so.0.1"
[sentry] DEBUG inspecting module "/usr/lib/libffi.so.7.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libicudata.so.67.1"
[sentry] DEBUG inspecting module "/usr/lib/libgraphite2.so.3.2.1"
[sentry] DEBUG inspecting module "/usr/lib/libglib-2.0.so.0.6400.3"
[sentry] DEBUG inspecting module "/usr/lib/libjpeg.so.8.2.2"
[sentry] DEBUG inspecting module "/usr/lib/libasyncns.so.0.3.1"
[sentry] DEBUG inspecting module "/usr/lib/libsystemd.so.0.28.0"
[sentry] DEBUG inspecting module "/usr/lib/libsndfile.so.1.0.28"
[sentry] DEBUG inspecting module "/usr/lib/libcom_err.so.2.1"
[sentry] DEBUG inspecting module "/usr/lib/libk5crypto.so.3.1"
[sentry] DEBUG inspecting module "/usr/lib/libkrb5.so.3.3"
[sentry] DEBUG inspecting module "/usr/lib/libgssapi_krb5.so.2.2"
[sentry] DEBUG inspecting module "/usr/lib/libssl.so.1.1"
[sentry] DEBUG inspecting module "/usr/lib/libpsl.so.5.3.2"
[sentry] DEBUG inspecting module "/usr/lib/libssh2.so.1.0.1"
[sentry] DEBUG inspecting module "/usr/lib/libnghttp2.so.14.19.0"
[sentry] DEBUG inspecting module "/usr/lib/libcrypto.so.1.1"
[sentry] DEBUG inspecting module "/usr/lib/libhogweed.so.6.0"
[sentry] DEBUG inspecting module "/usr/lib/libnettle.so.8.0"
[sentry] DEBUG inspecting module "/usr/lib/libtasn1.so.6.6.0"
[sentry] DEBUG inspecting module "/usr/lib/libunistring.so.2.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libidn2.so.0.3.7"
[sentry] DEBUG inspecting module "/usr/lib/libp11-kit.so.0.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libicuuc.so.67.1"
[sentry] DEBUG inspecting module "/usr/lib/libpng16.so.16.37.0"
[sentry] DEBUG inspecting module "/usr/lib/libexpat.so.1.6.11"
[sentry] DEBUG inspecting module "/usr/lib/libharfbuzz.so.0.20600.7"
[sentry] DEBUG inspecting module "/usr/lib/libv4lconvert.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libdbus-1.so.3.19.11"
[sentry] DEBUG inspecting module "/usr/lib/pulseaudio/libpulsecommon-13.0.so"
[sentry] DEBUG inspecting module "/usr/lib/libXdmcp.so.6.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libXau.so.6.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libdb-5.3.so"
[sentry] DEBUG inspecting module "/usr/lib/libmvec-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libogg.so.0.8.4"
[sentry] DEBUG inspecting module "/usr/lib/libgomp.so.1.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libgcc_s.so.1"
[sentry] DEBUG inspecting module "/usr/lib/libstdc++.so.6.0.28"
[sentry] DEBUG inspecting module "/usr/lib/libcurl.so.4.6.0"
[sentry] DEBUG inspecting module "/usr/lib/librt-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libuuid.so.1.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libvdpau.so.1.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libva-x11.so.2.700.0"
[sentry] DEBUG inspecting module "/usr/lib/libva-drm.so.2.700.0"
[sentry] DEBUG inspecting module "/usr/lib/libssh.so.4.8.5"
[sentry] DEBUG inspecting module "/usr/lib/libsrt.so.1.4.1"
[sentry] DEBUG inspecting module "/usr/lib/libgnutls.so.30.28.0"
[sentry] DEBUG inspecting module "/usr/lib/libgmp.so.10.4.0"
[sentry] DEBUG inspecting module "/usr/lib/libbluray.so.2.2.0"
[sentry] DEBUG inspecting module "/usr/lib/libmodplug.so.1.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libbz2.so.1.0.8"
[sentry] DEBUG inspecting module "/usr/lib/libxml2.so.2.9.10"
[sentry] DEBUG inspecting module "/usr/lib/libfreetype.so.6.17.2"
[sentry] DEBUG inspecting module "/usr/lib/libfontconfig.so.1.12.0"
[sentry] DEBUG inspecting module "/usr/lib/libvidstab.so.1.1"
[sentry] DEBUG inspecting module "/usr/lib/libass.so.9.0.2"
[sentry] DEBUG inspecting module "/usr/lib/libvmaf.so"
[sentry] DEBUG inspecting module "/usr/lib/libfribidi.so.0.4.0"
[sentry] DEBUG inspecting module "/usr/lib/libXv.so.1.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libv4l2.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libSDL2-2.0.so.0.12.0"
[sentry] DEBUG inspecting module "/usr/lib/libpulse.so.0.21.1"
[sentry] DEBUG inspecting module "/usr/lib/libxcb-xfixes.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libxcb-shape.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libxcb-shm.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libxcb.so.1.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libdrm.so.2.4.0"
[sentry] DEBUG inspecting module "/usr/lib/libjack.so.0.0.28"
[sentry] DEBUG inspecting module "/usr/lib/libiec61883.so.0.1.1"
[sentry] DEBUG inspecting module "/usr/lib/librom1394.so.0.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libavc1394.so.0.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libraw1394.so.11.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libmfx.so.1.32"
[sentry] DEBUG inspecting module "/usr/lib/libva.so.2.700.0"
[sentry] DEBUG inspecting module "/usr/lib/libxvidcore.so.4.3"
[sentry] DEBUG inspecting module "/usr/lib/libx265.so.188"
[sentry] DEBUG inspecting module "/usr/lib/libx264.so.159"
[sentry] DEBUG inspecting module "/usr/lib/libvorbisenc.so.2.0.11"
[sentry] DEBUG inspecting module "/usr/lib/libvorbis.so.0.4.8"
[sentry] DEBUG inspecting module "/usr/lib/libtheoradec.so.1.1.4"
[sentry] DEBUG inspecting module "/usr/lib/libtheoraenc.so.1.1.2"
[sentry] DEBUG inspecting module "/usr/lib/libspeex.so.1.5.1"
[sentry] DEBUG inspecting module "/usr/lib/libopus.so.0.8.0"
[sentry] DEBUG inspecting module "/usr/lib/libopenjp2.so.2.3.1"
[sentry] DEBUG inspecting module "/usr/lib/libopencore-amrnb.so.0.0.3"
[sentry] DEBUG inspecting module "/usr/lib/libmp3lame.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libgsm.so.1.0.13"
[sentry] DEBUG inspecting module "/usr/lib/libaom.so.0"
[sentry] DEBUG inspecting module "/usr/lib/libz.so.1.2.11"
[sentry] DEBUG inspecting module "/usr/lib/libopencore-amrwb.so.0.0.3"
[sentry] DEBUG inspecting module "/usr/lib/libdav1d.so.4.0.1"
[sentry] DEBUG inspecting module "/usr/lib/liblzma.so.5.2.5"
[sentry] DEBUG inspecting module "/usr/lib/libwebp.so.7.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libwebpmux.so.3.0.5"
[sentry] DEBUG inspecting module "/usr/lib/libvpx.so.6.2.0"
[sentry] DEBUG inspecting module "/usr/lib/libsoxr.so.0.1.2"
[sentry] DEBUG inspecting module "/usr/lib/libc-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libpthread-2.31.so"
[sentry] DEBUG inspecting module ".../cmake-build-debug/sentry-native/libsentry.so"
[sentry] DEBUG inspecting module "/usr/lib/libm-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libasound.so.2.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libXfixes.so.3.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libXext.so.6.4.0"
[sentry] DEBUG inspecting module "/usr/lib/libX11.so.6.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libICE.so.6.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libSM.so.6.0.1"
[sentry] DEBUG inspecting module ".../cmake-build-debug/fractal/input/libfractalInput.so"
[sentry] DEBUG inspecting module "/usr/lib/libdl-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libpostproc.so.55.5.100"
[sentry] DEBUG inspecting module "/usr/lib/libavutil.so.56.31.100"
[sentry] DEBUG inspecting module "/usr/lib/libswscale.so.5.5.100"
[sentry] DEBUG inspecting module "/usr/lib/libavformat.so.58.29.100"
[sentry] DEBUG inspecting module "/usr/lib/libavfilter.so.7.57.100"
[sentry] DEBUG inspecting module "/usr/lib/libavdevice.so.58.8.100"
[sentry] DEBUG inspecting module "/usr/lib/libavcodec.so.58.54.100"
[sentry] DEBUG inspecting module "/usr/lib/libswresample.so.3.5.100"
[sentry] DEBUG inspecting module "/usr/lib/ld-2.31.so"
[sentry] DEBUG inspecting module "linux-gate.so"
[sentry] DEBUG read 127 modules from /proc/self/maps
[sentry] DEBUG adding attachments to envelope
[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
[sentry] DEBUG executing task on worker thread
*   Trying 35.188.42.15:443...
* Connected to o420280.ingest.sentry.io (35.188.42.15) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
[sentry] INFO capturing event
[sentry] DEBUG merging scope into event
[sentry] DEBUG adding attachments to envelope
[sentry] DEBUG sending envelope
[sentry] DEBUG serializing envelope into buffer
[sentry] DEBUG serializing envelope into buffer
[sentry] DEBUG dumped 1 in-flight envelopes to disk
[sentry] INFO crash has been captured


Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
@Swatinem
Copy link
Member

Are you able to reproduce this reliably? Would you mind trying with master? We recently merged some code that reworked the thread synchronization with the http thread, maybe you were hitting some unsafety there.

@Shamazo
Copy link
Author

Shamazo commented Jul 14, 2020

I can try with master now, this happened every time I tried it, both with gdb and without.

@Shamazo
Copy link
Author

Shamazo commented Jul 14, 2020

image
I appear to have the same issue on master.

@Swatinem
Copy link
Member

I’m on Arch at home, with a bit more recent curl (7.71.1)

Your log shows the basically the same as your screenshot, that you are crashing inside of curl itself.
My log looks like this:

[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
[sentry] DEBUG shutting down transport
[sentry] DEBUG shutting down background worker thread
[sentry] DEBUG submitting task to background worker thread
*   Trying 35.188.42.15:443...
* Connected to sentry.io (35.188.42.15) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* NPN, negotiated HTTP1.1
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Functional Software, Inc.; CN=sentry.io
*  start date: Jun  2 00:00:00 2020 GMT
*  expire date: Jun  7 12:00:00 2022 GMT
*  subjectAltName: host "sentry.io" matched cert's "sentry.io"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> POST /api/1041156/envelope/ HTTP/1.1
Host: sentry.io
User-Agent: sentry.native/0.3.4
Accept: */*
x-sentry-auth:Sentry sentry_key=xxx, sentry_version=7, sentry_client=sentry.native/0.3.4
content-type:application/x-sentry-envelope
content-length:317

* upload completely sent off: 317 out of 317 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 15 Jul 2020 15:45:00 GMT
< Content-Type: application/json
< Content-Length: 2
< Connection: keep-alive
< access-control-expose-headers: x-sentry-rate-limits, x-sentry-error, retry-after
< vary: Origin
< x-envoy-upstream-service-time: 2
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< 
{}* Connection #0 to host sentry.io left intact

Your log cuts off (well, continues from inside the signal handler) at the moment of fetching CA certificates?

Can you just verify that you can use curl from the CLI by running curl -I -v $YOUR_DSN? It should print mostly the same debug output as from inside of sentry.

@Shamazo
Copy link
Author

Shamazo commented Jul 15, 2020 •
edited by Swatinem
Loading

This is the output of curl -I -v $MY_DSN from my terminal.

curl -I -v $DSN                                                                                                                 
*   Trying 35.188.42.15:443...                                                                                                                                                                                                          
* Connected to o420280.ingest.sentry.io (35.188.42.15) port 443 (#0)                                                                                                                                                                    
* ALPN, offering h2                                                                                                                                                                                                                     
* ALPN, offering http/1.1                                                                                                                                                                                                               
* successfully set certificate verify locations:                                                                                                                                                                                        
*   CAfile: /etc/ssl/certs/ca-certificates.crt                                                                                                                                                                                          
  CApath: none                                                                                                                                                                                                                          
* TLSv1.3 (OUT), TLS handshake, Client hello (1):                                                                                                                                                                                       
* TLSv1.3 (IN), TLS handshake, Server hello (2):                                                                                                                                                                                        
* NPN, negotiated HTTP1.1                                                                                                                                                                                                               
* TLSv1.2 (IN), TLS handshake, Certificate (11):                                                                                                                                                                                        
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):                                                                                                                                                                                
* TLSv1.2 (IN), TLS handshake, Server finished (14):                                                                                                                                                                                    
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):                                                                                                                                                                               
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):                                                                                                                                                                             
* TLSv1.2 (OUT), TLS handshake, Next protocol (67):                                                                                                                                                                                     
* TLSv1.2 (OUT), TLS handshake, Finished (20):                                                                                                                                                                                          
* TLSv1.2 (IN), TLS handshake, Finished (20):                                                                                                                                                                                           
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384                                                                                                                                                                            
* ALPN, server did not agree to a protocol                                                                                                                                                                                              
* Server certificate:                                
*  subject: CN=*.ingest.sentry.io                                                                                                                                                                                                       
*  start date: Jul  1 01:07:26 2020 GMT                                                                                                                                                                                                 
*  expire date: Sep 29 01:07:26 2020 GMT                                                                                                                                                                                                
*  subjectAltName: host "o420280.ingest.sentry.io" matched cert's "*.ingest.sentry.io"                                                                                                                                                  
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3                                                                                                                                                                         
*  SSL certificate verify ok.                                                                                                                                                                                                           
* Server auth using Basic with user 'xxx'                                                                                                                                                                  
> HEAD /5338251 HTTP/1.1                                                                                                                                                                                                                
> Host: o420280.ingest.sentry.io                                                                                                                                                                                                        
> Authorization: Basic xxx                                                                                                                                                                     
> User-Agent: curl/7.70.0                                                                                                                                                                                                               
> Accept: */*                                                                                                                                                                                                                           
>                                                                                                                                                                                                                                       
* Mark bundle as not supporting multiuse                                                                                                                                                                                                
< HTTP/1.1 404 Not Found                                                                                                                                                                                                                
HTTP/1.1 404 Not Found                                                                                                                                                                                                                  
< Server: nginx                                                                                                                                                                                                                         
Server: nginx                                                                                                                                                                                                                           
< Date: Wed, 15 Jul 2020 16:28:26 GMT                                                                                                                                                                                                   
Date: Wed, 15 Jul 2020 16:28:26 GMT                                                                                                                                                                                                     
< Content-Type: text/html                                                                                                                                                                                                               
Content-Type: text/html                                                                                                                                                                                                                 
< Content-Length: 281445                                                                                                                                                                                                                
Content-Length: 281445                                                                                                                                                                                                                  
< Connection: keep-alive                                                                                                                                                                                                                
Connection: keep-alive                                                                                                                                                                                                                  
< ETag: "5ecc29cf-44b65"                                                                                                                                                                                                                
ETag: "5ecc29cf-44b65"                                                                                                                                                                                                                  
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload                                                                                                                                                               
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload                                                                                                                                                                 
                                                                                                                                                                                                                                        
<                                                                                                                                                                                                                                       
* Connection #0 to host o420280.ingest.sentry.io left intact

@Swatinem
Copy link
Member

Looking through the code again, I can see two issues that we have and could improve, I will create a PR tomorrow and lets see if that improves things.

One thing I notice is that curl_global_init explicitly says its not thread safe. We do all the curl init functions in the same thread as sentry_init, but I don’t know if your code does any curl things concurrently?

@Shamazo
Copy link
Author

Shamazo commented Jul 15, 2020

We don't use curl in our code base. We do construct several web requests manually, but I don't think they should interact with curl.

@Swatinem
Copy link
Member

I tried to correct the curl initialization order, and also added some more logging here: #341
Do you mind trying again with that PR?

@Shamazo
Copy link
Author

Shamazo commented Jul 16, 2020 •
edited
Loading

Running that PR:

[sentry] INFO using database path "<my build path>/.sentry-native"
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG starting backend
[sentry] DEBUG background worker thread started
[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
[sentry] DEBUG executing task on worker thread
[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
[sentry] INFO capturing event
[sentry] DEBUG merging scope into event
[sentry] DEBUG trying to read modules from /proc/self/maps
[sentry] DEBUG inspecting module "<my build path>/<my executable name>"
[sentry] DEBUG inspecting module "/usr/lib/libgpg-error.so.0.29.0"
[sentry] DEBUG inspecting module "/usr/lib/libpcre.so.1.2.12"
[sentry] DEBUG inspecting module "/usr/lib/libgcrypt.so.20.2.5"
[sentry] DEBUG inspecting module "/usr/lib/liblz4.so.1.9.2"
[sentry] DEBUG inspecting module "/usr/lib/libFLAC.so.8.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libresolv-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libkeyutils.so.1.9"
[sentry] DEBUG inspecting module "/usr/lib/libkrb5support.so.0.1"
[sentry] DEBUG inspecting module "/usr/lib/libffi.so.7.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libicudata.so.67.1"
[sentry] DEBUG inspecting module "/usr/lib/libgraphite2.so.3.2.1"
[sentry] DEBUG inspecting module "/usr/lib/libglib-2.0.so.0.6400.3"
[sentry] DEBUG inspecting module "/usr/lib/libjpeg.so.8.2.2"
[sentry] DEBUG inspecting module "/usr/lib/libasyncns.so.0.3.1"
[sentry] DEBUG inspecting module "/usr/lib/libsystemd.so.0.28.0"
[sentry] DEBUG inspecting module "/usr/lib/libsndfile.so.1.0.28"
[sentry] DEBUG inspecting module "/usr/lib/libcom_err.so.2.1"
[sentry] DEBUG inspecting module "/usr/lib/libk5crypto.so.3.1"
[sentry] DEBUG inspecting module "/usr/lib/libkrb5.so.3.3"
[sentry] DEBUG inspecting module "/usr/lib/libgssapi_krb5.so.2.2"
[sentry] DEBUG inspecting module "/usr/lib/libssl.so.1.1"
[sentry] DEBUG inspecting module "/usr/lib/libpsl.so.5.3.2"
[sentry] DEBUG inspecting module "/usr/lib/libssh2.so.1.0.1"
[sentry] DEBUG inspecting module "/usr/lib/libnghttp2.so.14.19.0"
[sentry] DEBUG inspecting module "/usr/lib/libcrypto.so.1.1"
[sentry] DEBUG inspecting module "/usr/lib/libhogweed.so.6.0"
[sentry] DEBUG inspecting module "/usr/lib/libnettle.so.8.0"
[sentry] DEBUG inspecting module "/usr/lib/libtasn1.so.6.6.0"
[sentry] DEBUG inspecting module "/usr/lib/libunistring.so.2.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libidn2.so.0.3.7"
[sentry] DEBUG inspecting module "/usr/lib/libp11-kit.so.0.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libicuuc.so.67.1"
[sentry] DEBUG inspecting module "/usr/lib/libpng16.so.16.37.0"
[sentry] DEBUG inspecting module "/usr/lib/libexpat.so.1.6.11"
[sentry] DEBUG inspecting module "/usr/lib/libharfbuzz.so.0.20600.7"
[sentry] DEBUG inspecting module "/usr/lib/libv4lconvert.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libdbus-1.so.3.19.11"
[sentry] DEBUG inspecting module "/usr/lib/pulseaudio/libpulsecommon-13.0.so"
[sentry] DEBUG inspecting module "/usr/lib/libXdmcp.so.6.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libXau.so.6.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libdb-5.3.so"
[sentry] DEBUG inspecting module "/usr/lib/libmvec-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libogg.so.0.8.4"
[sentry] DEBUG inspecting module "/usr/lib/libgomp.so.1.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libgcc_s.so.1"
[sentry] DEBUG inspecting module "/usr/lib/libstdc++.so.6.0.28"
[sentry] DEBUG inspecting module "/usr/lib/libcurl.so.4.6.0"
[sentry] DEBUG inspecting module "/usr/lib/librt-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libuuid.so.1.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libvdpau.so.1.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libva-x11.so.2.700.0"
[sentry] DEBUG inspecting module "/usr/lib/libva-drm.so.2.700.0"
[sentry] DEBUG inspecting module "/usr/lib/libssh.so.4.8.5"
[sentry] DEBUG inspecting module "/usr/lib/libsrt.so.1.4.1"
[sentry] DEBUG inspecting module "/usr/lib/libgnutls.so.30.28.0"
[sentry] DEBUG inspecting module "/usr/lib/libgmp.so.10.4.0"
[sentry] DEBUG inspecting module "/usr/lib/libbluray.so.2.2.0"
[sentry] DEBUG inspecting module "/usr/lib/libmodplug.so.1.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libbz2.so.1.0.8"
[sentry] DEBUG inspecting module "/usr/lib/libxml2.so.2.9.10"
[sentry] DEBUG inspecting module "/usr/lib/libfreetype.so.6.17.2"
[sentry] DEBUG inspecting module "/usr/lib/libfontconfig.so.1.12.0"
[sentry] DEBUG inspecting module "/usr/lib/libvidstab.so.1.1"
[sentry] DEBUG inspecting module "/usr/lib/libass.so.9.0.2"
[sentry] DEBUG inspecting module "/usr/lib/libvmaf.so"
[sentry] DEBUG inspecting module "/usr/lib/libfribidi.so.0.4.0"
[sentry] DEBUG inspecting module "/usr/lib/libXv.so.1.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libv4l2.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libSDL2-2.0.so.0.12.0"
[sentry] DEBUG inspecting module "/usr/lib/libpulse.so.0.21.1"
[sentry] DEBUG inspecting module "/usr/lib/libxcb-xfixes.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libxcb-shape.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libxcb-shm.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libxcb.so.1.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libdrm.so.2.4.0"
[sentry] DEBUG inspecting module "/usr/lib/libjack.so.0.0.28"
[sentry] DEBUG inspecting module "/usr/lib/libiec61883.so.0.1.1"
[sentry] DEBUG inspecting module "/usr/lib/librom1394.so.0.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libavc1394.so.0.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libraw1394.so.11.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libmfx.so.1.32"
[sentry] DEBUG inspecting module "/usr/lib/libva.so.2.700.0"
[sentry] DEBUG inspecting module "/usr/lib/libxvidcore.so.4.3"
[sentry] DEBUG inspecting module "/usr/lib/libx265.so.188"
[sentry] DEBUG inspecting module "/usr/lib/libx264.so.159"
[sentry] DEBUG inspecting module "/usr/lib/libvorbisenc.so.2.0.11"
[sentry] DEBUG inspecting module "/usr/lib/libvorbis.so.0.4.8"
[sentry] DEBUG inspecting module "/usr/lib/libtheoradec.so.1.1.4"
[sentry] DEBUG inspecting module "/usr/lib/libtheoraenc.so.1.1.2"
[sentry] DEBUG inspecting module "/usr/lib/libspeex.so.1.5.1"
[sentry] DEBUG inspecting module "/usr/lib/libopus.so.0.8.0"
[sentry] DEBUG inspecting module "/usr/lib/libopenjp2.so.2.3.1"
[sentry] DEBUG inspecting module "/usr/lib/libopencore-amrnb.so.0.0.3"
[sentry] DEBUG inspecting module "/usr/lib/libmp3lame.so.0.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libgsm.so.1.0.13"
[sentry] DEBUG inspecting module "/usr/lib/libaom.so.0"
[sentry] DEBUG inspecting module "/usr/lib/libz.so.1.2.11"
[sentry] DEBUG inspecting module "/usr/lib/libopencore-amrwb.so.0.0.3"
[sentry] DEBUG inspecting module "/usr/lib/libdav1d.so.4.0.1"
[sentry] DEBUG inspecting module "/usr/lib/liblzma.so.5.2.5"
[sentry] DEBUG inspecting module "/usr/lib/libwebp.so.7.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libwebpmux.so.3.0.5"
[sentry] DEBUG inspecting module "/usr/lib/libvpx.so.6.2.0"
[sentry] DEBUG inspecting module "/usr/lib/libsoxr.so.0.1.2"
[sentry] DEBUG inspecting module "/usr/lib/libc-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libpthread-2.31.so"
[sentry] DEBUG inspecting module "<...>/cmake-build-debug/sentry-native/libsentry.so"
[sentry] DEBUG inspecting module "/usr/lib/libm-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libasound.so.2.0.0"
[sentry] DEBUG inspecting module "/usr/lib/libXfixes.so.3.1.0"
[sentry] DEBUG inspecting module "/usr/lib/libXext.so.6.4.0"
[sentry] DEBUG inspecting module "/usr/lib/libX11.so.6.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libICE.so.6.3.0"
[sentry] DEBUG inspecting module "/usr/lib/libSM.so.6.0.1"
[sentry] DEBUG inspecting module "<...>/cmake-build-debug/fractal/input/libfractalInput.so"
[sentry] DEBUG inspecting module "/usr/lib/libdl-2.31.so"
[sentry] DEBUG inspecting module "/usr/lib/libpostproc.so.55.5.100"
[sentry] DEBUG inspecting module "/usr/lib/libavutil.so.56.31.100"
[sentry] DEBUG inspecting module "/usr/lib/libswscale.so.5.5.100"
[sentry] DEBUG inspecting module "/usr/lib/libavformat.so.58.29.100"
[sentry] DEBUG inspecting module "/usr/lib/libavfilter.so.7.57.100"
[sentry] DEBUG inspecting module "/usr/lib/libavdevice.so.58.8.100"
[sentry] DEBUG inspecting module "/usr/lib/libavcodec.so.58.54.100"
[sentry] DEBUG inspecting module "/usr/lib/libswresample.so.3.5.100"
[sentry] DEBUG inspecting module "/usr/lib/ld-2.31.so"
[sentry] DEBUG inspecting module "linux-gate.so"
[sentry] DEBUG read 127 modules from /proc/self/maps
[sentry] DEBUG adding attachments to envelope
[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
*   Trying 35.188.42.15:443...
* Connected to o420280.ingest.sentry.io (35.188.42.15) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
[sentry] INFO capturing event
[sentry] DEBUG merging scope into event
[sentry] DEBUG adding attachments to envelope
[sentry] DEBUG sending envelope
[sentry] DEBUG serializing envelope into buffer
[sentry] DEBUG serializing envelope into buffer
[sentry] DEBUG dumped 4 in-flight envelopes to disk
[sentry] INFO crash has been captured

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

and the back trace:
image

Thanks for helping to try and get this working. Let me know if there is any other useful information I can provide.

@Swatinem
Copy link
Member

hm, TBH, I’m completely at a loss right now. Its still kind of strange that it crashes inside of crypto code, and according to the logs, its at the time it loads the CA certs. Also the stacktrace points to that. Not sure what to make of that.

@Shamazo
Copy link
Author

Shamazo commented Jul 16, 2020

I am going to try two things, running this version of our code in a ubuntu VM and running a C program which is literally just a main function linking to the SDK with the most basic sentry example:

sentry_options_t *options = sentry_options_new();
    sentry_options_set_debug(options, true);
    sentry_options_set_dsn(options, "<dsn>");
    sentry_init(options);
    sentry_capture_event(sentry_value_new_message_event(
            /*   level */ SENTRY_LEVEL_INFO,
            /*  logger */ "custom",
            /* message */ "It works!"
    ));

Ultimately the application I am working on needs to run on OSX, Windows and Ubuntu, but it would be really nice if it ran on Manjaro as well.

@ghost
Copy link

ghost commented Jul 17, 2020

I tried to correct the curl initialization order, and also added some more logging here: #341
Do you mind trying again with that PR?

I have the exact same issue.
Yocto, kernel 4.4, arm64 ! and this PR fixes it 👍

@Swatinem
Copy link
Member

this PR fixes it 👍

That is good to hear. The PR got quite a bit bigger than expected, hope to land all of this pretty soon.

@Ramjii
Copy link

Ramjii commented Jul 28, 2020

I had what seems to be the same issue on Windows with 0.3.4, and pulling from master (this PR has been merged) seems to have also resolved the issue. Thanks a lot for being so reactive!

@Swatinem
Copy link
Member

Interesting. We don’t use curl on Windows. But we overhauled the shutdown logic which cleaned up quite a few things.

@Shamazo
Copy link
Author

Shamazo commented Jul 30, 2020

I resolved this issue on my end. In the end it ended up being a linking issue. I realized the codebase I am working in statically links its own libcrypto. When I changed our build to use the system libcrypto things worked as expected. All of this running on the current master. Thank you so much for your help.

@Swatinem
Copy link
Member

Thanks for keeping us up to date, will close this then.

@jorgenpt
Copy link

jorgenpt commented Feb 8, 2023

@Shamazo Thank you for posting your issue here! Just starting hitting this exact crash after upgrading our machines to Ubuntu 22.04 (which has openssl 3). It took me a bit to find it, since the stack was different -- we got a segfault with a stack that looks like this:

    #0 0x7fe9113194d8 in OPENSSL_sk_value (/lib/x86_64-linux-gnu/libcrypto.so.3+0x2064d8)
    #1 0x7fe91133ba19 in X509_STORE_add_lookup (/lib/x86_64-linux-gnu/libcrypto.so.3+0x228a19)
    #2 0x7fe911341f5e in X509_STORE_load_file_ex (/lib/x86_64-linux-gnu/libcrypto.so.3+0x22ef5e)

After trying a bunch of different things without success (and finding that using libcurl w/ gnutls fixed it), I stumbled across your issue, and realized that we're statically linking OpenSSL 1.1 into our application as well as loading sentry-native built with dynamic OpenSSL 3. Removing the statically linked OpenSSL 1.1 from the build resolved the issue.

@supervacuus supervacuus mentioned this issue Apr 26, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jorgenpt @Swatinem @Shamazo @Ramjii