Segfault possibly caused by sentry-python 2.3

[larsent]

I'm creating a discussion instead of an issue because I'm not positive this is related to sentry-python, but I'm not sure how to debug further. I'd welcome advice.

I'm running Django + Celery. After upgrading sentry-python from 1.32.0 to 2.3.1, I started getting several (~3-5) errors per day like "WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV) Job: 3889.')" in my celery tasks.

I'm using Redis as both my task broker and results backend, and running the following

celery==5.4.0
redis==5.0.1
django==5.0.6
sentry-sdk==2.3.1

My celery worker's memory doesn't seem to spike, so I think I've ruled that out as a cause.

I used faulthandler to dump the stack, and pasted the output below. You'll notice code from segment's and scout's sdks in addition to sentry here, but these errors also occur without those sdks, so I don't think they are suspects.

After downgrading sentry-sdk to 2.0.1 yesterday, I have not seen this error again. Again, the error started after the 1.32.0 to 2.3.1 upgrade. That's why my best theory so far is that this is somehow related to sentry-python. Do you have any advice for debugging this further? Thanks.

CRITICALFatal Python error: Segmentation fault
INFO
INFOThread 0x00007fcd91248700 (most recent call first):
INFO  File "/usr/local/lib/python3.11/threading.py", line 331 in wait
INFO  File "/usr/local/lib/python3.11/queue.py", line 180 in get
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/segment/analytics/consumer.py", line 99 in next
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/segment/analytics/consumer.py", line 68 in upload
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/segment/analytics/consumer.py", line 57 in run
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 92 in _run_old_run_func
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 99 in run
INFO  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFOThread 0x00007fcd91e49700 (most recent call first):
INFO  File "/usr/local/lib/python3.11/threading.py", line 327 in wait
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/_queue.py", line 240 in get
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/worker.py", line 131 in _target
INFO  File "/usr/local/lib/python3.11/threading.py", line 982 in run
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 92 in _run_old_run_func
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 99 in run
INFO  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFOThread 0x00007fcd9264a700 (most recent call first):
INFO  File "/usr/local/lib/python3.11/threading.py", line 331 in wait
INFO  File "/usr/local/lib/python3.11/threading.py", line 629 in wait
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/scout_apm/core/samplers/thread.py", line 38 in run
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 92 in _run_old_run_func
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 99 in run
INFO  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFOThread 0x00007fcd92e4b700 (most recent call first):
INFO  File "/usr/local/lib/python3.11/threading.py", line 331 in wait
INFO  File "/usr/local/lib/python3.11/queue.py", line 180 in get
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/scout_apm/core/agent/socket.py", line 76 in run
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 92 in _run_old_run_func
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 99 in run
INFO  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFOCurrent thread 0x00007fcd93b4c700 (most recent call first):
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 262 in extract_stack
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 771 in <listcomp>
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 770 in _sample_stack
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 885 in run
INFO  File "/usr/local/lib/python3.11/threading.py", line 982 in run
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 92 in _run_old_run_func
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 99 in run
INFO  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFOThread 0x00007fcd9ca6c700 (most recent call first):
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/monitor.py", line 57 in _thread
INFO  File "/usr/local/lib/python3.11/threading.py", line 982 in run
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 92 in _run_old_run_func
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 99 in run
INFO  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFOThread 0x00007fcda0dbd740 (most recent call first):
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/connection.py", line 442 in send_packed_command
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/connection.py", line 464 in send_command
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/client.py", line 512 in _send_command_parse_response
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/client.py", line 537 in <lambda>
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/retry.py", line 46 in call_with_retry
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/client.py", line 536 in execute_command
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/redis/_sync_common.py", line 98 in sentry_patched_execute_command
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/scout_apm/instruments/redis.py", line 70 in wrapped_execute_command
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/commands/core.py", line 1829 in get
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 376 in get
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 995 in _get_task_meta_for
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 973 in _store_result
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 526 in store_result
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 157 in mark_as_done
DEBUG  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 520 in trace_task
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 326 in _inner
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1711 in runner
DEBUG  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 651 in fast_trace_task
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 361 in workloop
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 430 in sentry_workloop
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 291 in __call__
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/process.py", line 110 in run
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/process.py", line 323 in _bootstrap
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/popen_fork.py", line 77 in _launch
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/popen_fork.py", line 22 in __init__
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/context.py", line 331 in _Popen
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/process.py", line 120 in start
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 1157 in _create_worker_process
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/concurrency/asynpool.py", line 482 in _create_worker_process
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 1045 in __init__
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/concurrency/asynpool.py", line 464 in __init__
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/concurrency/prefork.py", line 109 in on_start
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/concurrency/base.py", line 130 in start
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bootsteps.py", line 365 in start
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bootsteps.py", line 116 in start
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/worker/worker.py", line 202 in start
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bin/worker.py", line 356 in worker
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bin/base.py", line 135 in caller
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/decorators.py", line 33 in new_func
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 783 in invoke
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1434 in invoke
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1688 in invoke
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1078 in main
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1157 in __call__
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bin/celery.py", line 236 in main
INFO  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/__main__.py", line 15 in main
INFO  File "/opt/render/project/src/.venv/bin/celery", line 8 in <module>
INFO
INFOExtension modules: scout_apm.core._objtrace, psutil._psutil_linux, psutil._psutil_posix, _cffi_backend, psycopg2._psycopg, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, lxml._elementpath, lxml.etree (total: 10)
ERROR[2024-05-29 22:24:43,829: ERROR/MainProcess] Process 'ForkPoolWorker-2' pid:83 exited with 'signal 11 (SIGSEGV)'
ERROR[2024-05-29 22:24:43,854: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV) Job: 3889.')

[sl0thentr0py]

@larsent this is gonna be tricky to figure out, but I have a hunch it's some interaction between our ThreadingIntegration, our profiler and the Celery workers.

First, can you tell me how you initialize Sentry in your Celery workers and the sentry_sdk.init configuration?

Second, I'd ask you to disable the ThreadingIntegration by setting default_integrations=False and see if the problem persists. That will help us narrow down the problem.

cc @antonpirker

[larsent]

@sl0thentr0py just got the segfault again with the above integrations.

CRITICAL
Fatal Python error: Segmentation fault
INFO
INFO
Thread 0x00007efe052c2700 (most recent call first):
INFO
  File "/usr/local/lib/python3.11/threading.py", line 327 in wait
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/_queue.py", line 240 in get
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/worker.py", line 131 in _target
INFO
  File "/usr/local/lib/python3.11/threading.py", line 982 in run
INFO
  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO
  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFO
Thread 0x00007efe05ac3700 (most recent call first):
INFO
  File "/usr/local/lib/python3.11/threading.py", line 331 in wait
INFO
  File "/usr/local/lib/python3.11/queue.py", line 180 in get
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/segment/analytics/consumer.py", line 99 in next
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/segment/analytics/consumer.py", line 68 in upload
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/segment/analytics/consumer.py", line 57 in run
INFO
  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO
  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFO
Current thread 0x00007efe062c4700 (most recent call first):
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 262 in extract_stack
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 771 in <listcomp>
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 770 in _sample_stack
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 885 in run
INFO
  File "/usr/local/lib/python3.11/threading.py", line 982 in run
INFO
  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO
  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFO
Thread 0x00007efe06ac5700 (most recent call first):
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/monitor.py", line 57 in _thread
INFO
  File "/usr/local/lib/python3.11/threading.py", line 982 in run
INFO
  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
INFO
  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
INFO
INFO
Thread 0x00007efe0f010740 (most recent call first):
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/_parsers/socket.py", line 65 in _read_from_socket
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/_parsers/socket.py", line 115 in readline
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 25 in _read_response
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 15 in read_response
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/connection.py", line 500 in read_response
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/client.py", line 553 in parse_response
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/client.py", line 513 in _send_command_parse_response
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/client.py", line 537 in <lambda>
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/retry.py", line 46 in call_with_retry
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/client.py", line 536 in execute_command
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/redis/commands/core.py", line 1829 in get
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 376 in get
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 995 in _get_task_meta_for
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 973 in _store_result
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 526 in store_result
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 157 in mark_as_done
DEBUG
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 520 in trace_task
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 326 in _inner
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1711 in runner
DEBUG
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 651 in fast_trace_task
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 361 in workloop
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 430 in sentry_workloop
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 291 in __call__
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/process.py", line 110 in run
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/process.py", line 323 in _bootstrap
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/popen_fork.py", line 77 in _launch
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/popen_fork.py", line 22 in __init__
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/context.py", line 331 in _Popen
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/process.py", line 120 in start
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 1157 in _create_worker_process
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/concurrency/asynpool.py", line 482 in _create_worker_process
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 1045 in __init__
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/concurrency/asynpool.py", line 464 in __init__
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/concurrency/prefork.py", line 109 in on_start
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/concurrency/base.py", line 130 in start
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bootsteps.py", line 365 in start
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bootsteps.py", line 116 in start
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/worker/worker.py", line 202 in start
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bin/worker.py", line 356 in worker
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bin/base.py", line 135 in caller
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/decorators.py", line 33 in new_func
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 783 in invoke
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1434 in invoke
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1688 in invoke
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1078 in main
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1157 in __call__
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/bin/celery.py", line 236 in main
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/celery/__main__.py", line 15 in main
INFO
  File "/opt/render/project/src/.venv/bin/celery", line 8 in <module>
INFO
[2024-06-11 18:45:43,281: INFO/ForkPoolWorker-3] Task apps.analytics.tasks.update_for_practiceuser[537eb88c-40ed-44ee-86a0-ce92e0b5c3e0] succeeded in 0.008800370153039694s: (282, 107)
INFO
[2024-06-11 18:45:43,282: INFO/ForkPoolWorker-5] Task apps.analytics.tasks.update_for_practiceuser[3fa409a7-d213-42c6-bdf9-a70f412275f4] succeeded in 0.01177242398262024s: (7, 84)
INFO
[2024-06-11 18:45:43,282: INFO/ForkPoolWorker-1] Task apps.analytics.tasks.update_for_practiceuser[5aad7c18-4e36-4d71-ab12-e420b568ff76] succeeded in 0.011021788232028484s: (223, 84)
INFO
INFO
Extension modules: psycopg2._psycopg, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, _cffi_backend, lxml._elementpath, lxml.etree (total: 7)
ERROR
[2024-06-11 18:45:43,383: ERROR/MainProcess] Process 'ForkPoolWorker-2' pid:66 exited with 'signal 11 (SIGSEGV)'
ERROR
[2024-06-11 18:45:43,499: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV) Job: 2165.')
INFO
Traceback (most recent call last):
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 1264, in mark_as_worker_lost
INFO
    raise WorkerLostError(
INFO
billiard.einfo.ExceptionWithTraceback: 
INFO
"""
INFO
Traceback (most recent call last):
INFO
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/billiard/pool.py", line 1264, in mark_as_worker_lost
INFO
    raise WorkerLostError(
INFO
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV) Job: 2165.
INFO
"""

[sl0thentr0py]

Okay, so it's not our ThreadingIntegration, we can rule that out.

I do see that you have profiling on and I can imagine some weird threading condition when the profiler is collecting samples, could you try without profiles_sample_rate once again?

[antonpirker]

Hey @larsent !

@sl0thentr0py is out for a week or so, so he told me to follow up.

As he said the only thing that looks suspicious now is the profiler. Could you try to remove profiles_sample_rate to disable the profiler and try again?

Thanks so much for your help debugging this!

[antonpirker]

@Zylphrex have you by any chance encountered the profiler segfaulting a Celery worker?

[antonpirker]

The only thing with the profiler and and a segfault was this: #2386

But the profiler has not been touched in a while and as downgrading to 2.0.1 did get rid of the problem, then it is probably not related to the profiler, because the profiler has not been touched since then...