"Segmentation Fault" happened when import faiss

[loner233]

Summary

"Segmentation Fault" happened when use azure tts sdk and faiss at the same time.

Platform

I'm not sure this problem can be reproducted on other platform
OS: Ubuntu 18.04

Faiss version: 1.7.2

Installed from: anaconda

Faiss compilation options: Not compiled from source
conda install -c pytorch faiss-cpu

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

import azure.cognitiveservices.speech as speechsdk
import faiss
import faulthandler
faulthandler.enable()

speech_config = speechsdk.SpeechConfig(subscription="hidden", region="chinanorth2")
synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=None)
result = synthesizer.speak_text_async('hello, world').get()

(test3) root@yinjiakang-devbox:/mnt3/demo/test# python test.py 
Fatal Python error: Segmentation fault

Current thread 0x00007f4f8f21a0c0 (most recent call first):
  File "/root/miniconda3/envs/test3/lib/python3.6/site-packages/faiss/swigfaiss_avx2.py", line 94 in __next__
  File "/root/miniconda3/envs/test3/lib/python3.6/site-packages/azure/cognitiveservices/speech/speech_py_impl.py", line 5428 in audio_data
  File "/root/miniconda3/envs/test3/lib/python3.6/site-packages/azure/cognitiveservices/speech/speech.py", line 1039 in __init__
  File "/root/miniconda3/envs/test3/lib/python3.6/site-packages/azure/cognitiveservices/speech/speech.py", line 504 in get
  File "test.py", line 8 in <module>
Segmentation fault

And if I comment the import faiss, the code is passed

import azure.cognitiveservices.speech as speechsdk
# import faiss
import faulthandler
faulthandler.enable()

speech_config = speechsdk.SpeechConfig(subscription="hidden", region="chinanorth2")
synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=None)
result = synthesizer.speak_text_async('hello, world').get()

And if I comment the azure tts code, the code is passed too

# import azure.cognitiveservices.speech as speechsdk
import faiss
import faulthandler
faulthandler.enable()

# speech_config = speechsdk.SpeechConfig(subscription="hidden", region="chinanorth2")
# synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=None)
# result = synthesizer.speak_text_async('hello, world').get()
print("everything is ok")

[mdouze]

So I assume speech_py_impl.py is the source code you show, otherwise there is no reason that faiss gets called from Azure code.

Would it be possible to get a C++ stack trace of the error with a debugger, to give an idea where the conflict is?

[loner233]

The speech_py_impl.py is not my source code, it's from azure-cognitiveservices-speech which installed through azure's doc install-the-speech-sdk-for-python

I'm confused too, like you said there is no reason azure call faiss when azure's dependencies owns no faiss.

And, any recommended tools for debug python's C stack?

[wx257osn2]

It seems that azure-cognitiveservices-speech also uses swig . I don't know swig well, but if swig is singleton software and it can't handle multiple calls of SwigPyIterator_siwgregister ,

import azure.cognitiveservices.speech as speechsdk
import faiss

will overwrite SwigPyIterator as faiss's one, then _speech_py_impl will call unexpected SwigPyIterator_siwgregister___next__ .

[wx257osn2]

According to SWIG doc Section 15.3, it seems that there needs some devices to use multiple swig modules, doesn't it?

[mdouze]

Excellent, thanks for the debugging. So what can we do?

[mdouze]

My suggestion as a workaround would be to call either speechsdk or faiss as a subprocess of the main code with a pool of a single process
https://docs.python.org/3/library/multiprocessing.html

the coordination between multiple swig modules lined out in the doc is only possible if their compilation is coordinated, this is not possible with Faiss and speechsdk.

Ideally it would be possible to make the symbols used by the two SWIG .so files completely disjoint (eg. with some module-specific prefix). However this functionality is not implemented in SWIG I think.

[wx257osn2]

the coordination between multiple swig modules lined out in the doc is only possible if their compilation is coordinated, this is not possible with Faiss and speechsdk.

I agree this. That is technically possible on the point of view from SWIG, but it would be practically impossible when at least speechsdk is not written to coexist with other SWIG modules.

Another option would be to stop using SWIG in faiss and generate Python bindings using pybind11 or something like that, but even if this project will decide to do it, there would be a lot of work... Anyway, the workaround using subprocess looks better way to go at moment IMO.

[loner233]

Thanks, I will try to use speechsdk in a single process pool, it seems the easiest way.