SegFault while trying to load binary into database

[asenabouth]

Python Version: 2.6.6
Postgresql Version: 9.4.4
Psycopg2 Version: 2.6.1

I've been working on a script that uses psycopg2's binary function to convert and store it in the database in a bytea field. When the script tries to insert it into the database via an 'execute' command it results in a segmentation fault. This happens whether the update string gets mogrified or not.

Here is what I have in the Python code for the conversion and database insert steps:

for file in filelist:
read_file = open(file, 'rb').read()
binary_file = psycopg2.Binary(read_file)
row = [datetime_1, datetime_2, binary_file]
inject_str = sqlcon.mogrify("(%s, %s, %s)", row)
sqlcon.execute("INSERT INTO database_name (datetime_1, datetime_2, data) VALUES " + inject_str)

This is what Python prints as an error:

Segmentation fault (core dumped)

This is the output for the trace I got via gdb:

78 if (PyObject_CheckBuffer(self->wrapped)) {

Looking at Frame 78:

#0 0x0000000000000000 in ?? ()

This is the backtrace:

Program received signal SIGSEGV, Segmentation fault.
binary_quote (self=0x7ffff7fdec38, args=0x0) at psycopg/adapter_binary.c:78
#0 binary_quote (self=0x7ffff7fdec38, args=0x0) at psycopg/adapter_binary.c:78
#1 binary_getquoted (self=0x7ffff7fdec38, args=0x0) at psycopg/adapter_binary.c:138
#2 0x0000003d45843c63 in PyObject_Call (func= , arg=>, kw=) at Objects/abstract.c:2492
#3 0x0000003d45843d51 in call_function_tail (callable= , args=()) at Objects/abstract.c:2524
#4 0x0000003d4584459b in _PyObject_CallMethod_SizeT (o=, name=0x7ffff0ef1708 "getquoted", format=0x0) at Objects/abstract.c:2640
#5 0x00007ffff0ee91b6 in microprotocol_getquoted (obj=, conn=0xb1f670) at psycopg/microprotocols.c:249
#6 0x00007ffff0ee139e in _mogrify (var= [, , ], fmt=, curs=0x921718, new=0x7fffffffe1c8) at psycopg/cursor_type.c:247
#7 0x00007ffff0ee1bb3 in _psyco_curs_mogrify (self=0x921718, args=, kwargs=) at psycopg/cursor_type.c:593
#8 psyco_curs_mogrify (self=0x921718, args=,
#9 0x0000003d458d59e4 in call_function (f=, throwflag=) at Python/ceval.c:3794
#10 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#11 0x0000003d458d6b8f in fast_function (f=, throwflag=) at Python/ceval.c:3880
#12 call_function (f=, throwflag=) at Python/ceval.c:3815
#13 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#14 0x0000003d458d7657 in PyEval_EvalCodeEx (co=0x7ffff7f0a210, globals=, locals=, args=, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044
#15 0x0000003d458d7732 in PyEval_EvalCode (co=, globals=, locals=) at Python/ceval.c:545
#16 0x0000003d458f1bac in run_mod (mod=, filename=
#17 0x0000003d458f1c80 in PyRun_FileExFlags (fp=0x68c6d0, filename= 0x7fffffffeb7c "load_netcdf2psql.py", start=, globals= {'upload_files': , 'process': , _args=(...(truncated), locals= {'upload_files': , 'process': , _args=(... ...(truncated), closeit=1, flags=0x7fffffffe7b0) at Python/pythonrun.c:1344
#18 0x0000003d458f316c in PyRun_SimpleFileExFlags (fp=0x68c6d0, filename= 0x7fffffffeb7c "load_netcdf2psql.py", closeit=1, flags=0x7fffffffe7b0) at Python/pythonrun.c:948
#19 0x0000003d458ff8a2 in Py_Main (argc=, argv=) at Modules/main.c:618
#20 0x00007ffff71bfd5d in __libc_start_main () from /lib64/libc.so.6
#21 0x0000000000400649 in _start ()

I think it's a similar issue to this issue - is this what is happening here?

Thanks.

[fogzot]

Please, try to avoid calling mogrify(), just do:

for file in filelist:
    read_file = open(file, 'rb').read()
    binary_file = psycopg2.Binary(read_file)
    row = (datetime_1, datetime_2, binary_file)
    sqlcon.execute("INSERT INTO database_name (datetime_1, datetime_2, data) VALUES (%s, %s, %s)", row) 

If this works then the error is in mogrify() and we can try to fix it there.

[asenabouth]

I tried to insert using the method you suggested - unfortunately it still resulted in a segmentation fault.

Python:

Segmentation fault (core dumped)

GDB:

Program received signal SIGSEGV, Segmentation fault.
binary_quote (self=0x7ffff7fe0c38, args=0x0) at psycopg/adapter_binary.c:78
78 if (PyObject_CheckBuffer(self->wrapped)) {

Frame 78:

#0 0x0000000000000000 in ?? ()

Backtrace is the same as above.

If it's relevant -
File upload was working on a different machine running an earlier version of 2.6. I actually just noticed that the machine I've been having issues with was compiled with the lo64 flag while the other was not. Each upload file is 290MB so I am not sure if it would be a factor.

[dvarrazzo]

I think the file size is relevant yes. lo64 probably not, as you are not using large objects (for so big values maybe you should...)

[asenabouth]

Think I'll try large objects then. Thank you for your time.

[dvarrazzo]

I tried to reproduce the issue: I crashed my machine running out of memory but didn't get a segfault in psycopg.