PyCall only loads during the session in which it's built

[swt30]

See #167 (comment)

deps.jl contains:

const python = "python"
const libpython = "libpython3.4m"
const pyprogramname = wstring("/home/scott/.virtualenvs/py3/bin/python")
const pyversion_build = v"3.4.3"

const PYTHONHOME = "/home/scott/.virtualenvs/py3:/home/scott/.virtualenvs/py3"

const PyUnicode_AsUTF8String = :PyUnicode_AsUTF8String
const PyUnicode_DecodeUTF8 = :PyUnicode_DecodeUTF8

const PyString_FromString = :PyBytes_FromString
const PyString_AsString = :PyBytes_AsString
const PyString_Size = :PyBytes_Size
const PyString_Type = :PyBytes_Type
const PyInt_Type = :PyLong_Type
const PyInt_FromSize_t = :PyLong_FromSize_t
const PyInt_FromSsize_t = :PyLong_FromSsize_t
const PyInt_AsSsize_t = :PyLong_AsSsize_t

const Py_hash_t = Int64

const pyunicode_literals = true

[swt30]

Other info that may be useful:

• virtualenvwrapper doesn't appear to use PYTHONHOME, but ENV["PYTHONHOME"] gets set to the above value upon Pkg.build("PyCall"), at which point everything works fine
• after restarting Julia and triggering the failure by using PyCall, PyCall.PYTHONHOME exists but ENV["PYTHONHOME"] doesn't

[stevengj]

@swt30, ENV["PYTHONHOME"] is currently only set (in PyCall.__init__) while initializing Python, so it wouldn't be set after using PyCall completes.

I should also really unset it and other environment variables after Pkg.build("PyCall"), in order to avoid triggering #91.

It looks like the right thing is for me not to use the environment variable at all, but rather to use Py_SetPythonHome.

[stevengj]

Okay, I've pushed a patch that uses Py_SetPythonHome, can you try it and see if it fixes your problem?

[swt30]

Not yet, but I'm reinstalling the virtual environment with an updated version of virtualenvwrapper to see if that changes things.

[swt30]

No luck, I'm afraid! Same behaviour as before.

[stevengj]

Does setting ENV["PYTHONHOME"] manually before loading PyCall (see the value in PyCall/deps/deps.jl) work? I'm confused.

[swt30]

Still no, unfortunately. The plot just thickened:

julia> using PyCall
WARNING: error initializing module PyCall:
ReadOnlyMemoryError()

This is in the same place, using a Python 2 virtual environment instead of Python 3. I don't have enough debugging skill to look much more deeply into where the error is coming from, but I'm happy to try any other suggestions to figure out what's screwed up with my virtual environment.

[swt30]

For some reason, I am now getting only the ReadOnlyMemoryError with any Python, whether it's the system version or either virtual environment. Still in the same place (after the session restart). I'm confused too.

[swt30]

Rebuilt Julia and now the error is back to the first one: cglobal: could not find symbol PyCObject_FromVoidPtr in library libpython3.4m. I'm reverting to an earlier version of PyCall for now, but do let me know if you have any steps you want me to try.

[stevengj]

@swt30, does the error persist if you manually set the PYTHONHOME environment variable?

[swt30]

It does.

[swt30]

I'm on Julia master; would it be worthwhile bisecting on Julia to see if this behaviour starts showing up with a specific Julia commit? Could 46e60f3 have exposed a Julia bug?

[swt30]

Can verify that the behaviour is still around for me on v0.3.11, but not on another machine. I suppose something strange is going on with virtualenvwrapper. I'll see what happens if I disable it.

[stevengj]

That is so bizarre. Py_SetPythonHome is supposed to be the preferred & equivalent replacement for setting PYTHONHOME (which can screw up subprocesses). The source code of Python (which hasn't changed much in this part since Python 2.7, except for the change from char* to wchar_t*) only reads PYTHONHOME from Py_GetPythonHome, which always prefers the value set by SetPythonHome.

[swt30]

No change upon deactivating virtualenvwrapper - whether using the system python or a virtual environment of any kind, it's either a ReadOnlyMemoryError or could not find symbol. What on earth is going on with my system!?

[swt30]

As of 493b3f0, this is fixed in my Python 2.7. The error still persists on Python 3.4. It's apparently coming from line 72 in pyinit.jl.

[swt30]

PyCObject was deprecated in Python 2.7 and removed somewhere in 3.x. So the fallback to CObject at pyinit line 74 is failing. Do we need the new PyCapsule API here instead?

[swt30]

... but then it still comes back to the fact that ctypes is failing to load unless PyCall was built that session. It's just that python 2.7 allows the fallback to PyCObject, while 3.4 doesn't. I think this is the same as #178. Closing in favour of that until I get a better handle on the situation.

julia> using PyCall # with python 2.7
julia> pyimport("ctypes")
ERROR: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('/usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so: undefined symbol: PyFloat_Type',)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 10, in <module>
    from _ctypes import Union, Structure, Array

 in pyimport at /home/scott/.julia/v0.4/PyCall/src/PyCall.jl:81

julia> Pkg.build("PyCall")
INFO: Building PyCall
INFO: PyCall is using python (Python 2.7.9) at /usr/bin/python, libpython = libpython2.7

julia> pyimport("ctypes")
PyObject <module 'ctypes' from '/usr/lib/python2.7/ctypes/__init__.pyc'>

[swt30]

for reference, this issue on another project notes a similar problem using Python's C API on Ubuntu, relating to symbol resolution on library load. They end up putting in another dlopen call before initing Python as well, which is the workaround I'm currently using.