Building for Python 3

[Vaakapallo]

Hi,

I've been trying to get PD working with Python 3 and even though I got this external to build with Python 3.6, PD says "can't load library". (A 2.7. build worked fine).

I heard that it is likely due to differences in the C API between Python 2.7 and Python 3, but I'm not too familiar with this area and lost on the topic.

Anyone have experience at this or any good resources? I found this, but don't really know where to go to from here.

[grrrr]

Hi Lassi, my impression is that the Python 3.6 headers are not used, but rather the 2.7 ones, which of course works until the linking stage. Then, linking with Pd used lazy linking (-undefined dynamic_lookup option), shadowing missing linker symbols. I am pretty sure the change to the Python 3 API is not as effortless as one might wish.

[grrrr]

I have made a python3 branch which makes tiny changes to the build system. It should still work with 2.7 but now use the proper paths to the Python headers.
Using it with 3.6 throws compilations errors immediately because of drastic changes in the Python3 API. These need to be fixed.

[Vaakapallo]

I went through the errors and fixed the ones I could. I have a commit in my fork that tries to fix the easiest changes, but had to comment out or inadequately fix some. f807ad6

It builds, but this version crashes PD when loaded (no surprises there). I was wondering if there's any way to get information on what is going wrong or some way to test that the code behaves like it did before the changes.

[grrrr]

Thanks, i have had a quick look but fail to understand a few replacements of old versus new functions. In any case, all adaptations of the code should be conditional so that compilation works for both python2 and python3 APIs.

[Vaakapallo]

I responded to the comments in the commit. And made a new commit with Unicode checks: 3908b2b

Yeah, compatibility with Python 2 and 3 would be ideal, but unfortunately that is probably beyond the scope of my abilities. I currently just need something that works with Python3, which would be a good milestone to getting something that works for both.

Though it seems I've hit a dead end. Without a really good understanding of the system (or comprehensive way to test it), replacing functions in the code will mostly lead to nonsensical changes. Gaining the understanding solely through reading code with my level in C++ would take a very long time. Currently I only see that PD crashes when I try to load py with the changes.

As it stands, my fork points out parts of the code that need some sort of update for Python3 compatibility. Don't really know how to move forward from here.

[Vaakapallo]

Found another quick resource for porting

[Vaakapallo]

module_obj = Py_InitModule(const_cast<char *>(PYEXT_MODULE), func_tbl);
module_dict = PyModule_GetDict(module_obj); // borrowed reference

The replacement for this part in pybase.cpp is the thing that is crashing. And it is related to the new module initialization system, but I don't know how to make it work correctly on the new system.

[ktatar]

Hi all,

I am trying to build the source using flext. I couldn't get rid of the following error:

source/pyprefix.h:20:10: fatal error: 'Python/Python.h' file not found
#include <Python/Python.h>

any ideas?

[umlaeute]

Python2 is reaching EOL in about 100 days from now.

Debian is currently fading out all packages that still require Python2 (including py).

Maybe it's time :-)

[grrrr]

Yes, absolutely. I have been working on a branch for python3 development. https://github.com/grrrr/py/tree/python3
However, module management has fundamentally changed and i am proceeding slower than i would want.

[umlaeute]

i became aware of the python3 branch after i posted.
but looking at it (and amending it to linux) i notice that porting is indeed not trivial... (so i decided to stay silent)

[ahihi]

hello,

i forked @grrrr's python3 branch and am working on it here https://github.com/SopiMlab/py/tree/python3 (also includes some build tweaks for Conda support on Mac)

i can build the code for Python 3.7 and 2.7, but i have some runtime issues...

on 3.7, the external initializes successfully. when i open simple-1.pd, the Python module loads (verified by putting a print at the end), but creating the pyext Pd object fails with no other information than "couldn't create".

on 2.7, i get a crash on init like this:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000008
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [44060]

VM Regions Near 0x8:
--> 
    __TEXT                 0000000100000000-0000000100162000 [ 1416K] r-x/rwx SM=COW  /var/folders/*/Pd-0.50-0.app/Contents/Resources/bin/pd

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libpython2.7.dylib            	0x0000000104a360db PyDict_SetItemString + 59
1   py.pd_darwin                  	0x00000001006f5592 pyext::Setup(flext_class*) + 1266
2   py.pd_darwin                  	0x00000001007025d2 flext_obj_multi::obj_add(bool, bool, bool, bool, char const*, char const*, void (*)(flext_class*), flext_obj_multi* (*)(int, _atom*), void (*)(flext_hdr*), int, ...) + 946
3   py.pd_darwin                  	0x00000001006f5044 pyext_setup() + 84
4   py.pd_darwin                  	0x00000001006fc74d pybase::lib_setup() + 429
5   py.pd_darwin                  	0x0000000100702147 flext_obj_multi::lib_init(char const*, void (*)()) + 55
6   pd                            	0x00000001000a7bfa sys_do_load_lib + 618
7   ???                           	0x612f73726573552f 0 + 7002942880749933871

i guess the next step is to get a debugging environment set up :) i found the instructions on the Pd site, but if anyone has experience doing this with pyext specifically, any tips are appreciated!

[umlaeute]

for me, it has been crashing with python2.7 for a while. given that python2 is dead now, i wouldn't invest time in fixing it

[ahihi]

i figured out how to debug with lldb and i'm making progress :) our fork now runs at least the examples simple-1, simple-2, simple-3, script-1 and thread-1 with Python 3 (though i am sure there are bugs).

notably, the buffer examples are currently broken. i will work on that next monday.

in the interest of staying compatible with existing code, i am still making sure pyext builds against Python 2, but not currently testing on it. i imagine the crashes will not be hard to fix.

[grrrr]

Hi @ahihi , just wanted to give a thumbs up... it's a great effort, will check it out shortly!

[ahihi]

i am happy to report that our fork is mostly working on both python 3 and 2! 🎉

i have tested with:

• macOS 10.14.6
• Pd 0.50
• python 3.7 & 2.7 (both from conda); 2.6 and earlier are definitely broken

remaining issues that i know of:

• methods from the pyext module are no longer copied into pyext._class. this could probably be reimplemented, but personally i find it kind of unnecessary, so for now i just changed all references to these methods in the examples.
• Pd array graphics fail to redraw after update in some cases. i don't quite understand how this is supposed to work — the dirty state is only passed on from pySamplebuffer to the underlying flext::Buffer in buffer_dealloc(), so i guess deallocations are happening less often than before? maybe a refcounting bug...
• in tcltk.pd, pressing the red button causes a crash, but i also see this on master, so i didn't spend any time on it.

one major issue i ran into was that the pySamplebuffer methods didn't correctly handle the memory layout of arrays in Pd-float32, causing arrays to end up with zeroes every other sample. i believe this also happens on master. i implemented a fix here, but haven't tested with Pd-float64 yet.

[omarcostahamido]

yay! really looking forward to be able to use Python3.7 with Max!

[jeremiahrose]

ahahi, unfortunately the python3 branch in your fork would not compile on my setup - here is the error:

"source/pyext.cpp:709:1: error: designator order for field ‘_typeobject::tp_basicsize’ does not match declaration order in ‘PyTypeObject’ {aka ‘_typeobject’} "

Your branch was my last hope at getting some Python3-only code running in pd. At this stage it looks like pyext is unusable as python 2 is no longer widely supported.

[ahihi]

@jeremiahrose which OS and Python version are you running? and how is your Python installed (e.g. installer from python.org, conda, etc)?

[jeremiahrose]

@ahihi Python 3.7.3 installed via apt on Rasbian 10 (Buster).

[ahihi]

@jeremiahrose our main target so far is macOS and unfortunately we don't currently have the resources to work on other plarforms :/

i may give it a try on a Pi when i have some spare time, but can't promise anything...

[omarcostahamido]

btw, @ahihi did you ever get to compile it successfully as an external for max?

[ahihi]

@omarcostahamido afraid not, i don't have Max.

[fhchl]

@ahihi I am just trying to build your fork but getting the following error message:

make PDINCLUDEDIR="../purr-data/pd/src" CPPFLAGS="-I ../flext/source -I /usr/include/python2.7"

gives

source/pybuffer.cpp: In function ‘int buffer_getbuffer(PyObject*, Py_buffer*, int)’:
source/pybuffer.cpp:261:54: error: ‘PY_NUMPY_BUFFER_FORMAT’ was not declared in this scope
  261 |     view->format = (flags & PyBUF_FORMAT) ? (char *) PY_NUMPY_BUFFER_FORMAT : NULL;
      |                                                      ^~~~~~~~~~~~~~~~~~~~~~

The same error occurs, if I include usr/include/python3.7m. Numpy is installed for both Python versions via pip. System is Ubuntu 19.04.

Any ideas?

[ahihi]

@fhchl i haven't had the chance to test on linux at all, but from a quick glance at the code, PY_NUMPY_BUFFER_FORMAT being not declared suggests that either PY_NUMPY is unset or PY_NUMARRAY is also set, which overrides the numpy code path.

[umlaeute]

@ahihi could you please allow issues in your fork? i think we should discuss all problems with the port in the fork (probably in multiple issues), rather than abusing this ticket...

[ahihi]

@umlaeute thanks for the suggestion, i didn't know that was an option! 😅 i don't have admin access but i will ask my supervisor to do it.

[ahihi]

just wanted to note that

• issues are open at https://github.com/SopiMlab/py/issues
• i have the linux build working as of today. i ran into the same field order problem @jeremiahrose described and have fixed it in https://github.com/SopiMlab/py/tree/python3