Boost.Python and segmentation faults with numpy 1.21

[bkpoon]

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

Due to NEP 42 being accepted as being the new datatype system for numpy 1.21, C++ extensions wrapped with Boost.Python will get segmentation faults. There is an issue created upstream by @jawsc-alan (boostorg/python#376). @dwpaley did the work to find the commit where the behavior changed. It is also possible to reproduce the behavior in numpy 1.20 by building with the NPY_USE_NEW_CASTINGIMPL environment variable.

Most likely, Boost.Python will need to be updated to handle the new datatype system.

PR to enable testing NEP 42 (numpy/numpy#17401)
PR to set NEP 42 as the default (numpy/numpy#18676)

Environment (conda list):

$ conda list

Details about conda and system ( conda info ):

$ conda info

[dwpaley]

A fix in numpy is described in this issue: boostorg/python#376 and I will open a numpy PR. The problem involved dereferencing a null pointer when checking convertibility of types (like boost_python ones) that haven't implemented the new numpy casting implementation.

[bkpoon]

So numpy>=1.21.5 has a temporary fix for this. Would it make sense to make a PR to https://github.com/conda-forge/conda-forge-repodata-patches-feedstock so that boost does not use numpy >=1.21,<1.21.5 if both dependencies are specified?

[jakirkham]

That could make sense. Though we will want to include that constraint in the package here as well

cc @conda-forge/core (in cases others have thoughts here)

[bkpoon]

Oops, numpy is always specified. I can make one or both PRs. Would run_constrained be the appropriate way of modifying the pin_compatible entry for numpy in this recipe? It looks like the additional arguments for pin_compatible are for including versions, not excluding versions.

[jakirkham]

Are people still running into this issue?

[dwpaley]

The problem is invisible with numpy >= 1.21.5 due to np PR 20616. Nominally that is a temporary fix that may be removed at some point. At the time I had made some type of reproducer in a container that could demonstrate this problem; I'm happy to try to pull it back together if you are interested in revisiting this.

[jakirkham]

Fair. Well I think the answer then is to fix in Boost.Python (as Sebastian put it in the issue). We can of course backport a patch once that makes its way upstream, but that would be the best use of time (if you have any and no worries if not)

[bkpoon]

We do nightly builds and tests of our conda package so we'll know if the temporary fix is no longer available pretty quickly. The Boost.Python and NumPy developers are aware of the issue, so I guess we can close this issue for now. Or should we keep it open for tracking?

[h-vetinari]

Closing this issue as obsolete. If there are still problems with numpy + libboost-python, please let us know!