CMake finding Python library and Python interpreter mismatch during pybind11 build

[jthetzel]

CMake finds mismatched PythonLibs and PythonInterp on my machine with multiple Python installations. PythonLibs finds 2.7.10, while PythonInterp finds 2.7.11:

$ cmake .
-- Setting build type to 'MinSizeRel' as none was specified.
-- Found PythonLibs: /usr/lib/libpython2.7.dylib (found version "2.7.10")
CMake Error at /usr/local/Cellar/cmake/3.4.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find PythonInterp: Found unsuitable version "2.7.11", but
  required is exact version "2.7.10" (found /usr/local/bin/python2.7)
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.4.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:386 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.4.1/share/cmake/Modules/FindPythonInterp.cmake:162 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:37 (find_package)

-- Configuring incomplete, errors occurred!
See also "/Users/jthetzel/src/pybind11/CMakeFiles/CMakeOutput.log".

Specifying -DPYTHON_INCLUDE_DIR seems to bypass the error, though it warns that EXACT is being ignored. However, PythonInterp now finds Python 3.5.1:

$ rm CMakeCache.txt
$ cmake -DPYTHON_INCLUDE_DIR=/usr/local/bin/python .
-- Setting build type to 'MinSizeRel' as none was specified.
-- Found PythonLibs: /usr/lib/libpython2.7.dylib
CMake Warning (dev) at CMakeLists.txt:37 (find_package):
  Ignoring EXACT since no version is requested.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found PythonInterp: /usr/local/bin/python3.5 (found version "3.5.1")
-- Performing Test HAS_CPP14_FLAG
-- Performing Test HAS_CPP14_FLAG - Success
-- Performing Test HAS_CPP11_FLAG
-- Performing Test HAS_CPP11_FLAG - Success
-- Performing Test HAS_LTO_FLAG
-- Performing Test HAS_LTO_FLAG - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/jthetzel/src/pybind11

I have three Pythons installed, 2.7.11 and 3.5.1 via homebrew in /usr/local/bin/, and 2.7.10 via the OSX system installation in /usr/bin/:

$ which -a python
/usr/local/bin/python
/usr/bin/python

$ which -a python3
/usr/local/bin/python3

$ /usr/local/bin/python --version
Python 2.7.11
$ /usr/bin/python --version
Python 2.7.10
$ /usr/local/bin/python3 --version
Python 3.5.1

/usr/local/bin/python is expected to be found first in PATH:

$ echo $PATH
/Users/jthetzel/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

$ python --version
Python 2.7.11

From a related closed pull request, this appears to be the fault of CMake ( https://github.com/wjakob/pybind11/pull/65 ), not specifically pybind11. Still, it would be nice to have a safe workaround, as having homebrew and system Python installations will likely be common for users in OSX environments. I'll take a closer look tonight.

Some version information:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.3
BuildVersion:   15D21

$ cmake --version
cmake version 3.4.1

$ git rev-parse --short HEAD
4f8902a

[wjakob]

After a number of attempts trying to patch up the detection logic, I welcome external patches to fix this.

[jthetzel]

After reading through the numerous related bug reports involving homebrew, Python, and CMake, especially Homebrew/legacy-homebrew#25118 and http://www.itk.org/Bug/view.php?id=14809 , which you have contributed to, I developed a strong headache and stopped. I especially enjoyed this four year-old quote: "Basically, CMake clings to Frameworks like a drowning sailor on OSX" from Homebrew/legacy-homebrew#10393 (comment) . The issue persists in CMake 3.5.0-rc1.

For now, I can ask CMake to use the OSX system Python interpreter with cmake -DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python . . I'll take another stab at a patch this weekend. Otherwise, the issue should be closed as upstream.

[jkhoogland]

I have some cmake logic I wrote to handle use of Anaconda python here.

It is not exactly your use case, but it might give you some idea how to proceed.

[wjakob]

Yes,that looks useful. Supporting the default Python distribution is obviously a requirement ;)

[ibell]

You might also want to investigate FindPythonLibsNew.cmake . For instance from here: https://github.com/usnistgov/REFPROP-wrappers/blob/master/cmake/FindPythonLibsNew.cmake

It helped me with some of these issues.

[ideoforms]

Encountered the same problem here. If you need to build against the non-system (Homebrew) Python install, this workaround does the trick:

cmake -DPYTHON_LIBRARY=$(python-config --prefix)/lib/libpython2.7.dylib -DPYTHON_INCLUDE_DIR=$(python-config --prefix)/include/python2.7 .

[wjakob]

Closing this as WONTFIX, as it's not a pybind11 issue.

[MrBean947724451]

Delete CMakeLists.txt which will cache make config .If the first time you use python 2.7 ,the file can remerber python path,and use it . so JUST DELETE CMAKELISTS.TXT!

[ibell]

Also, it can be helpful to specify the python executable directly like -DPYTHON_EXECUTABLE=/path/to/python/exe

…

On Wed, Dec 20, 2017 at 6:13 AM, MrBean947724451 ***@***.***> wrote: Delete CMakeLists.txt which will cache make config .If the first time you use python 2.7 ,the file can remerber python path,and use it . so JUST DELETE CMAKELISTS.TXT! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#99 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABxha-OJ0-DD2ecc9NAvxLiU975-6qXaks5tCQfngaJpZM4HV3JL> .

[MrBean947724451]

i solve my build error by clearing makelists 邮箱：[email protected] 签名由 网易邮箱大师 定制 On 12/21/2017 11:56, Ian Bell wrote: Also, it can be helpful to specify the python executable directly like -DPYTHON_EXECUTABLE=/path/to/python/exe On Wed, Dec 20, 2017 at 6:13 AM, MrBean947724451 <[email protected]> wrote:

Delete CMakeLists.txt which will cache make config .If the first time you use python 2.7 ,the file can remerber python path,and use it . so JUST DELETE CMAKELISTS.TXT! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#99 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABxha-OJ0-DD2ecc9NAvxLiU975-6qXaks5tCQfngaJpZM4HV3JL> .

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread. {"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/pybind/pybind11","title":"pybind/pybind11","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/pybind/pybind11"}},"updates":{"snippets":[{"icon":"PERSON","message":"@ibell in #99: Also, it can be helpful to specify the python executable directly like\n-DPYTHON_EXECUTABLE=/path/to/python/exe\n\nOn Wed, Dec 20, 2017 at 6:13 AM, MrBean947724451 \[email protected]\u003e\nwrote:\n\n\u003e Delete CMakeLists.txt which will cache make config .If the first time you\n\u003e use python 2.7 ,the file can remerber python path,and use it . so JUST\n\u003e DELETE CMAKELISTS.TXT!\n\u003e\n\u003e —\n\u003e You are receiving this because you commented.\n\u003e Reply to this email directly, view it on GitHub\n\u003e \u003chttps://github.com/pybind/pybind11/issues/99#issuecomment-353059633\u003e, or mute\n\u003e the thread\n\u003e \u003chttps://github.com/notifications/unsubscribe-auth/ABxha-OJ0-DD2ecc9NAvxLiU975-6qXaks5tCQfngaJpZM4HV3JL\u003e\n\u003e .\n\u003e\n"}],"action":{"name":"View Issue","url":"#99 (comment)"}}}

[xylcbd]

cmake -DPYTHON_EXECUTABLE=which python ..

[benDavidson6]

I had to provide the following flags

-DPYTHON_EXECUTABLE=C:\Users\Ben\Anaconda3\envs\py3.7\python.exe 
-DPYTHON_INCLUDE_DIR=C:\Users\Ben\Anaconda3\envs\py3.7\include 
-DPYTHON_LIBRARY=C:\Users\Ben\Anaconda3\envs\py3.7\libs

[xmba15]

I have written a script to find cmake variables for pyenv python.
https://github.com/xmba15/oh_my_cmakes/blob/master/cmake/FindPythonPyenv.cmake

So basically, add these lines before add_subdirectory(pybind11)

execute_process(
  COMMAND which python
  OUTPUT_VARIABLE PYTHON_EXECUTABLE OUTPUT_STRIP_TRAILING_WHITESPACE
)

execute_process(
  COMMAND python -c "from __future__ import print_function; from distutils.sysconfig import get_python_inc; print(get_python_inc())"
  OUTPUT_VARIABLE PYTHON_INCLUDE_DIR OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET
)

execute_process(
  COMMAND python -c "from __future__ import print_function; import os, numpy.distutils; print(os.pathsep.join(numpy.distutils.misc_util.get_numpy_include_dirs()))"
  OUTPUT_VARIABLE PYTHON_NUMPY_INCLUDE_DIR OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET
)

execute_process(
  COMMAND python -c "from __future__ import print_function; import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR') + '/' + sysconfig.get_config_var('LDLIBRARY'))"
  OUTPUT_VARIABLE PYTHON_LIBRARY OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET
)

execute_process(
  COMMAND python -c "from __future__ import print_function; import platform; print(platform.python_version())"
  OUTPUT_VARIABLE PYTHONLIBS_VERSION_STRING OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET
)

then ${PYTHON_EXECUTABLE}, ${PYTHON_INCLUDE_DIR}, ${PYTHON_NUMPY_INCLUDE_DIR}, ${PYTHON_LIBRARY} of pyenv will be fed into pybind11.

[misterjoa]

In case that might help, I found a workaround for a similar problem, looking at the cmake doc :
https://cmake.org/cmake/help/v3.0/module/FindPythonLibs.html

You must set two env vars for cmake to find coherent versions
Unfortunately this is not a generic solution but still, it works !

cmake -DPYTHON_LIBRARY=${HOME}/.pyenv/versions/3.8.0/lib/libpython3.8.a -DPYTHON_INCLUDE_DIR=${HOME}/.pyenv/versions/3.8.0/include/python3.8/ cern_root/

[amritas02]

I had the same problem, and once I managed to solve this problem - which was adding a couple lines setting the variables python_library and python_include in the Cmakelists.txt , I ran into Boost problems.

The way I solved this was uninstalling python using homebrew and reinstalling python and linking it properly. After that pip3 install . worked in one go.

[WilliamTambellini]

FYI: still an issue with latest cmake, cf:
https://cmake.org/cmake/help/v3.24/module/FindPython3.html#module:FindPython3