Installation via pip fails

[benediktdiemer]

I was trying to install the code to begin my review, but the pip installation fails on my laptop (2019 MacBook Pro). The full console output is below. The key error seems to be

clang: error: unsupported option '-fopenmp'

The installation section of the docs does not specify whether OpenMP is actually needed for cosmic-profiles? If so, some additional instructions would be welcome. Building with OpenMP is notoriously difficult on some compilers/systems, namely on Macs.

`Benedikts-MacBook:~ benedito$ sudo pip install cosmic-profiles
dbus[8926]: Dynamic session lookup supported but failed: launchd did not provide a socket path, verify that org.freedesktop.dbus-session.plist is loaded!
dbus[8926]: Dynamic session lookup supported but failed: launchd did not provide a socket path, verify that org.freedesktop.dbus-session.plist is loaded!
WARNING: The directory '/Users/benedito/Library/Caches/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Collecting cosmic-profiles
Downloading cosmic_profiles-1.2.7.tar.gz (1.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 4.0 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: matplotlib in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from cosmic-profiles) (3.3.3)
Collecting pathos
Downloading pathos-0.3.0-py3-none-any.whl (79 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.8/79.8 kB 113.5 MB/s eta 0:00:00
Requirement already satisfied: cython in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from cosmic-profiles) (0.29.24)
Collecting mpi4py
Downloading mpi4py-3.1.4.tar.gz (2.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 1.0 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: scipy in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from cosmic-profiles) (1.7.3)
Collecting cythongsl
Downloading CythonGSL-0.2.2.tar.gz (85 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.4/85.4 kB 109.2 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Requirement already satisfied: scikit-learn in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from cosmic-profiles) (1.0.1)
Requirement already satisfied: h5py in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from cosmic-profiles) (3.1.0)
Requirement already satisfied: numpy>=1.19.2 in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from cosmic-profiles) (1.21.5)
Collecting psutil
Downloading psutil-5.9.4-cp36-abi3-macosx_10_9_x86_64.whl (243 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 243.5/243.5 kB 24.0 MB/s eta 0:00:00
Requirement already satisfied: cached-property in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from h5py->cosmic-profiles) (1.5.2)
Requirement already satisfied: cycler>=0.10 in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from matplotlib->cosmic-profiles) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from matplotlib->cosmic-profiles) (1.3.1)
Requirement already satisfied: pillow>=6.2.0 in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from matplotlib->cosmic-profiles) (8.1.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from matplotlib->cosmic-profiles) (3.0.9)
Requirement already satisfied: python-dateutil>=2.1 in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from matplotlib->cosmic-profiles) (2.8.2)
Collecting ppft>=1.7.6.6
Downloading ppft-1.7.6.6-py3-none-any.whl (52 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.8/52.8 kB 94.2 MB/s eta 0:00:00
Collecting pox>=0.3.2
Downloading pox-0.3.2-py3-none-any.whl (29 kB)
Collecting multiprocess>=0.70.14
Downloading multiprocess-0.70.14-py37-none-any.whl (115 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.7/115.7 kB 110.2 MB/s eta 0:00:00
Collecting dill>=0.3.6
Downloading dill-0.3.6-py3-none-any.whl (110 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.5/110.5 kB 3.1 MB/s eta 0:00:00
Requirement already satisfied: joblib>=0.11 in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from scikit-learn->cosmic-profiles) (1.1.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from scikit-learn->cosmic-profiles) (3.0.0)
Requirement already satisfied: six in /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from cycler>=0.10->matplotlib->cosmic-profiles) (1.16.0)
Building wheels for collected packages: cosmic-profiles, cythongsl, mpi4py
Building wheel for cosmic-profiles (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for cosmic-profiles (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [102 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.15-x86_64-cpython-37
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles
copying cosmic_profiles/init.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles
creating build/lib.macosx-10.15-x86_64-cpython-37/example_scripts
copying example_scripts/gadget3_hdf5.py -> build/lib.macosx-10.15-x86_64-cpython-37/example_scripts
copying example_scripts/apply_ahf.py -> build/lib.macosx-10.15-x86_64-cpython-37/example_scripts
creating build/lib.macosx-10.15-x86_64-cpython-37/docs
creating build/lib.macosx-10.15-x86_64-cpython-37/docs/source
copying docs/source/conf.py -> build/lib.macosx-10.15-x86_64-cpython-37/docs/source
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/dens_profs
copying cosmic_profiles/dens_profs/dens_profs_tools.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/dens_profs
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/tests
copying cosmic_profiles/tests/test_shapes.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/tests
copying cosmic_profiles/tests/test_densities.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/tests
copying cosmic_profiles/tests/test_shapes_ex_script.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/tests
copying cosmic_profiles/tests/test_densities_ex_script.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/tests
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/mock_tools
copying cosmic_profiles/mock_tools/mock_halo_gen.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/mock_tools
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/shape_profs
copying cosmic_profiles/shape_profs/shape_profs_tools.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/shape_profs
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/gadget_hdf5
copying cosmic_profiles/gadget_hdf5/get_hdf5.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/gadget_hdf5
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/common
copying cosmic_profiles/common/config.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/common
copying cosmic_profiles/common/python_routines.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/common
copying cosmic_profiles/common/caching.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/common
copying cosmic_profiles/common/cosmo_tools.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/common
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs
copying cosmic_profiles/for_docs/setup_compile.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/dens_profs
copying cosmic_profiles/for_docs/dens_profs/dens_profs_tools.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/dens_profs
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/mock_tools
copying cosmic_profiles/for_docs/mock_tools/mock_uni.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/mock_tools
copying cosmic_profiles/for_docs/mock_tools/mock_halo_gen.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/mock_tools
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/shape_profs
copying cosmic_profiles/for_docs/shape_profs/shape_profs_tools.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/shape_profs
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/gadget_hdf5
copying cosmic_profiles/for_docs/gadget_hdf5/get_hdf5.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/gadget_hdf5
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/common
copying cosmic_profiles/for_docs/common/config.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/common
copying cosmic_profiles/for_docs/common/python_routines.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/common
copying cosmic_profiles/for_docs/common/init.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/common
copying cosmic_profiles/for_docs/common/caching.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/common
copying cosmic_profiles/for_docs/common/cosmo_tools.py -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/common
running egg_info
writing cosmic_profiles.egg-info/PKG-INFO
writing dependency_links to cosmic_profiles.egg-info/dependency_links.txt
writing requirements to cosmic_profiles.egg-info/requires.txt
writing top-level names to cosmic_profiles.egg-info/top_level.txt
reading manifest file 'cosmic_profiles.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE.md'
writing manifest file 'cosmic_profiles.egg-info/SOURCES.txt'
copying cosmic_profiles/init.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/cython_helpers
copying cosmic_profiles/cython_helpers/init.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/cython_helpers
copying cosmic_profiles/cython_helpers/helper_class.c -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/cython_helpers
copying cosmic_profiles/cython_helpers/helper_class.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/cython_helpers
copying cosmic_profiles/cython_helpers/helper_class.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/cython_helpers
copying cosmic_profiles/dens_profs/init.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/dens_profs
copying cosmic_profiles/dens_profs/dens_profs_algos.c -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/dens_profs
copying cosmic_profiles/dens_profs/dens_profs_algos.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/dens_profs
copying cosmic_profiles/dens_profs/dens_profs_classes.c -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/dens_profs
copying cosmic_profiles/dens_profs/dens_profs_classes.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/dens_profs
copying cosmic_profiles/dens_profs/dens_profs_classes.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/dens_profs
copying cosmic_profiles/shape_profs/init.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/shape_profs
copying cosmic_profiles/shape_profs/shape_profs_algos.c -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/shape_profs
copying cosmic_profiles/shape_profs/shape_profs_algos.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/shape_profs
copying cosmic_profiles/shape_profs/shape_profs_algos.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/shape_profs
copying cosmic_profiles/shape_profs/shape_profs_classes.c -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/shape_profs
copying cosmic_profiles/shape_profs/shape_profs_classes.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/shape_profs
copying cosmic_profiles/gadget_hdf5/init.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/gadget_hdf5
copying cosmic_profiles/gadget_hdf5/gen_catalogues.c -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/gadget_hdf5
copying cosmic_profiles/gadget_hdf5/gen_catalogues.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/gadget_hdf5
copying cosmic_profiles/common/init.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/common
copying cosmic_profiles/common/cosmic_base_class.c -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/common
copying cosmic_profiles/common/cosmic_base_class.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/common
copying cosmic_profiles/common/cosmic_base_class.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/common
creating build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/cython_helpers
copying cosmic_profiles/for_docs/cython_helpers/helper_class.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/cython_helpers
copying cosmic_profiles/for_docs/dens_profs/init.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/dens_profs
copying cosmic_profiles/for_docs/dens_profs/dens_profs_algos.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/dens_profs
copying cosmic_profiles/for_docs/dens_profs/dens_profs_classes.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/dens_profs
copying cosmic_profiles/for_docs/dens_profs/dens_profs_classes.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/dens_profs
copying cosmic_profiles/for_docs/shape_profs/shape_profs_algos.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/shape_profs
copying cosmic_profiles/for_docs/shape_profs/shape_profs_classes.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/shape_profs
copying cosmic_profiles/for_docs/gadget_hdf5/gen_catalogues.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/gadget_hdf5
copying cosmic_profiles/for_docs/common/init.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/common
copying cosmic_profiles/for_docs/common/cosmic_base_class.pxd -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/common
copying cosmic_profiles/for_docs/common/cosmic_base_class.pyx -> build/lib.macosx-10.15-x86_64-cpython-37/cosmic_profiles/for_docs/common
running build_ext
building 'cosmic_profiles.cython_helpers.helper_class' extension
creating build/temp.macosx-10.15-x86_64-cpython-37
creating build/temp.macosx-10.15-x86_64-cpython-37/cosmic_profiles
creating build/temp.macosx-10.15-x86_64-cpython-37/cosmic_profiles/cython_helpers
/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -I/opt/local/libexec/openssl11/include -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -I/private/tmp/pip-build-env-cawu8_a5/overlay/lib/python3.7/site-packages/numpy/core/include -I. -I/opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c cosmic_profiles/cython_helpers/helper_class.c -o build/temp.macosx-10.15-x86_64-cpython-37/cosmic_profiles/cython_helpers/helper_class.o -fopenmp
clang: error: unsupported option '-fopenmp'
error: command '/usr/bin/clang' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for cosmic-profiles
Building wheel for cythongsl (setup.py) ... done
Created wheel for cythongsl: filename=CythonGSL-0.2.2-py3-none-any.whl size=44179 sha256=bc4647fdd37e895904250af045df418261fa897994d9aefec7a80d4a7cd260ac
Stored in directory: /private/tmp/pip-ephem-wheel-cache-l3i_nr7l/wheels/b9/78/db/06ed1a1942089f436eec41cb761c99ba601a421cca9eb91915
Building wheel for mpi4py (pyproject.toml) ... done
Created wheel for mpi4py: filename=mpi4py-3.1.4-cp37-cp37m-macosx_10_15_x86_64.whl size=490145 sha256=0e7c526e60a2d9c4cb567ae2f0aa701cbf49f20a66271fc673b9dce7853e80d5
Stored in directory: /private/tmp/pip-ephem-wheel-cache-l3i_nr7l/wheels/87/81/66/c445deac61f7fb9b727c7b87ecc42dba6b948a03674b2fc56d
Successfully built cythongsl mpi4py
Failed to build cosmic-profiles
ERROR: Could not build wheels for cosmic-profiles, which is required to install pyproject.toml-based projects
Benedikts-MacBook:~ benedito$ `

[tibordome]

@benediktdiemer Apologies for the delay.

I haven't checked the installation myself on mac since the conda forge builds were successful and thus seemed ok. I think the conda install automatically takes care of the openmp dependency whereas the pip install might assume that openmp is available (via the -fopenmp flag of clang). Installing openmp on mac is possible (but difficult indeed) according to https://stackoverflow.com/questions/35134681/installing-openmp-on-mac-os-x-10-11. But (in the hope that this is not necessary) could you try the installation via conda?

[benediktdiemer]

@tibordome I've looked into this issue a bit more. I don't use conda, and tbh I don't wanna add a massive conda installation to my MacBook just for this. I checked that I can compile on my Mac with -fopenmp, and it works for gcc but not for clang; I believe both eventually call llvm, so the result should be the same. But where can I tell cosmic-profiles to compile using gcc rather than clang?

[tibordome]

@benediktdiemer I have managed (in a docker macOS container) to install cosmic-profiles in conda (quite hefty in terms of disk space I agree) and also pip. The steps were

• brew install llvm
• fixing PATH, DYLD_LIBRARY_PATH etc. as outlined here
• downgrade (on the macOS Catalina that I was using) command line tools to the one for XCode 11.5 (solving ld: unsupported tapi file type '!tapi-tbd' in YAML file issue)
• pip install cosmic-profiles succeeds with clang

I think in your case, one could either set a symlink from clang to gcc (which I assume you have installed manually since the default /usr/bin/gcc points to clang) as described here, or override the C/C++ compilers used during the build (same link, something like CC=/path/to/gcc CXX=/path/to/gcc /path/to/pip install cosmic-profiles).

[benediktdiemer]

I seems that the issue is that /usr/bin/clang on MacOS always refers to the system version of clang, and this version somehow does not seem to play with libomp. I just updated all my compilers (llvm, clang15, libomp15), and I can compile openmp code with clang15 now. However, /usr/bin/clang still doesn't refer to the new clang, and even setting $CC does not help.

How can I force the pip installation to use a different compiler?

[tibordome]

I would have expected setting $CC etc to work, since multiple sources claim that to be the preferred way to override the compilers.

I would check whether /path/to/clang15/bin is near the start of the $PATH environment variable, and set export PATH=/path/to/clang15/bin:$PATH if not. Compiler paths seem to be a common issue on macOS.

A brute-force workaround would be to move /usr/bin/clang somewhere safe, set a symbolic link to clang15 via ln -sf /path/to/clang15 /usr/bin/clang and revert back after successful installation. When setting the symbolic link, one might need to disable the SIP as described here.

[benediktdiemer]

I've changed my system such that clang points to a version that supports omp, as in, I can "clang -fopenmp test_omp.c -o test" compiles a test problem without errors.

Unfortunately, even when I have CC=clang, the cosmic-profiles installation still reverts specifically to /usr/bin/clang. This is the offending command:

  /usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -I/opt/local/libexec/openssl11/include -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -I/private/tmp/pip-build-env-bx3fcp1w/overlay/lib/python3.7/site-packages/numpy/core/include -I. -I/opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c cosmic_profiles/cython_helpers/helper_class.c -o build/temp.macosx-10.15-x86_64-cpython-37/cosmic_profiles/cython_helpers/helper_class.o -fopenmp

Can you please test this on your machines? Is there something in the make files that forces this behavior?

[tibordome]

I have tried again and can confirm that the CC=/path/to/clang/with/openmp/support flag works as expected. Specifically, the following throws an -fopenmp-related error,

/Users/tibordome/opt/anaconda3/envs/jan13_pip/bin/pip install --force-reinstall --no-cache-dir cosmic-profiles

where I have added the --no-cache-dir flag since otherwise the pre-built binary would be fetched from the cache which would pass trivially. However, setting the correct CC flag using

CC=/usr/local/opt/llvm/bin/clang /Users/tibordome/opt/anaconda3/envs/jan13_pip/bin/pip install --force-reinstall --no-cache-dir cosmic-profiles

leads to a successful installation, where /usr/local/opt/llvm/bin/clang is the version of clang that provides OpenMP support (i.e. /usr/local/opt/llvm/bin/clang -fopenmp -lstdc++ test.cpp passes). As a final test,

CC=/usr/bin/clang /Users/tibordome/opt/anaconda3/envs/jan13_pip/bin/pip install --force-reinstall --no-cache-dir cosmic-profiles

fails again, as expected.

[benediktdiemer]

Unfortunately, setting CC seems to work on some systems but not on others. Turns out the deeper reason is that the compiler is hard set in distutils.sysconfig:

https://shwina.github.io/custom-compiler-linker-extensions/

This can be verified in python:

import distutils.sysconfig
distutils.sysconfig.get_config_var("CC")
'/usr/bin/clang'

I will look into whether this can be changed.

[benediktdiemer]

One addition: apparently the way to alter the distutils config is to add a ~/.pydistutils.cfg file in the home directory. Indeed, this seems to be processed, but I cannot just put a path to a compiler there, as in,

[build]
compiler=/opt/local/bin/clang

does not work. I'm not a specialist for distutils and I can't find good documentation. Help would be appreciated.

[tibordome]

It's unfortunate that setting CC does not help on your machine. The ~/.pydistutils.cfg solution would be a cleaner one I agree, though according to this post, one should be able to override these configuration options by setting environment variables.

The best docs about distutils I have found are here and here. According to this post, setting the path as you showed might not work, but instead one could try

[build]
compiler=clang15

i.e. with the respective clang command. I am worried though that this only affects invocations of python3 setup.py build etc., which during the building of the wheel (in a pip installation) might not be invoked after all. (I can try this at the weekend but at the moment the docker macOS container has eaten up too much disk space on my Linux machine which I will have to reorganise.)

[benediktdiemer]

Indeed, on some people's system the env variables work and on some they don't; I am not sure why but it definitely doesn't work for me. I had found those same documentation pages, but they only say

--compiler (-c) specify the compiler type

I've tried compiler=clang, which points to the correct executable; but I get

  running build_ext
  error: don't know how to compile C/C++ code on platform 'posix' with 'clang' compiler
  [end of output]

and the same for gcc. So I really don't understand what values are allowed.

[tibordome]

I have checked and can confirm that I receive the same error messages when adding the compiler path or command name naively to the [build] section.

According to this post, the ~./pydistutils.cfg file follows the nomenclature of python setup.py option=value. I.e. having compiler=clang in the [build] section is equivalent to having python setup.py --compiler=clang during the building of the wheel when running pip install cosmic-profiles, which will fail.

This is also illustrated here, which again suggests to set the CC environment variable. The CC variable should get checked by distutils, and I can confirm again on a clean macOS Catalina machine that it works as expected. If setting the CC environment variable does not work on your machine, I would suggest (as the very same post does too) to set a symbolic link, which should definitely work as a last resort.

[benediktdiemer]

Thanks for these suggestions. I'm currently moving and will start teaching a new class next week, so not totally sure when I'll get to playing with this.

[tibordome]

No worries. Please take your time.

[adonath]

@tibordome To improve the pip install process you could maybe look into building and providing wheels for the package.

[tibordome]

@adonath A built wheel (.whl file) is provided for each version of the package on the pypi index, in addition to the source code (tar.gz file)

[nightuser]

@tibordome there is only one wheel for Python 3.9 on Linux https://pypi.org/project/cosmic-profiles/#files . It's often better to provide wheels for every combination of Python version and platform. For instance, this is how numpy does it: https://pypi.org/project/numpy/#files .

UPD: You may find https://github.com/pypa/cibuildwheel helpful.