Installing error messages in MacOS with conda/pip

[LDAmorim]

I am trying to follow up on the answer to issue #591 and install OpenPMD-api with MPI support through conda install -c conda-forge openpmd-api=*=mpi_mpich*.
But found an error that I don't know how to fix. I then tried the pip installation commands, but still encountered errors.
I will try spack next, but decided to leave this issue here in case I am doing something wrong that you can spot and correct me on.
(Sorry if this is related to issues already here or on conda's GitHub repo, but I couldn't find something that seemed similar to me.)

Initially I tried the command and when I found the error reported below, I decided to try its dependencies so I did:

$ conda clean --all
$ conda update --all
$ conda install -c conda-forge hdf5
$ conda install -c conda-forge adios
$ conda install -c conda-forge adios2
conda install -c conda-forge mpich 
$ conda install -c conda-forge openpmd-api=*=mpi_mpich*

From the latter command, I got the following (and the same as the one I saw initially) error messages:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - openpmd-api[build=mpi_mpich*]

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.anaconda.com/pkgs/main/osx-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/osx-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

So then I tried to use openPMD_USE_MPI=ON python3 -m pip install openpmd-api --no-binary :all: instead. This led to the error messages below:

$ openPMD_USE_MPI=ON pip install openpmd-api --no-binary :all:
Collecting openpmd-api
  Using cached https://files.pythonhosted.org/packages/10/e7/575daf3fe06d430713e5bf32ea68077c693a1aa82138209a85f8cbb715aa/openPMD-api-0.9.0a1.tar.gz
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: //anaconda3/bin/python //anaconda3/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-wqd9gkc6/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>38.6' wheel 'cmake>=3.11.0,<4.0.0' 'pybind11>=2.3.0,<3.0.0'
       cwd: None
  Complete output (17 lines):
  Collecting setuptools>38.6
    Using cached https://files.pythonhosted.org/packages/11/0a/7f13ef5cd932a107cd4c0f3ebc9d831d9b78e1a0e8c98a098ca17b1d7d97/setuptools-41.6.0.zip
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/59/b0/11710a598e1e148fb7cbf9220fd2a0b82c98e94efbdecb299cb25e7f0b39/wheel-0.33.6.tar.gz
  Collecting cmake<4.0.0,>=3.11.0
    Using cached https://files.pythonhosted.org/packages/0a/f5/3212616a15b4112d7ad075a407f007eac2cac59292a9d973f2ee7e3c4068/cmake-3.13.2.post1.tar.gz
      ERROR: Command errored out with exit status 1:
       command: //anaconda3/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-ov1rjr2x/cmake/setup.py'"'"'; __file__='"'"'/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-ov1rjr2x/cmake/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-ov1rjr2x/cmake/pip-egg-info
           cwd: /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-ov1rjr2x/cmake/
      Complete output (5 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-ov1rjr2x/cmake/setup.py", line 7, in <module>
          from skbuild import setup
      ModuleNotFoundError: No module named 'skbuild'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: //anaconda3/bin/python //anaconda3/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-wqd9gkc6/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>38.6' wheel 'cmake>=3.11.0,<4.0.0' 'pybind11>=2.3.0,<3.0.0' Check the logs for full command output.

I thought that that error was due to missing skbuildso I did conda install -c conda-forge scikit-build, but after came an error that seemed related to make (which I had installed from cmake). I uninstalled cmake and ran conda install -c anaconda cmake.

When I tried again to install OpenPMD, I got the error messages below:

$ openPMD_USE_MPI=ON python3 -m pip install openpmd-api --no-binary :all:
Collecting openpmd-api
  Using cached https://files.pythonhosted.org/packages/10/e7/575daf3fe06d430713e5bf32ea68077c693a1aa82138209a85f8cbb715aa/openPMD-api-0.9.0a1.tar.gz
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: //anaconda3/bin/python3 //anaconda3/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-z0hihqh2/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>38.6' wheel 'cmake>=3.11.0,<4.0.0' 'pybind11>=2.3.0,<3.0.0'
       cwd: None
  Complete output (169 lines):
  Collecting setuptools>38.6
    Using cached https://files.pythonhosted.org/packages/11/0a/7f13ef5cd932a107cd4c0f3ebc9d831d9b78e1a0e8c98a098ca17b1d7d97/setuptools-41.6.0.zip
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/59/b0/11710a598e1e148fb7cbf9220fd2a0b82c98e94efbdecb299cb25e7f0b39/wheel-0.33.6.tar.gz
  Collecting cmake<4.0.0,>=3.11.0
    Using cached https://files.pythonhosted.org/packages/0a/f5/3212616a15b4112d7ad075a407f007eac2cac59292a9d973f2ee7e3c4068/cmake-3.13.2.post1.tar.gz
  Collecting pybind11<3.0.0,>=2.3.0
    Using cached https://files.pythonhosted.org/packages/aa/91/deb6743e79e22ab01502296570b39b8404f10cc507a6692d612a7fee8d51/pybind11-2.4.3.tar.gz
  Skipping wheel build for setuptools, due to binaries being disabled for it.
  Skipping wheel build for wheel, due to binaries being disabled for it.
  Skipping wheel build for cmake, due to binaries being disabled for it.
  Skipping wheel build for pybind11, due to binaries being disabled for it.
  Installing collected packages: setuptools, wheel, cmake, pybind11
      Running setup.py install for setuptools: started
      Running setup.py install for setuptools: finished with status 'done'
      Running setup.py install for wheel: started
      Running setup.py install for wheel: finished with status 'done'
      Running setup.py install for cmake: started
      Running setup.py install for cmake: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: //anaconda3/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/setup.py'"'"'; __file__='"'"'/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-record-j6fnfbpr/install-record.txt --single-version-externally-managed --prefix /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-z0hihqh2/overlay --compile
           cwd: /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/
      Complete output (144 lines):
  
  
      --------------------------------------------------------------------------------
      -- Trying "Ninja" generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      Not searching for unused variables given on the command line.
      -- The C compiler identification is AppleClang 10.0.0.10001044
      -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
      -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- The CXX compiler identification is AppleClang 10.0.0.10001044
      -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
      -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- broken
      CMake Error at /anaconda3/share/cmake-3.15/Modules/CMakeTestCXXCompiler.cmake:53 (message):
        The C++ compiler
  
          "/Library/Developer/CommandLineTools/usr/bin/c++"
  
        is not able to compile a simple test program.
  
        It fails with the following output:
  
          Change Dir: /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/_cmake_test_compile/build/CMakeFiles/CMakeTmp
  
          Run Build Command(s)://anaconda3/bin/ninja cmTC_bf0ee && [1/2] Building CXX object CMakeFiles/cmTC_bf0ee.dir/testCXXCompiler.cxx.o
          warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
          1 warning generated.
          [2/2] Linking CXX executable cmTC_bf0ee
          FAILED: cmTC_bf0ee
          : && /Library/Developer/CommandLineTools/usr/bin/c++  -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.6 -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/cmTC_bf0ee.dir/testCXXCompiler.cxx.o  -o cmTC_bf0ee   && :
          clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
          ld: library not found for -lstdc++
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          ninja: build stopped: subcommand failed.
  
        CMake will not be able to correctly generate this project.
      Call Stack (most recent call first):
        CMakeLists.txt:4 (ENABLE_LANGUAGE)
  
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/_cmake_test_compile/build/CMakeFiles/CMakeError.log".
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying "Ninja" generator - failure
      --------------------------------------------------------------------------------
  
  
 
      --------------------------------------------------------------------------------
      -- Trying "Unix Makefiles" generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      Not searching for unused variables given on the command line.
      -- The C compiler identification is AppleClang 10.0.0.10001044
      -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
      -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- The CXX compiler identification is AppleClang 10.0.0.10001044
      -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
      -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- broken
      CMake Error at /anaconda3/share/cmake-3.15/Modules/CMakeTestCXXCompiler.cmake:53 (message):
        The C++ compiler
  
          "/Library/Developer/CommandLineTools/usr/bin/c++"
  
        is not able to compile a simple test program.
  
        It fails with the following output:
  
          Change Dir: /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/_cmake_test_compile/build/CMakeFiles/CMakeTmp
  
          Run Build Command(s):/usr/bin/make cmTC_09a3b/fast && /Library/Developer/CommandLineTools/usr/bin/make -f CMakeFiles/cmTC_09a3b.dir/build.make CMakeFiles/cmTC_09a3b.dir/build
          Building CXX object CMakeFiles/cmTC_09a3b.dir/testCXXCompiler.cxx.o
          /Library/Developer/CommandLineTools/usr/bin/c++    -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.6   -o CMakeFiles/cmTC_09a3b.dir/testCXXCompiler.cxx.o -c /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/_cmake_test_compile/build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
          warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
          1 warning generated.
          Linking CXX executable cmTC_09a3b
          /anaconda3/bin/cmake -E cmake_link_script CMakeFiles/cmTC_09a3b.dir/link.txt --verbose=1
          /Library/Developer/CommandLineTools/usr/bin/c++    -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.6 -Wl,-search_paths_first -Wl,-headerpad_max_install_names   CMakeFiles/cmTC_09a3b.dir/testCXXCompiler.cxx.o  -o cmTC_09a3b
          clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
          ld: library not found for -lstdc++
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          make[1]: *** [cmTC_09a3b] Error 1
          make: *** [cmTC_09a3b/fast] Error 2
  
        CMake will not be able to correctly generate this project.
      Call Stack (most recent call first):
        CMakeLists.txt:4 (ENABLE_LANGUAGE)
  
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/_cmake_test_compile/build/CMakeFiles/CMakeError.log".
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying "Unix Makefiles" generator - failure
      --------------------------------------------------------------------------------
  
      ********************************************************************************
      scikit-build could not get a working generator for your system. Aborting build.
  
      Building MacOSX wheels for Python 3.7 requires XCode.
      Get it here:
  
        https://developer.apple.com/xcode/
  
      ********************************************************************************
      ----------------------------------------
  ERROR: Command errored out with exit status 1: //anaconda3/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/setup.py'"'"'; __file__='"'"'/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-4slomz8_/cmake/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-record-j6fnfbpr/install-record.txt --single-version-externally-managed --prefix /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-z0hihqh2/overlay --compile Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: //anaconda3/bin/python3 //anaconda3/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-z0hihqh2/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>38.6' wheel 'cmake>=3.11.0,<4.0.0' 'pybind11>=2.3.0,<3.0.0' Check the logs for full command output.

Just for curiosity I tried to run the command without --no-binary :all:, i.e. as:

openPMD_USE_MPI=ON python3 -m pip install openpmd-api

And the installation finished with the Successfully built openpmd-api message. But I was not able to load the library:

$ python3
Python 3.7.5 (default, Oct 25 2019, 10:52:18) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import openpmd_api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/anaconda3/lib/python3.7/site-packages/openpmd_api.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libopenPMD.dylib
  Referenced from: /anaconda3/lib/python3.7/site-packages/openpmd_api.cpython-37m-darwin.so
  Reason: image not found

The location of that library is /anaconda3/lib/python3.7/site-packages/libopenPMD.dylib, which is in the PYTHONPATH that points to /anaconda3/lib/python3.7/site-packages as I think can be seen in:

$ env | grep PATH
PATH=//anaconda3/bin://anaconda3/condabin:/Users/diana/Documents/opt/spack/bin:/Users/diana/Documents/opt/spack/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/TeX/texbin
MODULEPATH=/Users/diana/Documents/opt/spack/share/spack/modules/darwin-highsierra-x86_64
PYTHONPATH=/Users/diana/Documents/opt/WarpX/Tools:/anaconda3/lib/python3.7/site-packages:

Do you know what I am doing wrong?
Or do you have any suggestions on how to fix this installation?
Do you think I need to update my software and re-try this?
Or that I should use gcc instead of clang?

Thanks and sorry for the trouble!

Environment used:

• MacOS High Sierra
• conda 4.7.12
• clang 10.0.0
• python 3.7.3
• jupyter-notebook 6.0.1
• numpy 1.17.3
• mpi4py 3.0.3

[LDAmorim]

I tried to update pip, but it seemed to already be up to date:

$ python3 -m pip install -U pip
Requirement already up-to-date: pip in /anaconda3/lib/python3.7/site-packages (19.3.1)

[ax3l]

Dear Dr. Amorim,

please excuse the inconveniences that this installation might have caused, let me explain what happened.

I forgot that conda-forge does not yet ship MPI-enabled binaries for openPMD-api with OSX, due to some environment issues: conda-forge/openpmd-api-feedstock#28 Sorry for that. I hope the linked PR will get resolved some time soon, but we have to figure out why the tests fail.

Just as a note: macOS 10.13 (High Sierra) is still supported by macOS, contrary to what I babbled at lunch.

In the meantime, and as you already did, going to install openPMD-api via pip inside a conda environment is a very suitable way to proceed. First install conda install -c conda-forge adios=*=mpi* adios2=*=mpi* hdf5=*=mpi* cmake numpy mpich mpi4py as you did - so all dependencies are definitely in place and from the same source.

The pip installation with --no-binary :all: was indeed too broad and I will update the documentation on this. openPMD_USE_MPI=ON pip install openpmd-api --no-binary openpmd-api would be better suited: #594
As you already spotted: removing :all: works since this will avoid that all dependencies are also build from source - which is where an error occurs right now for you. Usually, the removed flag would download a pre-compiled, non-MPI binary but since I do not ship binaries via PyPI ("pip") besides for Linux right now, this just builds openPMD-api from source in both cases.

Thanks for updating python3 -m pip install -U pip, we do indeed need pip 19 or newer, will also document that: #594

Finally, the dlopen issue seams to be a problem with the pip install that I would like to investigate, ideally directly on your computer if that is possible. For now, a work-around should be to set on the terminal before launching python/ipython:

export DYLD_LIBRARY_PATH=/anaconda3/lib/python3.7/site-packages/:${DYLD_LIBRARY_PATH}

[LDAmorim]

Dear Dr Hübl,

Thank you very much for your super quick reply! I understood the conda problem now.

Following your detailed instructions, I tried to re-do the installation of OpenPMD-api first running conda install -c conda-forge adios=*=mpi_mpich* adios2=*=mpi_mpich* hdf5=*=mpi_mpich* cmake numpy mpich mpi4py.

The long error message I get from openPMD_USE_MPI=ON python3 -m pip install openpmd-api --no-binary openpmd-api ends with:

[ 62%] Linking CXX shared library ../lib.macosx-10.7-x86_64-3.7/libopenPMD.dylib
  ld: file not found: @rpath/libblosc.1.dylib for architecture x86_64
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  make[2]: *** [../lib.macosx-10.7-x86_64-3.7/libopenPMD.dylib] Error 1
  make[1]: *** [CMakeFiles/openPMD.dir/all] Error 2
  make: *** [all] Error 2
  Traceback (most recent call last):
    File "/anaconda3/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/anaconda3/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/anaconda3/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 182, in build_wheel
      metadata_directory)
    File "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-34btri2a/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 209, in build_wheel
      wheel_directory, config_settings)
    File "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-34btri2a/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 194, in _build_with_temp_dir
      self.run_setup()
    File "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-34btri2a/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 237, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-34btri2a/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 177, in <module>
      ('License :: OSI Approved :: '
    File "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-34btri2a/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "//anaconda3/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "//anaconda3/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "//anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-build-env-34btri2a/overlay/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "//anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "//anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "//anaconda3/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "//anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "//anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "setup.py", line 36, in run
      self.build_extension(ext)
    File "setup.py", line 97, in build_extension
      cwd=self.build_temp
    File "//anaconda3/lib/python3.7/subprocess.py", line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2.
  Building wheel for openpmd-api (PEP 517) ... error
  ERROR: Failed building wheel for openpmd-api
  Running setup.py clean for openpmd-api
  Running command //anaconda3/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-og_bxsvz/openpmd-api/setup.py'"'"'; __file__='"'"'/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-og_bxsvz/openpmd-api/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
  running clean
  removing 'build/temp.macosx-10.7-x86_64-3.7' (and everything under it)
  removing 'build/lib.macosx-10.7-x86_64-3.7' (and everything under it)
  'build/bdist.macosx-10.7-x86_64' does not exist -- can't clean it
  'build/scripts-3.7' does not exist -- can't clean it
  removing 'build'
Failed to build openpmd-api
Cleaning up...
  Removing source in /private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-install-og_bxsvz/openpmd-api
Removed build tracker '/private/var/folders/5_/fw95mbwj7bl5mym1th0zmfw40000gr/T/pip-req-tracker-yrxbwcnh'
ERROR: Could not build wheels for openpmd-api which use PEP 517 and cannot be installed directly
Exception information:
Traceback (most recent call last):
  File "/anaconda3/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/anaconda3/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 426, in run
    ", ".join(r.name for r in build_failures)))
pip._internal.exceptions.InstallationError: Could not build wheels for openpmd-api which use PEP 517 and cannot be installed directly

The very end of it, pip._internal.exceptions.InstallationError: Could not build wheels for openpmd-api which use PEP 517 and cannot be installed directly, seems to me to echo the error from using --no-binary openpmd-api described in PEP 517.

The initial error message ld: file not found: @rpath/libblosc.1.dylib for architecture x86_64 seems to be related to MPI not being compatible with our installation and in some internet chats the suggestion to address this is to build from source.

Thanks,
Diana

[LDAmorim]

The problem does not appear on the latest development dev branch (3467ac6):

openPMD_USE_MPI=ON python3 -m pip install -v git+git://github.com/openPMD/openPMD-api.git

We still have to set

export DYLD_FALLBACK_LIBRARY_PATH=/anaconda3/lib/python3.7/site-packages:$DYLD_FALLBACK_LIBRARY_PATH

(or to the respective path on your machine) on OSX since relative RPATHS via $ORIGIN that we usually use on Linux do not work on this version of OSX:

$ python -c "import openpmd_api as api; print(api.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/anaconda3/lib/python3.7/site-packages/openpmd_api.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libopenPMD.dylib
  Referenced from: /anaconda3/lib/python3.7/site-packages/openpmd_api.cpython-37m-darwin.so
  Reason: image not found

[ax3l]

The export DYLD_FALLBACK_LIBRARY_PATH extension to work-around the dlopen issue has now been addressed by #595 :)

[ax3l]

Furthermore, we now have MPI-enabled OSX packages for conda-forge! :)
conda-forge/openpmd-api-feedstock#28

Only limitation: have to figure out a problem with the combination OSX+MPICH+ADIOS1, which is why I had to disable the ADIOS1 backend on OSX for now if we build with MPICH (but OpenMPI works already; also parallel HDF5 works with both MPICH as well as OpenMPI).

[ax3l]

I think we pushed through and have all issues addressed! :)
10 hours from report to triple-fix, amazing teamwork Dr. Amorim! 🚀 ✨