Best practices in linking against a conda-packaged library

[akhmerov]

I'm trying to make a wrapper for the MUMPS solver, primarily targeting the version of MUMPS packaged via conda-forge. That version lacks pkgconfig, and therefore I need to rely on a different mechanism of library discovery. Right now I'm using ctypes.util.find_library for that over here. This works well with meson, spin, and pip, however fails with python -m build, likely due to some subtle details of how python path is set.

I would like to know:

• Is my approach reasonable?
• If not, what is the right way to link against the library?
• If yes, how to make build work?

[rgommers]

Hi @akhmerov, thanks for asking. It is a little odd that pip works but build doesn't - they really should act identically here (they're both build front-ends that call the same hook in meson-python, and defaults for build isolations etc. are the same between them too). Are you sure that pip install . works in the exact same environment where python -m build fails?

If it really looks like they're different, I can try to reproduce and see what's going on.

That version lacks pkgconfig, and therefore I need to rely on a different mechanism of library discovery.

One option here is of course to submit a PR to conda-forge to resolve conda-forge/mumps-feedstock#104. But if there are no .pc files generated by the MUMPS build itself, that only help for conda-forge of course. I'd only do that if there's a way to upstream the changes to MUMPS itself.

Right now I'm using ctypes.util.find_library for that over here.

You've already found Meson's compiler.find_library I see. I'm unclear on why you'd need ctypes.util.find_library, because the Meson find_library works in a similar way and is much cleaner. I suspect what is happening there is that you are not using the conda-forge compilers, and hence the lib directory of your conda env isn't on the search path. With conda-forge compilers it should work.

So your approach isn't completely unreasonable, but it'd be better to drop ctypes.util.find_library. It has more downsides, like making it very difficult for distro packagers of your package to deal with your package .

[akhmerov]

Are you sure that pip install . works in the exact same environment where python -m build fails?

Just tested on ubuntu 23.10 with

git clone https://gitlab.kwant-project.org/kwant/python-mumps.git && cd python-mumps
mamba env create -n mumps && mamba activate mumps
mamba install -y -c conda-forge python=3.11 pip build mumps-seq=5.2 metis=5.1.0 compilers
pip install .  # succeeds
python -m build -s  # fails

The reason for the difference, however, is likely indeed due to me using ctypes.util.find_library, so if I manage to avoid it, that'd probably also solve the problem. I see now that the problem isn't that the libraries weren't found, but rather the include directories weren't. Here I am, however, at a loss. cc.find_library('mumps_common_seq') works, but cc.find_library('mumps_common_seq', has_headers: 'mumps_c_types.h') doesn't, and meson doesn't discover the header files by itself. I realize that this is out of scope for meson-python, but I'd really appreciate pointers for how to determine the include dir from a found library.

I suspect what is happening there is that you are not using the conda-forge compilers, and hence the lib directory of your conda env isn't on the search path. With conda-forge compilers it should work.

I confirmed that I'm using a conda-forge compiler by the way, and it still doesn't find the headers.

[rgommers]

I just tried, and things build fine with your package as is for me, as well as with this diff (which is what I'd propose as a cleanup):

diff --git a/meson.build b/meson.build
index 1a29cb5..dac5f59 100644
--- a/meson.build
+++ b/meson.build
@@ -39,19 +39,11 @@ inc_np = include_directories(incdir_numpy)
 np_dep = declare_dependency(include_directories: inc_np)
 mumps_names = get_option('mumps_names')

-# Use python's ctypes.util.find_library to find the library
-# Works for conda and Debian/Ubuntu
-mumps_lib_path = fs.parent(run_command(py,
-    [
-    '-c',
-    'import ctypes.util; print(ctypes.util.find_library("@0@"))'.format(mumps_names[0])
-    ],
-    check: true
-).stdout().strip())
-mumps_include = include_directories(mumps_lib_path / '..' / 'include')
-mumps_libs = []
 foreach mumps_name : mumps_names
-    mumps_libs += cc.find_library(mumps_name, dirs: mumps_lib_path)
+    mumps_dep = cc.find_library(mumps_name)
+    if mumps_dep.found()
+      break
+    endif
 endforeach

-subdir('mumps')
\ No newline at end of file
+subdir('mumps')
diff --git a/mumps/meson.build b/mumps/meson.build
index 865cc7b..176738e 100644
--- a/mumps/meson.build
+++ b/mumps/meson.build
@@ -21,15 +21,11 @@ foreach py_file : py_files
     fs.copyfile(py_file)
 endforeach

-py.extension_module(
-    '_mumps',
+py.extension_module('_mumps',
     pyx_files,
-    dependencies: [np_dep, mumps_libs],
-    include_directories: [inc_np, mumps_include],
-    install_rpath: mumps_lib_path,
-    build_rpath: mumps_lib_path,
+    dependencies: [np_dep, mumps_dep],
     install: true,
     subdir: 'mumps'
 )

-subdir('tests')
\ No newline at end of file
+subdir('tests')

Adding has_headers: 'mumps_c_types.h' keeps things working too.

If dependency detection doesn't work for you with that has_headers added, you usually find the answer in build/meson-logs/meson-log.txt. Output for me is:

Running compile:
Working directory:  /Users/rgommers/code/tmp/python-mumps/build/meson-private/tmpcb0g8qk5
Code:

        #ifdef __has_include
         #if !__has_include("mumps_c_types.h")
          #error "Header 'mumps_c_types.h' could not be found"
         #endif
        #else
         #include <mumps_c_types.h>
        #endif
-----------
Command line: `arm64-apple-darwin20.0.0-clang /Users/rgommers/code/tmp/python-mumps/build/meson-private/tmpcb0g8qk5/testfile.c -E -P -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /Users/rgommers/mambaforge/envs/mumps-debug/include -D_FORTIFY_SOURCE=2 -isystem /Users/rgommers/mambaforge/envs/mumps-debug/include -P -O0 -Werror=implicit-function-declaration -std=c99` -> 0
Has header "mumps_c_types.h" : YES
Running compile:
Working directory:  /Users/rgommers/code/tmp/python-mumps/build/meson-private/tmpp59n009v
Code:
 int main(void) { return 0; }

The -isystem /Users/rgommers/mambaforge/envs/mumps-debug/include is the location where mumps_c_types.h is installed indeed (comes from the separate mumps-include package in conda-forge for me).

[akhmerov]

Thank you for the advice! Firstly, I figured out the most likely source of the problem: I was using fish shell, and a bunch of conda-related environment variables were not set. I confirm that in bash your patch (modified to include all mumps libraries) works with pip install . and python -m build, so this does seem like the way forward, thanks a lot!

The only problem that I still see is that pip install --editable . succeeds, but produces a broken install (both in bash and fish):

python -c "import mumps"                                                                                                                                         (py311) 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1138, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1078, in _find_spec
  File "/home/anton/micromamba/envs/py311/lib/python3.11/site-packages/_python_mumps_editable_loader.py", line 271, in find_spec
    tree = self.rebuild()
           ^^^^^^^^^^^^^^
  File "/home/anton/micromamba/envs/py311/lib/python3.11/site-packages/_python_mumps_editable_loader.py", line 312, in rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=stdout, check=True)
  File "/home/anton/micromamba/envs/py311/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/anton/micromamba/envs/py311/bin/ninja']' returned non-zero exit status 1.

[rgommers]

Great. That last issue is easy luckily - you're missing --no-build-isolation for the editable install, see https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html for why.