Building external fortran dependencies using f2py

[KyleAcheson]

I'm having some trouble compiling fortran libraries with f2py. Using the simple example given by the numpy (https://numpy.org/doc/stable/f2py/buildtools/meson.html) meson.build file,

project('f2py_examples', 'c',                                                                                                                                                    
  version : '0.1',
  license: 'BSD-3',
  meson_version: '>=0.64.0',
  default_options : ['warning_level=2'],
)

add_languages('fortran')

py_mod = import('python')
py = py_mod.find_installation(pure: false)
py_dep = py.dependency()

incdir_numpy = run_command(py,
  ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
  check : true
).stdout().strip()

incdir_f2py = run_command(py,
    ['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
    check : true
).stdout().strip()

fibby_source = custom_target('fibbymodule.c',
  input : ['fib1.f'],  # .f so no F90 wrappers
  output : ['fibbymodule.c', 'fibby-f2pywrappers.f'],
  command : [py, '-m', 'numpy.f2py', '@INPUT@', '-m', 'fibby', '--lower']
)

inc_np = include_directories(incdir_numpy, incdir_f2py)

py.extension_module('fibby',
  ['fib1.f', fibby_source],
  incdir_f2py / 'fortranobject.c',
  include_directories: inc_np,
  dependencies : py_dep,
  install : true
)

the code successfully builds,

rm -rf builddir
meson setup builddir
meson compile -C builddir
cd builddir
python -c "import numpy as np; import fibby; a = np.zeros(9); fibby.fib(a); print (a)"
# [ 0.  1.  1.  2.  3.  5.  8. 13. 21.]

However, building a python module that depends on the fortran module fibby using meson-python fails. Now I include the python module fib.py which contains,

import numpy as np                                                                                                                                                               
import fibby                                                                       
                                                                                   
def pyfib(n):                                                                   
    a = np.zeros(n)                                                             
    return fibby.fib(a)

with a pyproject.toml file,

[build-system]                                                                                                                                                                   
build-backend = 'mesonpy'
requires = ['meson-python']

[project]
name = "pyfibby"
version = "0.1.0"
description = """ meson f2py example build"""
requires-python = ">=3.10"
authors = [ 
    {name = "", email = ""},
]
classifiers = []

dependencies = [ 
    "numpy ~=1.26",
]

when running pip install . I get the following error,

Defaulting to user installation because normal site-packages is not writeable
Processing /home/kyle/meson_test
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [22 lines of output]
      + meson setup /home/kyle/meson_test /home/kyle/meson_test/.mesonpy-yn6wevis -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/kyle/meson_test/.mesonpy-yn6wevis/meson-python-native-file.ini
      The Meson build system
      Version: 1.4.0
      Source dir: /home/kyle/meson_test
      Build dir: /home/kyle/meson_test/.mesonpy-yn6wevis
      Build type: native build
      Project name: f2py_examples
      Project version: 0.1
      C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
      C linker for the host machine: cc ld.bfd 2.38
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      ../meson.build:8: WARNING: add_languages is missing native:, assuming languages are wanted for both host and build.
      Fortran compiler for the host machine: gfortran (gcc 11.4.0 "GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
      Fortran linker for the host machine: gfortran ld.bfd 2.38
      Program python found: YES (/opt/miniconda3/bin/python)
      Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
      Run-time dependency python found: YES 3.10
      
      ../meson.build:14:15: ERROR: Command `/opt/miniconda3/bin/python -c 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'` failed with status 1.
      
      A full log can be found at /home/kyle/meson_test/.mesonpy-yn6wevis/meson-logs/meson-log.txt
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

which results from the command /opt/miniconda3/bin/python -c 'import os; os.chdir(".."); import numpy; print(numpy.get_include())' in meson.build throwing an error that numpy does not exist despite it being installed and the fact that running this command in the terminal returns the path /home/kyle/.local/lib/python3.10/site-packages/numpy/core/include. I am new to the meson-python backed and honestly have no idea how to debug this, any help would be great. I would eventually like to build a more complicated project that has many fortran dependencies and that can just be installed with pip install .

[eli-schwartz]

Pip uses "build isolation" by default.

You need numpy as a build-system.requires, not just as an install-time dependency, because it needs to be available at build time to run f2py. You're currently missing that, though. The build isolation means even if you have a preinstalled copy of numpy, the build process doesn't see it -- it is built inside a fresh, empty virtualenv.

Note: this isn't really specific to a build backend.

[pmaxted]

pip install meson; pip install ninja was needed for to do the test build meson setup builddir; meson compile -C builddir if you are following the example step-by-step.

[dnicolodi]

Then I don't understand the question about having pip install these dependencies. How can pip install the dependencies if you don't invoke pip?

[pmaxted]

I am using this example as a starting point to try to get pip install ellc working again for the users of my ellc software that currently installs in python 3.11 but this is broken in 3.12

[dnicolodi]

In that case, the build dependencies are handled by pip.

[pmaxted]

Thanks.