Simple copying of windows DLLs in a wheel

[kpenev]

I am trying to create windows wheels for a python project that includes a shared library which in turn depends on a compiled boost library. The library is only used through cdll.LoadLibrary so there is no need to fix rpaths like for an extension module for example. I simply need the internal and boost shared libraries to both be copied into the wheel. However, I do not see an option to accomplish this. Instead meson-python crashes at the end with a message:

Bundling libraries in wheel is not supported on win32

I am guessing there is a way to do this since it is mentioned in the documentation for example that users can manually invoke delvewheel and in a few discussion threads I found, but I could not figure out how to do it. Any help will be much appreciated.

I should mention that I tried adding to pyproject.toml:

[tool.cibuildwheel.windows]
repair-wheel-command = ""

But that did not seem to disable the above error.

[rgommers]

This should be possible indeed. If all you're doing is building a shared library and want to install it inside your package, you can use the install: true and install_dir: keywords to shared_library like so: https://github.com/scipy/scipy/blob/f2d4775e7762fad984f8f0acd8227c725ff21630/scipy/special/meson.build#L28-L34

The Boost library is different, since that's external. delvewheel should take care of that vendoring. Again an example from SciPy, which uses delvewheel: https://github.com/scipy/scipy/blob/f2d4775e7762fad984f8f0acd8227c725ff21630/pyproject.toml#L150

But that did not seem to disable the above error.

The error is about the internal library, so unrelated to delvewheel.

If you run into more trouble, please feel free to point us at your source code.

[rgommers]

Also see gh-551 for an in-progress PR to improve the handling of internal shared libraries on Windows.

[kpenev]

Thank you for the prompt response.

I already have my shared library defined as:

shared_library(
    'superphot', 
    cpp_src, 
    dependencies: dependencies,
    install: true,
    install_dir: py.get_install_dir() / 'superphot'
)

And that indeed does what it is supposed to do (i.e. copy the library next to the python code that uses it. The problem arises when buld_wheel is invoked (in my case by cibuildwheel) which then promptly crashes. I am assuming there is a way to disable the call to mesonpy._rpath.fix_rpath on the internally built library. Or is the method perhaps to build the library as a separate pre-build step and then just list it as an input? The full traceback I get (if it helps) is:

    Traceback (most recent call last):
      File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-3ymijnkn\cp37-win_amd64\build\venv\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
        main()
      File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-3ymijnkn\cp37-win_amd64\build\venv\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-3ymijnkn\cp37-win_amd64\build\venv\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 152, in prepare_metadata_for_build_wheel
        whl_basename = backend.build_wheel(metadata_directory, config_settings)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 1020, in wrapper
        return func(*args, **kwargs)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 1074, in build_wheel
        return project.wheel(out).name
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 925, in wheel
        return builder.build(directory)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 481, in build
        self._install_path(whl, src, dst)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\__init__.py", line 440, in _install_path
        mesonpy._rpath.fix_rpath(origin, libspath)
      File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zfbi8jnf\overlay\Lib\site-packages\mesonpy\_rpath.py", line 64, in fix_rpath
        raise NotImplementedError(f'Bundling libraries in wheel is not supported on {sys.platform}')
    NotImplementedError: Bundling libraries in wheel is not supported on win32

If it helps, my full meson.build file is:

#Control file for meson/meson-python (compiling and PIP packaging)

project(
    'superphot', 
    'cpp',
    version: '0.0.0',
    license: '<++>',
    meson_version: '>= 1.4.0',
    default_options: [
        'optimization=3', 
        'cpp_std=c++14', 
        'prefer_static=true'
    ],
)

py = import('python').find_installation(pure: false)

compiler_id = meson.get_compiler('cpp').get_id()
toolchain = {
    'gcc': 'TOOLCHAIN_GCC', 
    'clang': 'TOOLCHAIN_CLANG',
    'msvc': 'TOOLCHAIN_MSVC'
}[compiler_id]

add_project_arguments('-D' + toolchain,
                      '-DTRACK_PROGRESS', 
                      language : 'cpp')
if get_option('buildtype') == 'debug'
    add_project_arguments('-DVERBOSE_DEBUG', '-DDEBUG', language: 'cpp')
elif get_option('buildtype') == 'release'
    add_project_arguments('-DNDEBUG', language: 'cpp')
endif

dependencies = [
    dependency('boost', modules: ['program_options']),
    dependency('eigen3')
]


if host_machine.system() == 'windows'
    add_project_arguments(
      '-DBOOST_RESULT_OF_USE_TR1', 
      '-D_USE_MATH_DEFINES',
      '-DNO_VISIBILITY',
      language: 'cpp'
    )
else
    add_project_arguments(
      '-fvisibility=hidden', 
      '-DNIX_VISIBILITY',
      language : 'cpp'
    )
endif

cpp_src = [
   ... long list of cpp files ...
]

shared_library(
    'superphot', 
    cpp_src, 
    dependencies: dependencies,
    install: true,
    install_dir: py.get_install_dir() / 'superphot'
)

subdir('PythonPackage/superphot')

[rgommers]

I am assuming there is a way to disable the call to mesonpy._rpath.fix_rpath on the internally built library.

Looking at the code, fix_rpath is only called if _WheelBuilder._has_internal_libs is True, which in turn is the case if {libdir} or libdir_shared} are encountered in build/meson-info/intro-install_plan.json. Could you check that file to see what target has one of those locations? Fixing that should resolve the problem.

For the SciPy example I linked, the metadata in intro-install_plan.json is:

"/path/to/build/scipy/special/libsf_error_state.so": {"destination": "{py_platlib}/scipy/special/libsf_error_state.so", "tag": "runtime", "subproject": null},

[kpenev]

That clarifies things a lot, though still not sure how to get it to work.

The intro-install_plan.json file is too big to usefully quote here directly (121K), so I am attaching it (intro-install_plan.json) if the excerpt below is not enough. But it seems the issue is due to the fact that building our library requires the header only project Eigen, which we pull during the build from wrapdb. The relevant entry in intro-install_plan.json is:

"data": {"D:\\a\\SuperPhot\\SuperPhot\\build\\meson-private\\eigen3.pc": {"destination": "{libdir}\\pkgconfig\\eigen3.pc", "tag": "devel", "subproject": "eigen"}}

Since Eigen is header only, the resulting library does not need any @rpath to be fixed, but I guess mesonpy doesn't know that.

[dnicolodi]

meson-python assumes that anything installed in {libdir} is a shared library. If you are using Eigen as a dependency, you need to make sure that it is not installed as part of the Python wheel: it does not make sense to distribute a Python wheel containing an headers only library. I think that most likely the meson.build for Eigen in wrapdb should not install the headers in the first place, but that is another issue. You can pass the --skip-subprojects=eigen option to meson install adding

[tool.meson-python.args]
install = ['--skip-subprojects=eigen']

to pyproject.toml.

[kpenev]

Thank you very much. That worked beautifully. It would make sense by default to not install libraries which have nothing that needs to be included in the package, but I suspect removing install by default would run afoul of some of the intended uses of wrapdb. In any case the solution is rather clean in the end.

[rgommers]

but I suspect removing install by default would run afoul of some of the intended uses of wrapdb

I think so too, so I don't expect anything to change here.

That worked beautifully.

Awesome, glad that it's resolved for you. Would you mind hitting Mark as Answer on whatever you think is the most appropriate comment?

[kpenev]

Sorry. I did that now. I saw the button to close the discussion, but did not see the mark as answer button.

[rgommers]

No worried at all, thanks @kpenev

[WillAyd]

I'm running into this same issue, but actually need to install the subproject library into my Python package. The subproject I am using is nanoarrow from the wrapdb, whose configuration has an install: true default.

I read through the discussion but I'm not sure what to do since I do need to bundle nanoarrow into my library (a static lib would also be an option):

The default meson configuration for a windows run on cibuildwheel yielded:

NotImplementedError: Bundling libraries in wheel is not supported on win32

Though macOS and ubuntu were fine.

I tried using static linkage in my meson configuration via:

nanoarrow_dep = dependency('nanoarrow', default_options: ['default_library=static'])

which in turn yielded this error from meson-python (all platforms failed):

meson-python: error: Could not map installation path to an equivalent wheel directory: '{libdir_static}\\libnanoarrow.a'

After seeing issue #525 I tried reverting back to shared linkage while pinning meson-python 0.13.2 but ran into the same issue as the OP:

NotImplementedError: Bundling libraries in wheel is not supported on win32

I tried adding a delvewheel call to the shared linkage + version pin:

[tool.cibuildwheel.windows]
before-build = "pip install delvewheel"
repair-wheel-command = "delvewheel repair --add-path {libdir} -w {dest_dir} {wheel}"

But during the repair phase I get:

FileNotFoundError: Unable to find library: libnanoarrow.dll

Is there something simple I am overlooking here? Unfortunately I don't have access to a PC so have been trying this all from GHA; I'm assuming the delvewheel approach is the closest?

If it helps this is a PR showing a lot of the things I've walked through

WillAyd/pandas-bitmask#1

[WillAyd]

FWIW I would probably prefer statically linking the libraries (if that can solve the windows issue), and I see that meson-python should forward skip-subprojects, but doing that from the CLI via pip install . --config-settings=install-args="--skip-subprojects" or setting this in pyproject.toml:

[tool.meson-python.args]
install = ['--skip-subprojects']

seems to have no effect

[WillAyd]

Ah ignore the above comment - I had to remove the meson-python 0.13.2 pin in pyproject.toml to get the skipping subprojects to work during install.

So I can get the static libs working on non-Windows builds, but get a runtime error with Windows with that same configuration:

  E   ImportError: DLL load failed while importing <the_library>: The specified module could not be found.