How to include package-data files in a release using meson-python?

[adam2392]

Hi,

I was wondering if there was a way to include package-data in the tar balls when making a release using meson/meson-python?

Traditionally, one might have package_data defined in the setup.py file, but there does not seem to be an equivalent in meson?

[rgommers]

Hey @adam2392, there shouldn't be a need for a specific Meson feature, they're just files to be installed. Passing the list of files to py.install_sources should do the right thing. If not, can you point to the code you're trying to translate?

[adam2392]

I was trying to use spin and adding the following command: https://github.com/scientific-python/spin/blob/main/spin/cmds/build.py. This is because when using python -m build I was getting into errors because as part of my build process, I need to make sure a submodule is pulled in.

I.e. I run git submodule update --init within my meson.build file

r = run_command('git', 'submodule', 'update', '--init', check: true)

However, this way results in an error because

../../meson.build:56:2: ERROR: Problem encountered: MESON_BUILD_FAILED: Could not update submodules: fatal: not a git repository (or any of the parent directories): .git

[Update] Idk why but I guess the git files are not copied over, so technically git submodule update --init will fail inside the build environment, but succeed locally. In addition, since I do some renaming of submodule directories, I need to add that into the meson.build file:

r = run_command('mv', 'sktree/_lib/sklearn_fork/sklearn', 'sktree/_lib/sklearn', check: false)

[rgommers]

I recommend updating the submodules before calling python -m build, trying to invoke git during a build step is definitely not a good idea, and not necessary. SciPy for example has git submodules too; we've never had an issue with this. It's similar to cloning a repo; the repo itself needs to be in a ready and consistent state before building anything.

In addition, since I do some renaming of submodule directories, I need to add that into the meson.build file:

You don't want to do this either. Anything you can install you can also give a custom location directly (e.g. the subdir keyword to py.install_sources).

[adam2392]

Thanks for pointing out that I should use python -m build instead of ./spin sdist. So I am able to go the route of the python -m build, which underneath the hood leverages meson and this generates a valid tar file.

However, when I look at the https://test.pypi.org/project/scikit-tree/0.1.2/ pypi page, none of the pyproject.toml and README.md files are rendered there. My suspicion is that the packaged tarball places the submodules in different directories. Is this something we can control at the meson level, or is this an artifact of build?

Perhaps this is out of the scope of meson discussion now (lmk if it is), but is there some trick to controlling the meson layering of relevant package files inside the final tarball so it can get rendered on pypi correctly?

[rgommers]

Looking at https://github.com/neurodata/scikit-tree/blob/91392d6da7b849827a015501ef07ad7950448037/pyproject.toml, you are not using standard PEP 621 metadata but Poetry-specific tables. That isn't tested by us and I am unsure if it will work with meson-python. Poetry does not yet support PEP 621, so there's a conflict there - I think it's one or the other right now.

[adam2392]

Ah right you are... Maybe it's time for me to just abandon poetry.

The rendering works now as expected https://test.pypi.org/project/sktree/0.1.3/!

[rgommers]

Great 🎉