Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(template): Added pybind11 as an option for build system #163

Merged
merged 34 commits into from
Aug 9, 2023
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
f816654
add pybind11 build-system as option
ayeankit Aug 3, 2023
e9c6226
added setup.py
ayeankit Aug 4, 2023
ff88a54
added setup.py
ayeankit Aug 4, 2023
7e83d22
added setup.py
ayeankit Aug 5, 2023
2e84ac2
added setup.py
ayeankit Aug 5, 2023
1f04f8e
Update guide.md
ayeankit Aug 5, 2023
40433a4
Update setup.py
ayeankit Aug 7, 2023
7625f4e
Update CMakeLists.txt
ayeankit Aug 7, 2023
5e78f26
Update pybind11-pyproject.toml
ayeankit Aug 7, 2023
a61dfc6
Update base.sh
ayeankit Aug 7, 2023
4c4877c
Update base.sh
ayeankit Aug 7, 2023
5dfb8c5
Update setup.py
ayeankit Aug 7, 2023
0b252fc
Update base.sh
ayeankit Aug 7, 2023
c95ebad
Update setup.py
ayeankit Aug 7, 2023
c593005
Update CMakeLists.txt
ayeankit Aug 7, 2023
653e049
Update pybind11-pyproject.toml
ayeankit Aug 8, 2023
535c090
Update setup.py
ayeankit Aug 9, 2023
83dc6e0
Update setup.py
ayeankit Aug 9, 2023
5359d5c
Update Makefile
ayeankit Aug 9, 2023
c79bb3b
Create pybind11
ayeankit Aug 9, 2023
642b9ba
Merge branch 'main' into ayeankit/pybind11
ayeankit Aug 9, 2023
4ec2dfe
pre-commit clear
ayeankit Aug 9, 2023
bf7529d
Update Makefile
ayeankit Aug 9, 2023
bf5d323
Update src/scicookie/{{cookiecutter.project_slug}}/Makefile
ayeankit Aug 9, 2023
bcff667
Update pybind11-pyproject.toml
ayeankit Aug 9, 2023
11ccc28
Update guide.md
ayeankit Aug 9, 2023
4ce85f4
Update pybind11-pyproject.toml
ayeankit Aug 9, 2023
410d29d
Update Makefile
ayeankit Aug 9, 2023
a18f98d
Update README.md
ayeankit Aug 9, 2023
cd8ed24
Update contributing.md
ayeankit Aug 9, 2023
b885513
Update tests/smoke/base.sh
ayeankit Aug 9, 2023
b52b882
Update Makefile
ayeankit Aug 9, 2023
d7114a6
Update Makefile
ayeankit Aug 9, 2023
2e4cbf7
added build as dependency
ayeankit Aug 9, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,11 @@ for a Python package.
[Flit](https://flit.pypa.io),
[meson-python](https://meson-python.readthedocs.io/en/latest/index.html),
[Setuptools](https://setuptools.pypa.io/en/latest/),
[PDM](https://pdm.fming.dev/) or
[Hatch](https://hatch.pypa.io)
[Maturin](https://pypi.org/project/maturin/0.8.2/)
[scikit-build-core](https://scikit-build-core.readthedocs.io/en/latest/)
[PDM](https://pdm.fming.dev/),
[Hatch](https://hatch.pypa.io),
[Maturin](https://pypi.org/project/maturin/0.8.2/),
[scikit-build-core](https://scikit-build-core.readthedocs.io/en/latest/) or
[pybind11](https://pybind11.readthedocs.io/en/stable/)
based on your preference.
- The structure of the project can use the *src layout* or *flat
layout*. The “src layout” moving the code that is intended to be
Expand Down
15 changes: 14 additions & 1 deletion docs/guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,20 @@ packages. SciCookie support the following:
build requirements. With its capabilities, it facilitates cross-platform
builds using CMake and effortless integration with C/C++ libraries, making
it a valuable asset for research software engineers.


- [**Pybind11**](https://pybind11.readthedocs.io/en/stable/): It's build system designed
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we call it something like setuptools + pybind11 to be more clear, given that pybind11 itself is not a backend?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or atleast using "setuptools + pybind11" in all the user-facing locations, and just "pybind11" in the internals of scicookie.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or atleast using "setuptools + pybind11" in all the user-facing locations, and just "pybind11" in the internals of scicookie.

that makes sense to me.

Copy link
Contributor Author

@ayeankit ayeankit Aug 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or atleast using "setuptools + pybind11" in all the user-facing locations, and just "pybind11" in the internals of scicookie.

that makes sense to me.

Wouldn't that be confusing since we are using setuptools as default build-sytem?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really ... setuptools is the default option because it was the first one historically, but a lot of people want to kill that. That is why a bunch of project are looking for new alternatives.
but for c/c++ code setuptools is still a great option.
so specifying setuptools + pybind11 makes a lot of sense, because it is clear about its structure.

for C++ library that simplifies the creation of Python bindings for C++ code,
enabling easy integration of C++ functions and classes into Python scripts. It
acts as a bridge between the two languages, allowing C++ algorithms and functionality
to be directly called from Python as if they were native Python modules. Pybind11's
user-friendly syntax reduces boilerplate code, making binding generation more
straightforward, while standard C++ build systems like CMake or Make facilitate the
compilation of projects using pybind11. Its efficiency, ease of use, and strong community
support have made it a popular choice for projects requiring seamless interoperability
between C++ and Python, ranging from scientific computing to game development and
automation. Staying up-to-date with the latest pybind11 documentation ensures the
best practices are followed.

The idea behind the options in SciCookie is that you can choose from some of the
most popular system compilers to suit your needs and preferences for developing
Python packages. If you think we should add more options, you can submit your
Expand Down
3 changes: 2 additions & 1 deletion src/scicookie/cookiecutter.json
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@
"pdm",
"hatch",
"maturin",
"scikit-build-core"
"scikit-build-core",
"pybind11"
],
"use_bandit": "no",
"use_black": "yes",
Expand Down
23 changes: 23 additions & 0 deletions src/scicookie/hooks/post_gen_project.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@
BUILD_SYSTEM = "maturin"
{% elif cookiecutter.build_system == "scikit-build-core" -%}
BUILD_SYSTEM = "scikit-build-core"
{% elif cookiecutter.build_system == "pybind11" -%}
BUILD_SYSTEM = "pybind11"
{%- else %}
BUILD_SYSTEM = None
{%- endif %}
Expand Down Expand Up @@ -217,6 +219,19 @@ def clean_up_build_system():
build_system_dir / "skcdemo.cpp",
PROJECT_DIRECTORY / 'skcdemo.cpp'
)
elif BUILD_SYSTEM == "pybind11":
shutil.move(
build_system_dir / "pybind11-pyproject.toml",
PROJECT_DIRECTORY / 'pyproject.toml'
)
shutil.move(
build_system_dir / "CMakeLists.txt",
PROJECT_DIRECTORY / 'CMakeLists.txt'
)
shutil.move(
build_system_dir / "setup.py",
PROJECT_DIRECTORY / 'setup.py'
)
else:
shutil.move(
build_system_dir / "base-pyproject.toml",
Expand Down Expand Up @@ -291,6 +306,14 @@ def add_binding_source_files():
else:
os.makedir(src_system_dir)
shutil.move(build_system_dir / "lib.rs", src_system_dir)
elif BUILD_SYSTEM == "pybind11" :
build_system_dir = PROJECT_DIRECTORY / "build-system"
src_system_dir = PROJECT_DIRECTORY/ "src"
Comment on lines +310 to +311
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think PROJECT_DIRECTORY automatically checks for flat or src layout?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it for main.cpp which should be in src directory like this.
your_project/
|-- setup.py
|-- pyproject.toml
|-- src/
| |-- main.cpp
| |-- other_cpp_files.cpp
| |-- your_module_name_here.cpp
| |-- your_module_name_here.h
|-- your_package/
| |-- init.py
| |-- your_module_name_here.py
|-- tests/
| |-- test_your_module_name_here.py
|-- README.md
|-- LICENSE

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is an interesting discussion ... maybe we can return to this discussion to check how we should proceed not just for this build-system but maybe for all build-systems.
also maybe in our repo, maybe we could have this files in folders called by the programming language (cpp, rust, etc)

if USE_SRC_LAYOUT :
shutil.move(build_system_dir / "main.cpp", "src")
else:
os.makedir(src_system_dir)
shutil.move(build_system_dir / "main.cpp", src_system_dir)
else:
pass

Expand Down
1 change: 1 addition & 0 deletions src/scicookie/profiles/base.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ build_system:
- hatch
- maturin
- scikit-build-core
- pybind11
enabled: false

command_line_interface:
Expand Down
3 changes: 3 additions & 0 deletions src/scicookie/{{cookiecutter.project_slug}}/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,9 @@ build:
hatch build
{%- elif cookiecutter.build_system == "maturin" %}
maturin build
{%- elif cookiecutter.build_system == "pybind11" %}
python3 -m pip install build
ayeankit marked this conversation as resolved.
Show resolved Hide resolved

{%- endif %}

.PHONY:release-ci
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,20 @@ cmake_minimum_required(VERSION 3.15...3.26)

project(${SKBUILD_PROJECT_NAME} VERSION ${SKBUILD_PROJECT_VERSION} LANGUAGES CXX)

{% if cookiecutter.build_system == "scikit-build-core" -%}

find_package(pybind11 CONFIG REQUIRED)

pybind11_add_module(skcdemo MODULE skcdemo.cpp)

install(TARGETS skcdemo DESTINATION .)

{% elif cookiecutter.build_system == "pybind11" -%}

find_package(pybind11 CONFIG REQUIRED)

pybind11_add_module(_core MODULE src/{{cookiecutter.project_slug}}/main.cpp)

install(TARGETS _core DESTINATION src/{{cookiecutter.project_slug}}/main.cpp )

{%- endif %}
28 changes: 28 additions & 0 deletions src/scicookie/{{cookiecutter.project_slug}}/build-system/main.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#include <pybind11/pybind11.h>

int add(int i, int j) { return i + j; }

namespace py = pybind11;

PYBIND11_MODULE(_core, m) {
m.doc() = R"pbdoc(
Pybind11 example plugin
-----------------------
.. currentmodule:: python_example
.. autosummary::
:toctree: _generate
add
subtract
)pbdoc";

m.def("add", &add, R"pbdoc(
Add two numbers
Some other explanation about the add function.
)pbdoc");

m.def(
"subtract", [](int i, int j) { return i - j; }, R"pbdoc(
Subtract two numbers
Some other explanation about the subtract function.
)pbdoc");
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
[build-system]
requires = ["setuptools>=65", "wheel", "pybind11~=2.11.1"]
build-backend = "setuptools.build_meta"

[project]
name = "{{ cookiecutter.project_slug }}"
authors = [
{ name = "{{ cookiecutter.author_full_name }}", email = "{{ cookiecutter.author_email }}" },
]
description = "{{ cookiecutter.project_short_description }}"
readme = "README.md"
classifiers = [
"Development Status :: 1 - Planning",
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
{%- if cookiecutter.project_license == "MIT" %}
"License :: OSI Approved :: MIT License",
{%- elif cookiecutter.project_license == "BSD 3 Clause" %}
"License :: OSI Approved :: BSD License",
{%- elif cookiecutter.project_license == "Apache Software License 2.0" %}
"License :: OSI Approved :: Apache Software License",
{%- elif cookiecutter.project_license == "GNU General Public License v3" %}
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
{%- endif %}
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Topic :: Scientific/Engineering",
"Typing :: Typed",
]
dynamic = ["version"]
requires-python = ">=3.8.1"
dependencies = [
{# keep this line here #}
{%- if cookiecutter.use_pytest == "yes" -%}
"pytest >= 7.3.2,<8",
{% if cookiecutter.use_coverage == "yes" -%}
"pytest-cov >= 4.1.0,<5",
{% endif %}
{%- endif -%}{#- end of use_pytest -#}
{%- if cookiecutter.use_hypothesis == "yes" -%}
"hypothesis >= 6.0,<7",
{% endif %}
{%- if cookiecutter.use_coverage == "yes" -%}
"coverage >= 7.2.7,<8",
{% endif %}
{%- if cookiecutter.use_blue == "yes" -%}
"blue >= 0.9.1,<1",
{% endif %}
{%- if cookiecutter.use_black == "yes" -%}
"black >= 23.3.0,<24",
{% endif %}
{%- if cookiecutter.use_isort == "yes" -%}
"isort >= 5.12.0,<6",
{% endif %}
{%- if cookiecutter.use_pre_commit -%}
"pre-commit >= 3.3.2,<4",
{% endif %}
{%- if cookiecutter.use_flake8 == "yes" -%}
"flake8 >= 4.0.1, < 7",
{% endif %}
{%- if cookiecutter.use_ruff == "yes" -%}
"ruff >= 0.0.272,<1",
{% endif %}
{%- if cookiecutter.use_mypy == "yes" -%}
"mypy >= 1.3.0,<2",
{% endif %}
{%- if cookiecutter.use_bandit == "yes" -%}
"bandit >= 1.7.5,<2",
{% endif %}
{%- if cookiecutter.use_pydocstyle == "yes" -%}
"pydocstyle >= 6.3.0,<7",
{% endif %}
{%- if cookiecutter.use_vulture == "yes" -%}
"vulture >= 2.7,<3",
{% endif %}
{%- if cookiecutter.use_mccabe == "yes" -%}
"mccabe >= 0.6.1,<1",
{% endif %}
{%- if cookiecutter.use_containers in ['Docker', 'Podman'] -%}
# if you want to use docker-compose from your system, remove compose-go here
"compose-go >= 2.18.1,<3",
{% endif %}
"ipython < 8",
"ipykernel >=6.0.0",
{%- if cookiecutter.documentation_engine == 'mkdocs' -%}
"Jinja2 >=3.1.2,<4",
"mkdocs >=1.4.3,<2",
"mkdocs-exclude >= 1.0.2,<2",
"mkdocs-jupyter >= 0.24.1,<1",
"mkdocs-literate-nav >= 0.6.0,<1",
"mkdocs-macros-plugin >= 0.7.0, < 1",
"mkdocs-material >= 9.1.15,<10",
"mkdocstrings >= 0.21.2,< 1",
"mkdocstrings-python >= 1.1.2,<2",
{% elif cookiecutter.documentation_engine == 'sphinx' -%}
"Sphinx >= 6.2.1,<7",
"sphinx-rtd-theme >= 1.2.2,<2",
"importlib-metadata >= 6.5.1,<7",
"myst-parser >= 0.19.2,<1",
"nbsphinx >= 0.9.2,<1",
"pandoc >= 2.3,<3",
{% elif cookiecutter.documentation_engine == 'jupyter-book' -%}
"jupyter-book >= 0.15.1,<1",
"myst-parser >= 0.18.1,<1",
{% endif %}
]

[project.urls]
Homepage = "{{ cookiecutter.project_url }}"
"Bug Tracker" = "{{ cookiecutter.project_url }}/issues"
Discussions = "{{ cookiecutter.project_url }}/discussions"
Changelog = "{{ cookiecutter.project_url }}/releases"


{% include "build-system/base-pyproject.toml" %}
{#- keep this line at the end of the file -#}
21 changes: 21 additions & 0 deletions src/scicookie/{{cookiecutter.project_slug}}/build-system/setup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
from setuptools import setup # isort:skip

# Available at setup time due to pyproject.toml
from pybind11.setup_helpers import Pybind11Extension # isort:skip

# Note:
# Sort input source files if you glob sources to ensure bit-for-bit
# reproducible builds (https://github.com/pybind/python_example/pull/53)

ext_modules = [
Pybind11Extension(
"{{ cookiecutter.project_slug }}._core",
["src/main.cpp"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should take a look at the path here again.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is an interesting discussion .. but I think we can have this discussion after we merge this PR
I am also checking this discussion with Leah (pyOpenSci) to see if there is already something recommended by pyOpenSci

cxx_std=11,
),
]


setup(
ext_modules=ext_modules,
)
5 changes: 4 additions & 1 deletion src/scicookie/{{cookiecutter.project_slug}}/conda/dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,10 @@ dependencies:
- rust
{%- elif cookiecutter.build_system == "scikit-build-core" %}
- scikit-build-core
- cmake
- cmake
{%- elif cookiecutter.build_system == "pybind11" %}
- pybind11
- cmake
{%- endif %}
- nodejs # used by semantic-release
{%- if cookiecutter.use_shellcheck == "yes" %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ In addition, you should know that to build our package we use
{%- elif cookiecutter.build_system == "scikit-build-core" -%}
In addition, you should know that to build our package we use
[scikit-build-core](https://scikit-build-core.readthedocs.io/en/latest/): It's a Python packaging tool and build system an improved build system generator for CPython C extensions. It provides better support for additional compilers, build systems, cross compilation, and locating dependencies and their associated build requirements.This tool improves package management in the scientific Python ecosystem, enabling cross-platform builds with CMake, and seamless integration with C/C++ libraries for research software engineers.
{%- elif cookiecutter.build_system == "pybind11" -%}
In addition, you should know that to build our package we use
[pybind11](https://pybind11.readthedocs.io/en/stable/): It's a Python packaging tool for C++ build system that simplifies creating Python bindings for C++ code, allowing easy integration of C++ functions and classes into Python scripts. Acting as a bridge between the two languages, it enables direct calls to C++ functionality from Python as if it were a native Python module. Its user-friendly syntax reduces boilerplate code, while standard C++ build systems like CMake or Make aid in project compilation. Pybind11's efficiency and strong community support make it a popular choice for projects requiring seamless interoperability between C++ and Python, from scientific computing to game development.

{%- endif %}

Contributions are welcome, and they are greatly appreciated! Every little bit
Expand Down
6 changes: 5 additions & 1 deletion tests/smoke/base.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,11 @@ elif command -v hatch &> /dev/null; then
elif command -v maturin &> /dev/null; then
pip install .
elif [ "$(pip list|grep -c scikit_build_core)" -ne "0" ]; then
pip install .
pip install -e .
ayeankit marked this conversation as resolved.
Show resolved Hide resolved
elif [ "$(pip list|grep -c pybind11)" -ne "0" ]; then
# Assuming you are inside the root of the CMake source directory
pip install .
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should still be -e

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my last PR I removed -e from everywhere, so we will be sure to test the package correctly (as much as possible). my main concern is about testing this extension files .. so we would ensure to test if that is in the package .. maybe the test is not good enough yet ... but at least it is a first step


else
# use setuptools
pip install .
Expand Down
1 change: 1 addition & 0 deletions tests/smoke/build-systems.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ SMOKE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
. ${SMOKE_DIR}/base.sh "build_system=hatch"
. ${SMOKE_DIR}/base.sh "build_system=maturin"
. ${SMOKE_DIR}/base.sh "build_system=scikit-build-core"
. ${SMOKE_DIR}/base.sh "build_system=pybind11"
Loading