feat(template): Added pybind11 as an option for build system #163

ayeankit · 2023-08-03T17:41:39Z

Pull Request description

Fixes #70

Activate option in TUI to pybind11
Adding documentation about pybind11
Adding pybind11 as a build-system:
Creating a pybind11-pyproject.toml
Created cmake file for pybind11.
Editing post_gen_project.py
Creating a smoke test (build-system.sh)
Editing cookicutter.json
Added pybind11 in Readme.md

How to test these changes

...

Pull Request checklists

This PR is a:

bug-fix
new feature
maintenance

About this PR:

it includes tests.
the tests are executed on CI.
the tests generate log file(s) (path).
pre-commit hooks were executed locally.
this PR requires a project documentation update.

Author's checklist:

I have reviewed the changes and it contains no misspelling.
The code is well commented, especially in the parts that contain more complexity.
New and old tests passed locally.

Additional information

Reviewer's checklist

Copy and paste this template for your review's note:

## Reviewer's Checklist

- [ ] I managed to reproduce the problem locally from the `main` branch
- [ ] I managed to test the new changes locally
- [ ] I confirm that the issues mentioned were fixed/resolved .

xmnlab

I am not super familiar with pybind11
but maybe it would be necessary to have an extra setup.py file
ref: https://github.com/pybind/python_example/blob/master/setup.py

xmnlab · 2023-08-04T00:00:03Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/skcdemo.cpp

@@ -4,6 +4,6 @@ namespace py = pybind11;

 PYBIND11_MODULE(skcdemo, m) {
    m.def("hello", [](){
-        py::print("Hello, scikit-build-core!");
+        py::print("Hello !!!");


maybe you should change the name of the file to something more generic than skcdemo
maybe example.cpp ... or as it uses pybind11 .. maybe pybind11.cpp ..

xmnlab · 2023-08-04T00:04:21Z

tests/smoke/base.sh

@@ -55,6 +55,14 @@ elif command -v maturin &> /dev/null; then
  pip install -e .
 elif [ "$(pip list|grep -c scikit_build_core)" -ne "0" ]; then
  pip install -e .
+elif [ "$(pip list|grep -c pybind11)" -ne "0" ]; then
+  # Assuming you are inside the root of the CMake source directory
+  mkdir build


ideally you should run just pip install -e . and it should run cmake under the hook

xmnlab · 2023-08-04T00:06:24Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/pybind11-pyproject.toml

+Discussions = "{{ cookiecutter.project_url }}/discussions"
+Changelog = "{{ cookiecutter.project_url }}/releases"
+
+[tool.hatch.version]


I don't think it needs hatch session here

Saransh-cpp

Great work, thanks, @ayeankit! Some comments below -

Saransh-cpp · 2023-08-04T15:44:53Z

docs/guide.md

@@ -334,7 +334,21 @@ packages. SciCookie support the following:
  build requirements. With its capabilities, it facilitates cross-platform
  builds using CMake and effortless integration with C/C++ libraries, making
  it a valuable asset for research software engineers. 
-
+
+- [**Pybind11**](https://pybind11.readthedocs.io/en/stable/): It's build system designed


Should we call it something like setuptools + pybind11 to be more clear, given that pybind11 itself is not a backend?

Or atleast using "setuptools + pybind11" in all the user-facing locations, and just "pybind11" in the internals of scicookie.

Or atleast using "setuptools + pybind11" in all the user-facing locations, and just "pybind11" in the internals of scicookie.

that makes sense to me.

Or atleast using "setuptools + pybind11" in all the user-facing locations, and just "pybind11" in the internals of scicookie.

that makes sense to me.

Wouldn't that be confusing since we are using setuptools as default build-sytem?

not really ... setuptools is the default option because it was the first one historically, but a lot of people want to kill that. That is why a bunch of project are looking for new alternatives.
but for c/c++ code setuptools is still a great option.
so specifying setuptools + pybind11 makes a lot of sense, because it is clear about its structure.

Saransh-cpp · 2023-08-04T15:45:50Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/CMakeLists.txt

+
+install(TARGETS pybind11 DESTINATION .)
+
+{%- endif %}                           


Suggested change

{%- endif %}

{%- endif %}

Saransh-cpp · 2023-08-04T15:46:47Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/pybind11.cpp

+    m.def("hello", [](){
+        py::print("Hello, pybind11!");
+    });
+}


Suggested change

}

}

Saransh-cpp · 2023-08-04T15:47:29Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/setup.py

+    name="OSL Python package",
+    author="{{cookiecutter.author_full_name}}",
+    author_email="{{cookiecutter.author_email}}",
+    url="https://{{ cookiecutter.project_slug }}.com",
+    description="A test project using pybind11",
+    long_description="",
+    #extras_require={"test": "pytest"},
+    # Currently, build_ext only provides an optional "highest supported C++
+    # level" feature, but in the future it may provide more features.
+    python_requires=">=3.7",


All of this is already specified in pyproject.toml. We would need something like this - https://github.com/scientific-python/cookie/blob/main/%7B%7Bcookiecutter.project_name%7D%7D/%7B%25%20if%20cookiecutter.backend%3D%3D'pybind11'%20%25%7Dsetup.py%7B%25%20endif%20%25%7D

xmnlab · 2023-08-04T19:14:14Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/CMakeLists.txt

@@ -1,3 +1,4 @@
+{% if cookiecutter.build_system == "scikit-build-core" -%}


where are you declaring this variable? SKBUILD_PROJECT_NAME

it doesn't look correct to resuse the name sk... for both build system
also the first two lines seem to be duplicated .. probably you could move it to some first lines in order to reuse that for both build system

Saransh-cpp · 2023-08-04T20:44:25Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/pybind11.cpp

+
+namespace py = pybind11;
+
+PYBIND11_MODULE(skcdemo, m) {


skcdemo can be called maybe pybinddemo?

Saransh-cpp

Some comments below that might push you in the right direction for fixing the CI error 🙂

Saransh-cpp · 2023-08-07T02:32:21Z

src/scicookie/{{cookiecutter.project_slug}}/Makefile

@@ -115,6 +115,9 @@ build:
 	hatch build
 {%- elif cookiecutter.build_system == "maturin" %}
 	maturin build	
+{%- elif cookiecutter.build_system == "pybind11" %}
+	pybind11 build	


Are you sure this command exists?

I think you can use something like python3 -m build https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#packaging-your-project

ensure you have build as a dependency in pyproject.toml/build-system

Saransh-cpp · 2023-08-07T02:33:49Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/CMakeLists.txt

+
+find_package(pybind11 CONFIG REQUIRED)
+
+pybind11_add_module(_core MODULE src/main.cpp)


I think the src should not be here? Or it should be src/<package_name>/main.cpp?

Saransh-cpp · 2023-08-07T02:35:25Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/pybind11-pyproject.toml

@@ -0,0 +1,121 @@
+[build-system]
+requires = ["setuptools>=42", "wheel", "pybind11~=2.6.1"]


IIRC, editable install with pyproject.toml support was first supported with setuptools version 65. Could you please check and confirm?

Saransh-cpp · 2023-08-07T02:36:18Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/setup.py

@@ -0,0 +1,23 @@
+from __future__ import annotations


We are not adding type annotations anywhere in this file. What purpose is this import serving then?

Saransh-cpp · 2023-08-07T02:37:04Z

tests/smoke/base.sh

@@ -55,6 +55,9 @@ elif command -v maturin &> /dev/null; then
  pip install -e .
 elif [ "$(pip list|grep -c scikit_build_core)" -ne "0" ]; then
  pip install -e .
+elif [ "$(pip list|grep -c pybind11)" -ne "0" ]; then
+  # Assuming you are inside the root of the CMake source directory
+  pip install -e .   


Yes! This -e would not work with pyproject.toml and setuptools<65, IIRC.

Saransh-cpp · 2023-08-07T17:07:00Z

src/scicookie/{{cookiecutter.project_slug}}/build-system/setup.py

+ext_modules = [
+    Pybind11Extension(
+        "{{ cookiecutter.project_slug }}._core",
+        ["src/main.cpp"],


You should take a look at the path here again.

this is an interesting discussion .. but I think we can have this discussion after we merge this PR
I am also checking this discussion with Leah (pyOpenSci) to see if there is already something recommended by pyOpenSci

Saransh-cpp · 2023-08-07T17:07:16Z

tests/smoke/base.sh

@@ -55,6 +55,9 @@ elif command -v maturin &> /dev/null; then
  pip install -e .
 elif [ "$(pip list|grep -c scikit_build_core)" -ne "0" ]; then
  pip install -e .
+elif [ "$(pip list|grep -c pybind11)" -ne "0" ]; then
+  # Assuming you are inside the root of the CMake source directory
+  pip install .   


This should still be -e

in my last PR I removed -e from everywhere, so we will be sure to test the package correctly (as much as possible). my main concern is about testing this extension files .. so we would ensure to test if that is in the package .. maybe the test is not good enough yet ... but at least it is a first step

src/scicookie/{{cookiecutter.project_slug}}/Makefile

Saransh-cpp · 2023-08-09T17:34:02Z

src/scicookie/hooks/post_gen_project.py

+        build_system_dir = PROJECT_DIRECTORY / "build-system"
+        src_system_dir = PROJECT_DIRECTORY/ "src"


I think PROJECT_DIRECTORY automatically checks for flat or src layout?

this is an interesting discussion ... maybe we can return to this discussion to check how we should proceed not just for this build-system but maybe for all build-systems.
also maybe in our repo, maybe we could have this files in folders called by the programming language (cpp, rust, etc)

Co-authored-by: Saransh Chopra <[email protected]>

ayeankit · 2023-08-09T18:58:00Z

Hey @xmnlab , All test cases passed !!, I have changed pybind11 to setuptools + pybind11 in user-facing locations as suggested by Saransh. Can you please review PR?

tests/smoke/base.sh

Co-authored-by: Ivan Ogasawara <[email protected]>

xmnlab · 2023-08-09T19:29:21Z

src/scicookie/{{cookiecutter.project_slug}}/Makefile

@@ -115,6 +115,9 @@ build:
 	hatch build
 {%- elif cookiecutter.build_system == "maturin" %}
 	maturin build	
+{%- elif cookiecutter.build_system == "pybind11" %}
+	python -m pip install build


you are installing build here, but it is already defined by pyproject in the build section: https://github.com/osl-incubator/scicookie/pull/163/files#diff-ffc1694c12f08f07de37e5816475fd014df381d8fc20a600597559c8072b61fdR2

this step here is to build the package, not to install a dependency for building

@Saransh-cpp and I already told you what the command you should try here
because in another words, you are not doing what it should do, it should create a package in this step

xmnlab

thanks for working on that @Saransh-cpp
the only missing point (in my opinion) is this issue: https://github.com/osl-incubator/scicookie/pull/163/files#r1289103642

ayeankit · 2023-08-09T20:51:17Z

hey @xmnlab , I did the required changes, and added "build>=0.10.0", in dependency in pyproject.toml for pybind11. Can you please review it again?

xmnlab

thanks for working on that @ayeankit !

lgtm!

the only thing that is not working well is the version, as you can see in the logs:

creating osl-python-package-0.0.0
creating osl-python-package-0.0.0/src
creating osl-python-package-0.0.0/src/osl_python_package
creating osl-python-package-0.0.0/src/osl_python_package.egg-info
creating osl-python-package-0.0.0/tests
copying files to osl-python-package-0.0.0...
copying LICENSE -> osl-python-package-0.0.0
copying README.md -> osl-python-package-0.0.0
copying pyproject.toml -> osl-python-package-0.0.0
copying setup.py -> osl-python-package-0.0.0
copying src/main.cpp -> osl-python-package-0.0.0/src
copying src/osl_python_package/__init__.py -> osl-python-package-0.0.0/src/osl_python_package
copying src/osl_python_package/osl_python_package.py -> osl-python-package-0.0.0/src/osl_python_package
copying src/osl_python_package.egg-info/PKG-INFO -> osl-python-package-0.0.0/src/osl_python_package.egg-info
copying src/osl_python_package.egg-info/SOURCES.txt -> osl-python-package-0.0.0/src/osl_python_package.egg-info
copying src/osl_python_package.egg-info/dependency_links.txt -> osl-python-package-0.0.0/src/osl_python_package.egg-info
copying src/osl_python_package.egg-info/requires.txt -> osl-python-package-0.0.0/src/osl_python_package.egg-info
copying src/osl_python_package.egg-info/top_level.txt -> osl-python-package-0.0.0/src/osl_python_package.egg-info
copying tests/test_osl_python_package.py -> osl-python-package-0.0.0/tests

but we can address this issue in a follow-up.

github-actions · 2023-08-11T15:33:20Z

🎉 This PR is included in version 0.6.0 🎉

The release is available on:

0.6.0
GitHub release

Your semantic-release bot 📦🚀

add pybind11 build-system as option

f816654

xmnlab requested changes Aug 4, 2023

View reviewed changes

added setup.py

e9c6226

ayeankit force-pushed the ayeankit/pybind11 branch from 0855584 to e9c6226 Compare August 4, 2023 15:01

Saransh-cpp requested changes Aug 4, 2023

View reviewed changes

added setup.py

ff88a54

ayeankit force-pushed the ayeankit/pybind11 branch from 22f7d50 to ff88a54 Compare August 4, 2023 15:50

xmnlab changed the title ~~Feat(template): Added pybind11 as an option for build system~~ feat(template): Added pybind11 as an option for build system Aug 4, 2023

xmnlab reviewed Aug 4, 2023

View reviewed changes

Saransh-cpp requested changes Aug 4, 2023

View reviewed changes

added setup.py

7e83d22

ayeankit force-pushed the ayeankit/pybind11 branch from 8e3b63a to 7e83d22 Compare August 5, 2023 14:07

ayeankit and others added 2 commits August 5, 2023 19:44

added setup.py

2e84ac2

Update guide.md

1f04f8e

Saransh-cpp requested changes Aug 7, 2023

View reviewed changes

ayeankit added 4 commits August 7, 2023 22:19

Update setup.py

40433a4

Update CMakeLists.txt

7625f4e

Update pybind11-pyproject.toml

5e78f26

Update base.sh

a61dfc6

Saransh-cpp requested changes Aug 7, 2023

View reviewed changes

ayeankit added 10 commits August 7, 2023 23:12

Update base.sh

4c4877c

Update setup.py

5dfb8c5

Update base.sh

0b252fc

Update setup.py

c95ebad

Update CMakeLists.txt

c593005

Update pybind11-pyproject.toml

653e049

Update setup.py

535c090

Update setup.py

83dc6e0

Update Makefile

5359d5c

Create pybind11

c79bb3b

ayeankit and others added 3 commits August 9, 2023 20:32

Merge branch 'main' into ayeankit/pybind11

642b9ba

pre-commit clear

4ec2dfe

Update Makefile

bf7529d

Saransh-cpp requested changes Aug 9, 2023

View reviewed changes

ayeankit and others added 7 commits August 9, 2023 23:26

Update src/scicookie/{{cookiecutter.project_slug}}/Makefile

bf5d323

Co-authored-by: Saransh Chopra <[email protected]>

Update pybind11-pyproject.toml

bcff667

Update guide.md

11ccc28

Update pybind11-pyproject.toml

4ce85f4

Update Makefile

410d29d

Update README.md

a18f98d

Update contributing.md

cd8ed24

xmnlab reviewed Aug 9, 2023

View reviewed changes

tests/smoke/base.sh Outdated Show resolved Hide resolved

Update tests/smoke/base.sh

b885513

Co-authored-by: Ivan Ogasawara <[email protected]>

xmnlab reviewed Aug 9, 2023

View reviewed changes

xmnlab requested changes Aug 9, 2023

View reviewed changes

ayeankit added 3 commits August 10, 2023 01:02

Update Makefile

b52b882

Update Makefile

d7114a6

added build as dependency

2e4cbf7

xmnlab approved these changes Aug 9, 2023

View reviewed changes

xmnlab merged commit b3d59cd into osl-incubator:main Aug 9, 2023
10 checks passed

github-actions bot added the released label Aug 11, 2023

		@@ -1,3 +1,4 @@
		{% if cookiecutter.build_system == "scikit-build-core" -%}


		find_package(pybind11 CONFIG REQUIRED)

		pybind11_add_module(_core MODULE src/main.cpp)

		@@ -0,0 +1,121 @@
		[build-system]
		requires = ["setuptools>=42", "wheel", "pybind11~=2.6.1"]

		build_system_dir = PROJECT_DIRECTORY / "build-system"
		src_system_dir = PROJECT_DIRECTORY/ "src"

feat(template): Added pybind11 as an option for build system #163

feat(template): Added pybind11 as an option for build system #163

Conversation

ayeankit commented Aug 3, 2023 • edited Loading

Pull Request description

How to test these changes

Pull Request checklists

Additional information

Reviewer's checklist

xmnlab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Saransh-cpp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ayeankit Aug 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Saransh-cpp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ayeankit commented Aug 9, 2023

Choose a reason for hiding this comment

xmnlab left a comment

Choose a reason for hiding this comment

ayeankit commented Aug 9, 2023

xmnlab left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 11, 2023

ayeankit commented Aug 3, 2023 •

edited

Loading

ayeankit Aug 4, 2023 •

edited

Loading