Support Python/C++ interoperability (native Python modules)

[davidzchen]

According to the documentation for py_library.deps:

The list of other libraries to be linked in to the binary target. See general comments about deps at Attributes common to all build rules. These can be py_binary rules, py_library rules or cc_library rules,

However, when I try to build a py_library target that depends on a cc_library, I get the following error:

❯❯❯ bazel build //external:markupsafe
ERROR: /private/var/tmp/_bazel_dzc/a4c09a1fcefb8b21c5b0e7d7db6f08ba/external/markupsafe_archive/BUILD:9:12: in deps attribute of py_library rule @markupsafe_archive//:markupsafe: cc_library rule '@markupsafe_archive//:speedups' is misplaced here (expected py_binary or py_library).
ERROR: Analysis of target '//external:markupsafe' failed; build aborted.
INFO: Elapsed time: 0.213s

According to BazelPyRuleClasses.java, the only allowed rule types in py_library.deps is, in fact, py_binary and py_library:

  public static final String[] ALLOWED_RULES_IN_DEPS = new String[] {
      "py_binary",
      "py_library",
  };

What is the plan for Python-C interoperability? Should we update the documentation to remove the mention of depending on cc_library in the meantime?

The background for this is that I am trying to use Jinja for generating HTML for the Skylark docgen tool that I am working on. However, Jinja depends on Markupsafe, which is partly implemented in C.

[davidzchen]

Some related discussion here: https://groups.google.com/forum/#!searchin/bazel-discuss/pex/bazel-discuss/-DFOrVqK8aY/-7h4e69hfkEJ

[ulfjack]

Recategorizing this as a feature request to support Python-C++ interop. I marked the other report that asked for that as a dupe.

[evanj]

A workarounds I found: you can include pre-compiled .sos in the py_library's data attribute. No idea if you can somehow get the output of a cc_library to work in this way.

[benley]

The trick to building a .so that works as a python extension seems to be making it a cc_binary instead of a cc_library. I think you have to build it with linkshared = 1 to make it work, but I'd love to be wrong about that.

[MarkusTeufelberger]

I stumbled upon this one also recently. Please at least change the documentation or (ideally) increase the priority to get this fixed for good. Python libraries and binaries can and do depend on C libraries. This is currently not really easy to model in bazel.

In the meantime it would also be nice to have this workaround documented somewhere.

[buchgr]

@MarkusTeufelberger we love contributions 😃

[ulfjack]

The problem is that we don't know how to fix it. We could export our internal model, but we're unsure if that's what people want. Our internal model is that all cc_library rules in deps of py_library rules get merged into a single .so at the py_binary level, together with all cc_library dependencies of py_extension rules, except the py_extension code itself, which gets compiled into a separate .so. It sounds awkward, and it is, but there doesn't seem to be any other principled approach to avoid ODR violations in the C++ code.

[ThomasColthurst]

+1 for exporting the internal model. My project is considering switching from bazel to Makefiles solely to avoid the ODR violations we are currently seeing when using bazel + clif + protobufs.

[hlopko]

Work on this is blocked by #4570. #4570 is fixed, nothing should block this effort.

[brandjon]

See related discussion in #1475.

[quval]

I have a draft of a Starlark implementation of something like the internal Google model (all native deps are linked together into a statically-linked shared object; the modules themselves are dynamically linked to it). Would it be interesting to share this? Perhaps on rules_python?

[pvcnt]

@quval It would certainly be interesting yes. :)

[quval]

OK, here goes: https://github.com/quval/rules_native_python/
Got it to work on both Linux and Mac, but only with a toy setup, so I'd still consider this a draft. More importantly, I'm not sure I like the hack I did to work around the circular dependency (native deps -- compiled into native module -- linked against native deps). Any thoughts and suggestions are welcome.

(Edit: just pushed a revised version. Edit 2: seems to have some issues when using gold - will update again once resolved. Edit 3: seems to be all right now. Edit 4, 11/2022: just pushed a major update after the library got some real use.)

[mfarrugi]

The following implementation seems to work for me without issue, are there any deficiencies I'm overlooking?

# py_native_library.bzl

def py_native_library(name, **kwargs):
	native.cc_binary(
		name = name + ".so",
		linkshared = True,
		**kwargs
	)
	native.py_library(
		name = name,
		data = [name + ".so"],
	)

Edit:

# py_native_library.bzl

def py_native_library(name, **kwargs):
	native.cc_binary(
		name = name + ".so",
		linkshared = True,
                linkstatic = False,
		**kwargs
	)
	native.py_library(
		name = name,
		data = [name + ".so"],
	)

Is more correct, covering the common case where more than one native extension is loaded into a python executable (and so shared dependencies are properly dynamically linked, rather than statically linked into each extension).

[kkpattern]

The problem is that we don't know how to fix it. We could export our internal model, but we're unsure if that's what people want. Our internal model is that all cc_library rules in deps of py_library rules get merged into a single .so at the py_binary level, together with all cc_library dependencies of py_extension rules, except the py_extension code itself, which gets compiled into a separate .so. It sounds awkward, and it is, but there doesn't seem to be any other principled approach to avoid ODR violations in the C++ code.

Hi ulfjack, how does google handle multiple py_library with some common cc_library depends? For example, if I have two py_library both depends on some X cc_library. Initially we may use those two py_library seperately, but at some point people may use both of these two py_library. I think in this situation there is still ODR violations?

[vkresch]

@mfarrugi thanks for your example. This works when I am using pybind11. Meaning if I want to call C++ functions via python this works. But how about the other way around call python functions from C++. Can I do this then:

def cc_native_library(name, py_srcs, py_deps, cc_srcs, cc_deps, **kwargs):
	native.py_library(
		name = name + ".py",
                srcs = py_srcs,
                deps = py_deps,
		**kwargs
	)
	native.cc_library(
		name = name,
                srcs = cc_srcs,
                deps = cc_deps,
		data = [name + ".py"],
		linkshared = True,
                linkstatic = False,
                deps = [":python3"], # depend on python3 path location
	)

I want to use the cc_native_library for a cc_binary.
I tried this approach for pybind11 but the problem is here with the third party packages which do not get imported correctly since they are used in the data argument of cc_library. If using only builtin libraries this works. Any suggestions or examples? Thanks in advance!

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please post @bazelbuild/triage in a comment here and we'll take a look. Thanks!

[jab]

@bazelbuild/triage