Skip to content

Commit

Permalink
release: 0.14.0
Browse files Browse the repository at this point in the history
  • Loading branch information
davidhewitt committed Jun 30, 2021
1 parent b58fe20 commit fa304d2
Show file tree
Hide file tree
Showing 7 changed files with 113 additions and 90 deletions.
48 changes: 35 additions & 13 deletions Architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,20 +17,22 @@ protocol traits (e.g., `PyIterProtocol`) for supporting object protocols (i.e.,
Since implementing `PyClass` requires lots of boilerplate, we have a proc-macro `#[pyclass]`.

To summarize, there are six main parts to the PyO3 codebase.

1. [Low-level bindings of Python/C API.](#1-low-level-bindings-of-python-capi)
- [`src/ffi`]
- [`src/ffi`]
2. [Bindings to Python objects.](#2-bindings-to-python-objects)
- [`src/instance.rs`] and [`src/types`]
- [`src/instance.rs`] and [`src/types`]
3. [`PyClass` and related functionalities.](#3-pyclass-and-related-functionalities)
- [`src/pycell.rs`], [`src/pyclass.rs`], and more
- [`src/pycell.rs`], [`src/pyclass.rs`], and more
4. [Protocol methods like `__getitem__`.](#4-protocol-methods)
- [`src/class`]
- [`src/class`]
5. [Procedural macros to simplify usage for users.](#5-procedural-macros-to-simplify-usage-for-users)
- [`src/derive_utils.rs`], [`pyo3-macros`] and [`pyo3-macros-backend`]
- [`src/derive_utils.rs`], [`pyo3-macros`] and [`pyo3-macros-backend`]
6. [`build.rs`](#6-buildrs)
- [`build.rs`](https://github.com/PyO3/pyo3/tree/master/build.rs)
- [`build.rs`](https://github.com/PyO3/pyo3/tree/master/build.rs)

## 1. Low-level bindings of Python/C API

[`src/ffi`] contains wrappers of [Python/C API].

We aim to provide straight-forward Rust wrappers resembling the file structure of
Expand All @@ -44,24 +46,27 @@ In the [`src/ffi`] module, there is lots of conditional compilation such as `#[c
`#[cfg(Py_37)]`, and `#[cfg(PyPy)]`.
`Py_LIMITED_API` corresponds to `#define Py_LIMITED_API` macro in Python/C API.
With `Py_LIMITED_API`, we can build a Python-version-agnostic binary called an
[abi3 wheel](https://pyo3.rs/v0.13.2/building_and_distribution.html#py_limited_apiabi3).
[abi3 wheel](https://pyo3.rs/main/building_and_distribution.html#py_limited_apiabi3).
`Py_37` means that the API is available from Python >= 3.7.
There are also `Py_38`, `Py_39`, and so on.
`PyPy` means that the API definition is for PyPy.
Those flags are set in [`build.rs`](#6-buildrs).

## 2. Bindings to Python objects

[`src/types`] contains bindings to [built-in types](https://docs.python.org/3/library/stdtypes.html)
of Python, such as `dict` and `list`.
For historical reasons, Python's `object` is called `PyAny` in PyO3 and located in [`src/types/any.rs`].
Currently, `PyAny` is a straightforward wrapper of `ffi::PyObject`, defined as:

```rust
#[repr(transparent)]
pub struct PyAny(UnsafeCell<ffi::PyObject>);
```

All built-in types are defined as a C struct.
For example, `dict` is defined as:

```c
typedef struct {
/* Base object */
Expand All @@ -77,6 +82,7 @@ typedef struct {

However, we cannot access such a specific data structure with `#[cfg(Py_LIMITED_API)]` set.
Thus, all builtin objects are implemented as opaque types by wrapping `PyAny`, e.g.,:

```rust
#[repr(transparent)]
pub struct PyDict(PyAny);
Expand All @@ -92,12 +98,14 @@ Since we need lots of boilerplate for implementing common traits for these types
[`src/types/mod.rs`].

## 3. `PyClass` and related functionalities

[`src/pycell.rs`], [`src/pyclass.rs`], and [`src/type_object.rs`] contain types and
traits to make `#[pyclass]` work.
Also, [`src/pyclass_init.rs`] and [`src/pyclass_slots.rs`] have related functionalities.

To realize object-oriented programming in C, all Python objects must have the following two fields
at the beginning.

```rust
#[repr(C)]
pub struct PyObject {
Expand All @@ -106,16 +114,19 @@ pub struct PyObject {
...
}
```

Thanks to this guarantee, casting `*mut A` to `*mut PyObject` is valid if `A` is a Python object.

To ensure this guarantee, we have a wrapper struct `PyCell<T>` in [`src/pycell.rs`] which is roughly:

```rust
#[repr(C)]
pub struct PyCell<T: PyClass> {
object: crate::ffi::PyObject,
inner: T,
}
```

Thus, when copying a Rust struct to a Python object, we first allocate `PyCell` on the Python heap and then
move `T` into it.
Also, `PyCell` provides [RefCell](https://doc.rust-lang.org/std/cell/struct.RefCell.html)-like methods
Expand All @@ -131,6 +142,7 @@ For example, you can see `type({})` shows `dict` and `type(type({}))` shows `typ
`T: PyTypeObject` implies that `T` has a corresponding type object.

## 4. Protocol methods

Python has some built-in special methods called dunder, such as `__iter__`.
They are called [abstract objects layer](https://docs.python.org/3/c-api/abstract.html) in
Python/C API.
Expand All @@ -147,40 +159,50 @@ Also, [`src/class/methods.rs`] has utilities for `#[pyfunction]` and [`src/class
some internal tricks for making `#[pyproto]` flexible.

## 5. Procedural macros to simplify usage for users.

[`pyo3-macros`] provides six proc-macro APIs: `pymodule`, `pyproto`, `pyfunction`, `pyclass`,
`pymethods`, and `#[derive(FromPyObject)]`.
[`pyo3-macros-backend`] has the actual implementations of these APIs.
[`src/derive_utils.rs`] contains some utilities used in code generated by these proc-macros,
such as parsing function arguments.

## 6. `build.rs`

PyO3's [`build.rs`](https://github.com/PyO3/pyo3/tree/master/build.rs) is relatively long
(about 900 lines) to support multiple architectures, interpreters, and usages.
Below is a non-exhaustive list of its functionality:

- Cross-compiling support.
- If `TARGET` architecture and `HOST` architecture differ, we find cross compile information
from environment variables (`PYO3_CROSS_LIB_DIR` and `PYO3_CROSS_PYTHON_VERSION`) or system files.
from environment variables (`PYO3_CROSS_LIB_DIR` and `PYO3_CROSS_PYTHON_VERSION`) or system files.
- Find the interpreter for build and detect the Python version.
- We have to set some version flags like `Py_37`.
- If the interpreter is PyPy, we set `PyPy`.
- If `PYO3_NO_PYTHON` environment variable is set then the interpreter detection is bypassed
entirely and only abi3 extensions can be built.
entirely and only abi3 extensions can be built.
- Check if we are building a Python extension.
- If we are building an extension (e.g., Python library installable by `pip`),
we don't link `libpython`.
Currently we use the `extension-module` feature for this purpose. This may change in the future.
See [#1123](https://github.com/PyO3/pyo3/pull/1123).
we don't link `libpython`.
Currently we use the `extension-module` feature for this purpose. This may change in the future.
See [#1123](https://github.com/PyO3/pyo3/pull/1123).

<!-- External Links -->
[Python/C API]: https://docs.python.org/3/c-api/

[python/c api]: https://docs.python.org/3/c-api/

<!-- Crates -->

[`pyo3-macros`]: https://github.com/PyO3/pyo3/tree/master/pyo3-macros
[`pyo3-macros-backend`]: https://github.com/PyO3/pyo3/tree/master/pyo3-macros-backend

<!-- Directories -->

[`src/class`]: https://github.com/PyO3/pyo3/tree/master/src/class
[`src/ffi`]: https://github.com/PyO3/pyo3/tree/master/src/ffi
[`src/types`]: https://github.com/PyO3/pyo3/tree/master/src/types

<!-- Files -->

[`src/derive_utils.rs`]: https://github.com/PyO3/pyo3/tree/master/src/derive_utils.rs
[`src/instance.rs`]: https://github.com/PyO3/pyo3/tree/master/src/instance.rs
[`src/pycell.rs`]: https://github.com/PyO3/pyo3/tree/master/src/pycell.rs
Expand Down
Loading

0 comments on commit fa304d2

Please sign in to comment.