Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add abi3-py* features #1263

Merged
merged 6 commits into from
Dec 8, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

### Added
- Add support for building for CPython limited API. This required a few minor changes to runtime behaviour of of pyo3 `#[pyclass]` types. See the migration guide for full details. [#1152](https://github.com/PyO3/pyo3/pull/1152)
- Add feature flags `abi3-py*` to set the minimum Python version when using the limited API. [#1263]((https://github.com/PyO3/pyo3/pull/1263))
- Add argument names to `TypeError` messages generated by pymethod wrappers. [#1212](https://github.com/PyO3/pyo3/pull/1212)
- Add `PyEval_SetProfile` and `PyEval_SetTrace` to FFI. [#1255](https://github.com/PyO3/pyo3/pull/1255)
- Add context.h functions (`PyContext_New`, etc) to FFI. [#1259](https://github.com/PyO3/pyo3/pull/1259)
Expand Down
8 changes: 6 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,13 @@ rustversion = "1.0"
[features]
default = ["macros"]
macros = ["ctor", "indoc", "inventory", "paste", "pyo3cls", "unindent"]
# Use the Python limited API. See https://www.python.org/dev/peps/pep-0384/ for
# more.
# Use the Python limited API. See https://www.python.org/dev/peps/pep-0384/ for more.
abi3 = []
# With abi3, we can manually set the minimum Python version.
abi3-py36 = ["abi3-py37"]
abi3-py37 = ["abi3-py38"]
abi3-py38 = ["abi3-py39"]
abi3-py39 = ["abi3"]

# Optimizes PyObject to Vec conversion and so on.
nightly = []
Expand Down
45 changes: 40 additions & 5 deletions build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@ use std::{
str::FromStr,
};

const PY3_MIN_MINOR: u8 = 5;
/// Minimum required Python version.
const PY3_MIN_MINOR: u8 = 6;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not related to this PR, but we dropped 3.5 support in #1250.

/// Maximum Python version that can be used as minimum required Python version with abi3.
const ABI3_MAX_MINOR: u8 = 9;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Binaries trying to link against a higher python version than the actual installed libpython will fail because symbols will mismatch. So this new feature can only work for extension modules.

(And even worse, only extension modules on linux - because on windows they always link!)

This would mean that an extension module which uses the abi3-pyXY features would further break cargo test even after #1123 , and would also not be cross-platform.

const CFG_KEY: &str = "py_sys_config";

type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;
Expand Down Expand Up @@ -770,12 +773,25 @@ fn configure(interpreter_config: &InterpreterConfig) -> Result<String> {
bail!("Python 2 is not supported");
}

if env::var_os("CARGO_FEATURE_ABI3").is_some() {
let minor = if env::var_os("CARGO_FEATURE_ABI3").is_some() {
println!("cargo:rustc-cfg=Py_LIMITED_API");
}
// Check any `abi3-py3*` feature is set. If not, use the interpreter version.
let abi3_minor = (PY3_MIN_MINOR..=ABI3_MAX_MINOR)
.find(|i| env::var_os(format!("CARGO_FEATURE_ABI3_PY3{}", i)).is_some());
match (abi3_minor, interpreter_config.version.minor) {
(Some(abi3_minor), Some(interpreter_minor)) if abi3_minor > interpreter_minor => bail!(
"You cannot set a mininimum Python version {} higher than the interpreter version {}",
abi3_minor,
interpreter_minor
),
_ => abi3_minor.or(interpreter_config.version.minor),
}
} else {
interpreter_config.version.minor
};

if let Some(minor) = interpreter_config.version.minor {
for i in 6..=minor {
if let Some(minor) = minor {
for i in PY3_MIN_MINOR..=minor {
println!("cargo:rustc-cfg=Py_3_{}", i);
flags += format!("CFG_Py_3_{},", i).as_ref();
}
Expand Down Expand Up @@ -820,7 +836,26 @@ fn check_target_architecture(interpreter_config: &InterpreterConfig) -> Result<(
Ok(())
}

fn abi3_without_interpreter() -> Result<()> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good start, in the future I think we can replace this with something more flexible.

See this discussion with the PyOxidiser maintainer who suggested if PyOxidiser used PyO3 they would like the ability to set an environment variable with a path to a JSON file which configures PyO3: indygreg/PyOxidizer#324 (comment)

println!("cargo:rustc-cfg=Py_LIMITED_API");
let mut flags = "FLAG_WITH_THREAD=1".to_string();
for minor in PY3_MIN_MINOR..=ABI3_MAX_MINOR {
println!("cargo:rustc-cfg=Py_3_{}", minor);
flags += &format!(",CFG_Py_3_{}", minor);
}
println!("cargo:rustc-cfg=py_sys_config=\"WITH_THREAD\"");
println!("cargo:python_flags={}", flags);
Ok(())
}

fn main() -> Result<()> {
// If PYO3_NO_PYTHON is set with abi3, we can build PyO3 without calling Python (UNIX only).
// We only check for the abi3-py3{ABI3_MAX_MINOR} because lower versions depend on it.
if env::var_os("PYO3_NO_PYTHON").is_some()
&& env::var_os(format!("CARGO_FEATURE_ABI3_PY3{}", ABI3_MAX_MINOR)).is_some()
{
return abi3_without_interpreter();
}
// 1. Setup cfg variables so we can do conditional compilation in this library based on the
// python interpeter's compilation flags. This is necessary for e.g. matching the right unicode
// and threading interfaces. First check if we're cross compiling, if so, we cannot run the
Expand Down
11 changes: 10 additions & 1 deletion guide/src/building_and_distribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,16 @@ pyo3 = { version = "...", features = ["abi3"]}

3. Ensure that the `.whl` is correctly marked as `abi3`. For projects using `setuptools`, this is accomplished by passing `--py-limited-api=cp3x` (where `x` is the minimum Python version supported by the wheel, e.g. `--py-limited-api=cp35` for Python 3.5) to `setup.py bdist_wheel`.

### Minimum Python version for `abi3`

Because a single `abi3` wheel can be used with many different Python versions, PyO3 has feature flags `abi3-py36`, `abi3-py37`, `abi-py38` etc. to set the minimum required Python version for your `abi3` wheel.
For example, if you set the `abi3-py36` feature, your extension wheel can be used on all Python 3 versions from Python 3.6 and up. `maturin` and `setuptools-rust` will give the wheel a name like `my-extension-1.0-cp36-abi3-manylinux2020_x86_64.whl`.
If you set more that one of these api version feature flags the highest version always wins. For example, with both `abi3-py36` and `abi3-py38` set, PyO3 would build a wheel which supports Python 3.8 and up.
PyO3 is only able to link your extension module to api3 version up to and including your host Python version. E.g., if you set `abi3-py38` and try to compile the crate with a host of Python 3.6, the build will fail.

As an advanced feature, you can build PyO3 wheel without calling Python interpreter with
the environment variable `PYO3_NO_PYTHON` set, but this only works on *NIX.

## Cross Compiling

Cross compiling PyO3 modules is relatively straightforward and requires a few pieces of software:
Expand Down Expand Up @@ -107,4 +117,3 @@ For an example of how to build python extensions using Bazel, see https://github

[maturin]: https://github.com/PyO3/maturin
[setuptools-rust]: https://github.com/PyO3/setuptools-rust