Update PyTryFrom for PyMapping and PySequence to more accurately check types

[aganders3]

This is intended to fix #2072 - see further discussion in that thread.

The implementation is such that for both PyMapping and PyTryFrom, they will call into python to check isinstance(<obj>, collections.abc.Mapping) before successfully downcasting. Please note this is a breaking change so it is important the documentation and migration guide are clear.

Also note the implementation has changed from the initial draft of this PR. See discussion below and in #2072 for more information.

TODO

• an entry in CHANGELOG.md
• docs to all new functions and / or detail in the guide
• tests for all new or changed functions

[davidhewitt]

Thanks for working on this! Got a few thoughts to get us started...

[aganders3]

As a sanity check, I wonder if there is any evidence we can collect (or tests we can write) that this check is functionally equivalent to the slow path? If there's any lingering concern that they may be different, we probably need to always use the slow path.

I did a little research and testing on this. In normal usage I think they will give the same result, but in the end they are not identical because ultimately the collections.abc.Mapping __instancecheck__ and __subclasscheck__ don't look at tp_flags.

With custom classes there's the chance that (for example) a pyo3 user could do something like this - this feels like a bit of a contrived example but I did test it:

#[pyclass(mapping)]
struct Mapping {
    index: HashMap<String, usize>,
}

#[pymethods]
impl Mapping {
    #[new]
    fn new(elements: Option<&PyList>) -> PyResult<Self> {
    // ...
    // truncated implementation of this mapping pyclass - basically a wrapper around a HashMap
}

// force the Mapping class to have Py_TPFLAGS_MAPPING set
#[pyfunction]
fn set_tpflags_mapping(py: Python<'_>) {
    unsafe {
        let ty = Mapping::type_object_raw(py);
        (*ty).tp_flags |= ffi::Py_TPFLAGS_MAPPING;
    }
}

Then testing in Python:

import collections.abc

import pymapping

pymapping.set_tpflags_mapping()

for obj in (dict(), list(), pymapping.Mapping()):
    print(f"type: {type(obj)}")
    print(f"\tmapping_by_tpflags: {pymapping.check_tpflags(obj)}")
    print(f"\tmapping_by_isinstance: {pymapping.check_isinstance(obj)}")

Here's the output - note the discrepancy between mapping_by_tpflags and mapping_by_isinstance for the custom Mapping class:

~/src/rust/pymapping via 🐍 v3.10.2 (.venv) via 🦀 v1.61.0
❯ python test.py
type: <class 'dict'>
	mapping_by_tpflags: True
	mapping_by_isinstance: True
type: <class 'list'>
	mapping_by_tpflags: False
	mapping_by_isinstance: False
type: <class 'builtins.Mapping'>
	mapping_by_tpflags: True
	mapping_by_isinstance: False

It also gets a little hairy because in CPython there are actually two implementations of the ABCs (in C and Python). Only the C-implementation sets Py_TPFLAGS_MAPPING for Foo when you call collections.abc.Mapping.register(Foo), but this does not happen in the Python implementation. This distinction may apply to directly subclassing as well. I'm not sure under which circumstances the Python implementation is used.

[davidhewitt]

Thanks for the really thorough investigation. So it seems a little unfortunate but I'm now wondering if the best thing is for us to always do the isinstance check? In combination with Mapping.register that's probably the only way to be reliable in all combinations of abi / C implementations?

We could at least cache collections.abc.Mapping in a GILOnceCell to reduce overhead.

[aganders3]

Yeah it's a bit sad but it would be more consistent behavior, even if the discrepancy would be rare. Still I'll defer to your (and others') expertise and experience on this. I don't have a good feel for how big this performance hit will be and/or how common the operation is in real use.

For performance-critical situations could a user still do their own check and use try_from_unchecked? It might be sufficient to just document this in the PyMapping docs with an explanation that there is no fast perfect check for mapping types at this time.

[davidhewitt]

Sorry for the slight delay - had a backlog of things to get through.

I think consistent behaviour is most important, so I think we should go for the isinstance check always. As you say, users can use unsafe APIs to check the flags / use try_from_unchecked if they really need performance and are willing to accept the differences.

[davidhewitt]

Thanks, this is shaping up well. Posted some implementation suggestions.

I'm beginning to wonder if we should do equivalent treatment for PySequence? Seems like it would be good to have both for consistency.

Also I keep going back and forth on whether we should have #[pyclass(mapping)] automatically register with the abc. I think it's still the right choice to make users opt-in, however if they don't when upgrading to 0.17 their casts to PyMapping will break so we should make a very clear entry about this in the migration guide.

[davidhewitt]

Well @aganders3 nobody has raised concerns about also changing sequences, so I guess let's move ahead with that. Do you want to add to this PR, or shall I push separately?

[aganders3]

Well @aganders3 nobody has raised concerns about also changing sequences, so I guess let's move ahead with that. Do you want to add to this PR, or shall I push separately?

I think it makes sense to add it here as there will be overlap at least in the docs/changelog for it. I'm just getting back from vacation but should be able to do that some time this week.

[davidhewitt]

That'd be great, thanks! There's still a few items to go in #2493 before we're ready to go, so there's some time to get this in. I'm crossing things off that list when I can! :)

[davidhewitt]

Thanks, this looks really solid. Just a couple of final things to decide on before we merge this in. I've commented on the sequence implementation, the equivalent ideas apply to the mapping implementation.

[aganders3]

Thanks again for the review. It was trickier than I thought to change the static to hold a PyResult, but I'm getting a better feel for result and error types now and hopefully what I did makes sense. Does the clone_ref negate the benefit of the GILOnceCell? Without that the return was Result<&PyType, &PyErr> which was more awkward to use.

I rebased on main again so hopefully it's still green. Please take another look when you have a chance!

[davidhewitt]

Looking brilliant! Just two final tiny suggestions and then let's merge this. Thanks again for a really solid PR here 👍

[aganders3]

Thanks, this looks really solid. Just a couple of final things to decide on before we merge this in. I've commented on the sequence implementation, the equivalent ideas apply to the mapping implementation.

Thanks for all the feedback and review!

[davidhewitt]

Looks great, thanks very much again!

[Gobot1234]

@aganders3 thank you so much for fixing this!