Can't pickle <class 'builtins.RustClass'>: attribute lookup RustClass on builtins failed

[jafioti]

When attempting to pickle a PyO3 class, I am getting an error:
Can't pickle <class 'builtins.RustClass'>: attribute lookup RustClass on builtins failed

I have copied everything from this example here:
https://gist.github.com/ethanhs/fd4123487974c91c7e5960acc9aa2a77

Is pickle a supported feature or is this simply unavailable?

[davidhewitt]

Hi @jafioti, while we don't have an official solution or documentation, what you're doing is mostly correct. Looking at the error, you need to make sure RustClass has the module set to the name of your extension module (otherwise it defaults to builtins).

You can do this with #[pyclass(module = "mymodule")].

[jafioti]

Thanks for the response, now I have a new error:
_pickle.PicklingError: Can't pickle <class 'RustModule.RustSubmodule.RustClass'>: import of module 'RustModule.RustSubmodule' failed

Since this class is in a submodule, should I set the module to "RustModule.RustSubmodule" or just "RustModule"? It fails either way, but when I use "RustModule" it gives a different error:
_pickle.PicklingError: Can't pickle <class 'RustModule.RustClass'>: attribute lookup RustClass on RustModule failed

[jafioti]

I found out that when I try to directly do import RustModule.RustSubmodule python does not find it, however when I do from RustModule import RustSubmodule it gets properly imported. I would assume I am missing something from my submodule or my main module, because as far as I know, those imports should be interchangable.

Here is my top-level module implementation:

#[pymodule]
fn RustModule(py: Python, module: &PyModule) -> PyResult<()> {
    let submodule = PyModule::new(py, "RustSubmodule")?;
    submodule::init_tokenization(submodule)?;
    module.add_submodule(submodule)?;

    Ok(())
}

[birkenfeld]

The reason it works for from X import Y is that there, Y can be any object in the module's dict, while for import X.Y X must be a package and Y must be a module.

That would indicate that the RustModule is not properly set up as a package (i.e. a module that contains submodules). I don't quite remember what is needed for that - probably __path__.

[birkenfeld]

So - some experimentation results: on Python 3.7, the error message for your case is

No module named 'RustModule.RustSubmodule'; 'RustModule' is not a package

Adding a __path__ = [] attribute changes it to just

No module named 'RustModule.RustSubmodule'

so it does make Python regard the top module as a package, but still doesn't find the submodule. This is probably because it does not care that the submodule is already present in the module's namespace and rather wants to look for some file on the filesystem.

It seems that what you need to do in addition is to pretend the module has already been imported, by placing it in sys.modules (with the correct name of RustModule.RustSubmodule). So this snippet worked for me:

#[pymodule]
fn RustModule(py: Python, module: &PyModule) -> PyResult<()> {
    let submodule = PyModule::new(py, "RustModule.RustSubmodule")?;
    // py_run! is quick-and-dirty; should be replaced by PyO3 API calls in actual code
    py_run!(py, submodule, "import sys; sys.modules['RustModule.RustSubmodule'] = submodule");
    // this is actually not needed now that we don't trigger the import mechanism...
    // module.setattr("__path__", PyList::empty(py))?;
    module.add_submodule(submodule)?;

    Ok(())
}

[birkenfeld]

@davidhewitt if this use case is to be supported properly, I think there should be a helper API to do this easily, maybe even in add_submodule (which currently does nothing more than add).

[davidhewitt]

@birkenfeld nice find - I'd looked at the submodule import problem before but hadn't managed to find that placing it in sys.modules is the correct solution!

Agreed we should implement first-class support for this case. It would be nice if the module path was computed automatically rather than having to provide the full path to PyModule::new. This might require a little redesign of the module syntax.

[birkenfeld]

Well, I can't promise it's the correct solution, but it works because it bypasses the import machinery.

I didn't remember an example of such a module hierarchy entirely implemented in C to look up how others are doing it.

[jafioti]

@birkenfeld @davidhewitt Thanks for the solution, although hacky it works!