[Bug] Python 3.10 builds are not working with PyOxidizer

[Cellebyte]

Steps to reproduce

Hey looks like there is an issue with the python-standalone build.
Between version 3.9 and 3.10.
I don't know if it is an issue with pyOxidizer or the building tools.

mkdir -p blub
python3.10 -m venv .venv 
source .venv/bin/activate
pip install pyoxidizer
cd ..
pyoxidizer init-config-file blub
cd blub
# add python3.10 to pyoxidizer.bzl
vim pyoxidizer.bzl # dist = default_python_distribution(python_version = "3.10") and save
pyoxidizer run

Output Python 3.9

Python 3.9.10 (main, Feb 27 2022, 23:33:24) 
[Clang 13.0.1 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Output Python 3.10

Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/home/cellebyte/git/dttechnik/blub/build/x86_64-unknown-linux-gnu/debug/install'
  program name = '/home/cellebyte/git/dttechnik/blub/./build/x86_64-unknown-linux-gnu/debug/install/blub'
  isolated = 1
  environment = 0
  user site = 0
  import site = 1
  sys._base_executable = '/home/cellebyte/git/dttechnik/blub/./build/x86_64-unknown-linux-gnu/debug/install/blub'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/home/cellebyte/git/dttechnik/blub/./build/x86_64-unknown-linux-gnu/debug/install/blub'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/home/cellebyte/git/dttechnik/blub/build/x86_64-unknown-linux-gnu/debug/install',
    '/install/lib/python310.zip',
    '/install/lib/python3.10',
    '/install/lib/lib-dynload',
  ]
error instantiating embedded Python interpreter: during initializing Python core: init_fs_encoding: failed to get the Python codec of the filesystem encoding
error: cargo run failed

[indygreg]

That init_fs_encoding: failed to get the Python codec of the filesystem encoding is the error message that Python spews when it tries to import some low-level stdlib module during startup but it can't find the files. This error is described at https://pyoxidizer.readthedocs.io/en/latest/pyoxidizer_rust_generic_embedding.html#embed-python-with-pyo3.

We should not see this issue with PyOxidizer unless the pyoxidizer.bzl is somehow doing something that would cause it.

Do you have a pyoxidizer.bzl we could look it?

And since this looks like a PyOxidizer issue, I'm going to attempt to transfer the issue there...

[Cellebyte]

@indygreg as mentioned above {ups will update the reproduce steps} I just created a init config file with the latest PyOxidizer release and changed python_version to version 3.10

[Cellebyte]

@indygreg I updated the instructions so that you can reproduce it.

[indygreg]

This might be due to PyO3/pyo3#2370. The Rust struct representing PyConfig is missing a field. So all offsets after the missing field are wrong, leading to a corrupted struct from the perspective of CPython. This can result in crashes or wrong behavior.

[ThatNerdSquared]

Bumping this — seems like the relevant PyO3 bug has been fixed, if I've been following the right PRs? Would love to switch to PyOxidizer from PyInstaller as can be seen from my referenced commit above, but the broken 3.10 builds are holding me back from the switch.

Thanks for your work on PyOxidizer, it looks great from what I've played with so far 👍

[Cellebyte]

@ThatNerdSquared I think you need to wait for the next release or build PyOxidizer from the main branch

[Cellebyte]

Release for
PyOxidizer v0.22.0 fixes it.