ffi: add PyConfig::warn_default_encoding to 3.10+

[indygreg]

This field was added in 3.10 but we missed it. This was causing
offsets of subsequent fields to be wrong. This could lead to
unexpected behavior or even crashes.

This bug is causing many PyOxidizer built binaries to crash when using 3.10.

[indygreg]

I confirmed that this patch fixes the crashes that occur with PyOxidizer when using 3.10.

[adamreichold]

I think it would be nice to mention the symptom - PyOxidizer crashes - here so that people experiencing them can match that to the changelog.

[davidhewitt]

Argh, I'm sorry we missed this. Given that 0.17 release is probably still a little way away (got a few unfinished bits merged), we should perhaps backport this to 0.16?

[davidhewitt]

A passing thought - would you be willing to add a new file test_pep_587 which tests spinning up an interpreter using these APIs? That should help avoid this happening again if anything changes in the future.

[indygreg]

Argh, I'm sorry we missed this. Given that 0.17 release is probably still a little way away (got a few unfinished bits merged), we should perhaps backport this to 0.16?

I would strongly prefer a backport and 0.16.x release at your earliest convenience.

A passing thought - would you be willing to add a new file test_pep_587 which tests spinning up an interpreter using these APIs? That should help avoid this happening again if anything changes in the future.

If it isn't too much work, yes. However, I'm unsure the state of your testing infra. These interpreter-level config options result in global variables getting set in CPython and CPython - especially older versions - isn't great about ensuring all values are normalized during interpreter initialization or finalization. For equivalent tests I already have in the pyembed crate, I need to run each test in its own process to prevent state leakage between tests. This gets thorny, especially on Windows. But if you already have an easy way to spin up new interpreters in new processes in tests, it shouldn't be difficult to implement test coverage.

[davidhewitt]

👍 I'll do my best to find time to push this out on Saturday.

But if you already have an easy way to spin up new interpreters in new processes in tests, it shouldn't be difficult to implement test coverage.

My understand was that cargo runs each test file from the tests/ subdirectory in a new process? If so, adding a new file with a single test should be sufficient. I was hoping you have something trivial from pyembed you might be able to strip down and repurpose as a test.

[indygreg]

👍 I'll do my best to find time to push this out on Saturday.

Thank you!

But if you already have an easy way to spin up new interpreters in new processes in tests, it shouldn't be difficult to implement test coverage.

My understand was that cargo runs each test file from the tests/ subdirectory in a new process? If so, adding a new file with a single test should be sufficient. I was hoping you have something trivial from pyembed you might be able to strip down and repurpose as a test.

Yes, I think that's how Cargo works. But I was hoping to avoid separate files because the repetition gets tedious. For pyembed, I use rusty_fork, which has a macro for spawning a test function in a new process. There are some rough edges getting shared libraries to all resolve properly in the context of the test process. But I have it working. See https://github.com/indygreg/PyOxidizer/blob/7b21c2b950b21e9d39002703c5dcc45a6280b73a/pyembed/src/test/interpreter_config.rs for my existing tests.

[indygreg]

It also looks like pathconfig_warnings and program_name have the wrong order on 3.10. I'll be amending this PR accordingly.

I'll likely also send a PR to add 3.11 changes to initconfig.h. But I don't expect or need this to be backported to 0.16.

[davidhewitt]

Thanks! I've managed this evening to push a regression test to this branch which should help ensure things keep working in the future (I could verify it failed on main and passes with this branch). I'll finish up and release in the morning.

[davidhewitt]

Aha perfect, the test now fails on 3.11 alpha which I guess is expected! For speed I'll just merge #2371 which has both your commits and then verify with a new MR for the test.