Add a note about pyenv in README and the guide

[kngwyu]

Closes #762 and #742

When using pyo3 with pyenv, env PYTHON_CONFIGURE_OPTS="--enable-shared" is required.

[kngwyu]

But... how should we do when we want to use a static link?

[kngwyu]

Here's my experiments note(on Arch Linux):

1. With pyenv install 3.7.6
   I got

---- buffer::test::test_array_buffer stdout ----
thread 'buffer::test::test_array_buffer' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr { type: Py(0x564b5afc12e0, PhantomData) }', src/buffer.rs:712:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- exceptions::test::test_check_exception stdout ----
thread 'exceptions::test::test_check_exception' panicked at 'Can not import module: socket: PyErr { type: Py(0x564b5afc12e0, PhantomData) }', src/exceptions.rs:360:5


failures:
    buffer::test::test_array_buffer
    **exceptions::test::test_check_exception**

2. With env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.7.6
   I got

/home/yu/Programs/PyO3/target/debug/deps/pyo3-1aaccc8eaf7dd9d3: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
error: test failed, to rerun pass '--lib'

🤔

[kngwyu]

@Alexander-N
Thank you for suggestions!

[konstin]

Is this necessary on all platforms or only on mac? I'm using pyenv on ubuntu and I do not remember setting that option.

[kngwyu]

I don't have a macOS machine, but weirdly, Ubuntu 18.04 + Python 3.7.6(from pyenv) works but Arch Linux + Python 3.7.6 doesn't work.
There's certainly some OS-specific issue, but I still haven't detect it 😰

[konstin]

I guess in this it's best to just document this as "you might --enable--shared on some linux distributions"

[davidhewitt]

FWIW I took a quick look at this today, and I think it might be a general issue when linking against a static Python lib, not just pyenv.

As far as I can tell, when you link an executable on unix against a static Python, only the symbols which the executable needs are included. But to run arbitrary python code when embedding the python interpreter in pyo3, all the Python interpreter symbols need to be in the final executable.

Because the unused symbols are stripped out, this leads to the kind of failures seen on #742 and #762 when loading some other .so that needs the missing symbol fails.

The correct fix might be we might have to disable certain functionality when linking statically. Or maybe there is a way to force all the symbols from libpython to be included in the final executable?

I've asked at https://users.rust-lang.org/t/embed-whole-python-interpreter-using-static-linking/42509 , no response yet...

[nfwvogt]

On the topic of this being a general issue when linking against a static Python lib, I think I see the same issue with a conda environment on linux. ("Linux Mint 19.3", conda version : 4.8.2, conda-build version : 3.18.9, python 3.7.4.final.0)
Normal python modules can be imported just fine, but everything with a compiled component panics (including numpy).

I could not get any combination of force_load options to work for static linking.
However as far as I understand this anaconda discussion it is built without --enable-shared but ships with a shared libpython.so anyway.

It seems I could get it to work using that libpython.so.
When overwriting config.shared in the pyo3 to true even though Py_ENABLE_SHARED == False a minimal test builds fine and can import numpy after setting the LD_LIBRARY_PATH to the anaconda directory.

Not sure if that workaround is known or obvious, but at least I couldn't find it anywhere.

[Hizoul]

When overwriting config.shared in the pyo3 to true even though Py_ENABLE_SHARED == False a minimal test builds fine and can import numpy after setting the LD_LIBRARY_PATH to the anaconda directory.

Can confirm this is true. It even loads tensorflow so definitely a "working" solution.

Not sure if that workaround is known or obvious, but at least I couldn't find it anywhere.

I didn't know it and it's not obvious, so thank you very much for sharing this!
Pyo3 always runs instantly on my usual setups. But I had to setup my code on a system where I don't have root priviliges and outdated python as well as outdated pythonlibs so I was forced to use conda.
Even when running conda activate myenv with everything installed I still had to do the following to get everything to work:

1. Create libpython.so in the miniconda/lib directory because it was missing ln -s libpython3.8.so.1.0 libpython.so
2. Set environment variables so pyo3 uses condas python stuff and not conflicting already installed outdated system libraries

export PYTHON_SYS_EXECUTABLE=python3.8
export PYO3_CROSS_INCLUDE_DIR=$CONDA_PREFIX/include/python3.8
export PYO3_CROSS_LIB_DIR=$CONDA_PREFIX/lib
export PYTHONPATH=$CONDA_PREFIX/lib/python3.8

3. apply the build.rs workaround in pyo3/build.rs suggested by @nfwvogt

[aviramha]

is this still relevant?

[davidhewitt]

I don't think we will merge this PR in this form. However, there is definitely scope to have a new section in the guide about "environment setup" which describes how to install Rust and Python for new users on a range of OSes, and macOS users would benefit from having this information about pyenv in that section.

[aviramha]

Should we maybe close this then and create a new issue?

[davidhewitt]

Created #2017