pex vendoring does not (always) isolate itself from a colliding setuptools install in site-packages #1031
Comments
|
As requested: @stuhood |
|
Thanks! I repro the issue, with or without |
|
According to pypa/setuptools#1167, this is likely a collision between multiple So I suspect that this might be bug in the vendoring that Two workarounds:
|
stuhood
changed the title
databricksruntime/python-virtualenv based image.
|
This backtrace is in a Pex execution of Pip. There, Pex dogfoods itself to run a Pip PEX in isolation - with the exception of setuptools and wheel whose vendored versions are deliberately exposed so Pip can execute So that's where to start spelunking. |
|
I'll dig on this a bit. |
|
Getting back to the conversation above, I wanted to correct the record on:
Pex isolates itself whether run from a pex file or not. It does this by:
My answer above about Pip PEX dogfood concerns point 2. |
|
FWIW, I tested that both 1) and 2) from my answer resolved the issue in the repro. I might have rushed things though. |
|
Aha, ok. That's not making sense but good to know. That said, I don't repro: And the pex seems ok: @jperkelens or @stuhood does the original bug report still repro for you? Probably most importantly, does it repro after appropriate |
|
@jsirois : Yea, I repro with the exact commands from the description via: docker image prune
docker run \
-v "$(pwd):/workdir" \
-w '/workdir' \
--rm \
-it \
"databricksruntime/python-virtualenv:latest" \
/bin/bash
# ... run each of the commands from the descriptionThe docker-image-prune reports zero bytes pruned though, so I might be on a different copy of the EDIT: @jsirois : but also, FWIW, @jperkelens was able to resolve his issue by removing the |
|
If docker output says
They have not changed the image for 13 months, so that's unlikely it, but the image incluides a
Yeah - thanks. I'd still like to squash the issue if possible. |
It wasn't... all of the steps ran. Probably (at least) because I was actually copy pasting them into a live shell one by one, rather than running them via a Dockerfile. |
|
Hrm. Not being able to repro is ... maddening. If either of you have time could you run this on your image substituting your image id for |
As mentioned above, I have not been using a |
|
Thanks - the means by which the command are run in the image - by hand or by Dockefile RUN, are immaterial, so the listing is useful. But now I see you ran this against the base image and not the commands you subsequently ran - gotcha. Yeah, not so useful. Thanks - I'll try to debug this by other means. |
|
OK, so focusing on I can instead construct PYTHONPATH from the paths of the vendored setuptools and wheel directly, the main difference being this will pre-pend those paths to the sys.path of spawned Pip subprocesses. Right now they come in the order they are in sys.path which may put |
|
Noting the abive fix does not solve the general hermeticity problem here, just the specific problem for the Pip PEX used at PEX buildtime. For example: Here the
In this particular builtime case, we're burned by |
|
It looks like one way to solve this is to invert how we calculate which That solves the case above and seems as robust as you can possibly be since the answer is delegated to the python interpreter in question. |
|
Checking back in here. Although I still cannot repro the failure you get using the commands above I can repro the leak by inspection of sys.path in a PEX built in the databaricks image. I've done some work over in #1128 that makes Pex both aware of virtual environments and equips it with the capability to bust out of a virtual environment. I'll use that to fix this. The approach above using ~ |
This fixes binary canonicalization to handle virtual environments created with virtualenv instead of pyvenv. It also adds support for resolving the base interpreter used to build a virtual environment. The ability to resolve a virtual environment intepreter will be used to fix pex-tool#1031 where virtual environments created with `--system-site-packages` leak those packages through as regular sys.path entries otherwise undetectable by PEX. Work towards pex-tool#962 and pex-tool#1115.
Virtual environments created with --system-site-packages can break PEX sys.path scrubbing. Avoid this class of environment leaks by breaking out of venvs unless PEX_INHERIT_PATH has been requested. A test is added that fails for virtualenv-16.7.10 that fails without the fix. Fixes pex-tool#1031
Virtual environments created with --system-site-packages can break PEX sys.path scrubbing. Avoid this class of environment leaks by breaking out of venvs unless PEX_INHERIT_PATH has been requested. A test is added that fails for virtualenv-16.7.10 that fails without the fix. Fixes pex-tool#1031
This fixes binary canonicalization to handle virtual environments created with virtualenv instead of pyvenv. It also adds support for resolving the base interpreter used to build a virtual environment. The ability to resolve a virtual environment intepreter will be used to fix #1031 where virtual environments created with `--system-site-packages` leak those packages through as regular sys.path entries otherwise undetectable by PEX. Work towards #962 and #1115.
Virtual environments created with --system-site-packages can break PEX sys.path scrubbing. Avoid this class of environment leaks by breaking out of venvs unless PEX_INHERIT_PATH has been requested. A test is added that fails for virtualenv-16.7.10 that fails without the fix. Fixes pex-tool#1031
Virtual environments created with --system-site-packages can break PEX sys.path scrubbing. Avoid this class of environment leaks by breaking out of venvs unless PEX_INHERIT_PATH has been requested. A test is added that fails for virtualenv-16.7.10 without the fix. Fixes #1031
Having issues creating pex files in an image based on
databricksruntime/python-virtualenv.Dockerfile:
Command run in container:
pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3Portion of failure StackTrace:
The text was updated successfully, but these errors were encountered: