Support PEP 405 include-system-site-packages configuration

[pradyunsg]

Closes #5702
Supercedes and closes #7149

Basically, pip was never updated to also look at PEP 405's system-site-packages mechanism.

That's what caused a fairly widespread "bad" bug. :(

[pradyunsg]

Ahahahah.

This breaks our tests, since they were installing in --user because of this bug. :)

[chrahunt]

To be clear, this is specifically to fix a bug where the expected pip behavior of "aborting if --user is provided while using a virtual environment" doesn't work under venv (only virtualenv).

[chrahunt]

I don't think this closes #6984. For that I think we just need to provide clear indication of where to take that discussion - I think #4575?

[pradyunsg]

For addressing review comments from the GitHub UI.

[pradyunsg]

I don't think this closes #6984

You're right, this probably doesn't do that. Amended.

FWIW, IIUC, that PR is suggesting a behavior change from "aborting if --user is provided while using a virtual environment" to "ignore --user while using a virtual environment".

[pradyunsg]

sigh

Our tests setup for venv is a lot more broken than I'd like. The following test fails with you run tox -e py37 -- -k test_ensure_script_isolates_environment --use-venv. Dropping the --use-venv test, makes the tests pass, implying that --use-venv does not isolate tests properly. :(

def test_ensure_script_isolates_environment(script):
    """
    Test that sys.path when using `script`, does not reach into project directory.

    This catches issues with nox and tox environments being used to run tests
    and to ensure that they aren't being made available directly by `script`.
    """
    result = script.run("python", "-c", "import sys\nprint(sys.path)")
    got = ast.literal_eval(result.stdout)

    problematic_locations = [
        location
        for location in got
        if location and os.path.realpath(location).startswith(SRC_DIR)
    ]
    assert not problematic_locations

[pradyunsg]

This change already missed one release window because our test suite was sub-par for dealing with venv.

I'm inclined to merge this PR in, after adding skip markers to the failing tests (skip when using --use-venv), since the underlying problem is with our test suite setup and not the code change.

[pradyunsg]

Oh, I missed a test.

[xavfernandez]

As mentionned in #7270 we'd likely want to add some test running with --use-venv in our CI.

[pradyunsg]

We do -- Travis CI and Appveyor run with --use-venv.

[pradyunsg]

Merging since there's an approval and the only build failing is RTD's new preview hooks that we've opted-in for testing.