add requests_ca_bundle to settable_env_vars

[thamenato]

Problem

If the user has a proxy set with http_proxy and https_proxy and the proxy uses a self-signed certificate you might still get invalid SSL Cert errors when the subprocess tries to get the packages externally.

  Complete output (10 lines):
  Looking in indexes: https://pypi.org/simple/
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /simple/setuptools/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /simple/setuptools/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /simple/setuptools/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /simple/setuptools/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /simple/setuptools/
  Could not fetch URL https://pypi.org/simple/setuptools/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/setuptools/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)) - skipping
  ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
  ERROR: No matching distribution found for setuptools>=41.0
  Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)) - skipping

Solution

Allow the user to pass the REQUESTS_CA_BUNDLE environment variable that points to the self-signed certificate.

[coveralls]

Coverage remained the same at 0.0% when pulling c37ee09 on thamenato:ssl-to-settable-env-vars into 923f461 on pantsbuild:master.

[jsirois]

Thanks @thamenato.

[benjyw]

Thanks for the fix @thamenato !

[benjyw]

@thamenato Can you clarify which process needed this env var set?

[jsirois]

I assumed it had to be all PEX creation processes since those are the ones that use Pip which uses requests to do index scans and downloads.

[benjyw]

That's what I was imagining, but then why didn't this suffice? #10837

[jsirois]

Hrm.

I just tested Pex directly on the CLI and using a relative path for --cert (as #10837 does) works; i:e: it gets plumbed through to the Pip command as revealed with -vvv and if the --cert passed does not exist or exists but is bogus (I used a copy of /etc/hosts), Pip fails.

@thamenato can provide more detailed output in a new issue? Perhaps modify the previously failing Pants command to use: ./pants -ldebug --pex-verbosity=9 ....

[jsirois]

Aha! Plugins. @thamenato I'm guessing you have "setuptools>=41.0" in your plugins list in pants.toml.

[jsirois]

Fix is here: #10910

[jsirois]

#10910 was not the fix for this issue, but with @thamenato's help this is getting narrowed down over here: pex-tool/pex#1058