Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pex fails to consistently use https_proxy + --cert #1058

Closed
jsirois opened this issue Oct 5, 2020 · 6 comments · Fixed by #1060 or #1063
Closed

Pex fails to consistently use https_proxy + --cert #1058

jsirois opened this issue Oct 5, 2020 · 6 comments · Fixed by #1060 or #1063
Assignees
@jsirois
Labels
bug resolver

Comments

@jsirois
Copy link
Member

jsirois commented Oct 5, 2020

No self-contained repro yet, but a user reports the following:

In a container where the ambient REQUESTS_CA_BUNDLE points to a host-system self-signed cert authority:

$ docker run --rm -it -e HTTP_PROXY -e HTTPS_PROXY -v $REQUESTS_CA_BUNDLE:/ca-cert.crt -v $(pwd):/app python:3.7 bash
$ pip install --cert /ca-cert.crt pex
$ pex -vvvvvvvvv --cert /ca-cert.crt black setuptools
...
pex: Building pex :: Resolving distributions (['black', 'setuptools']) :: Resolving for:
pex: Spawning a maximum of 12 parallel jobs to process:r/local/bin/python3.7', PythonIdentity('/usr/local/bin/python', 'cp37', 'cp37m', 'manylinux2014_x86_64', (3, 7, 9))))
  DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.7', PythonIdentity('/usr/local/bin/python', 'cp37', 'cp37m', 'manylinux2014_x86_64', (3, 7, 9))))
pex: Hashing pex: 39.5ms
pex: Isolating pex: 97.5ms
pex:   Extracting pex to /root/.pex/isolated/a8ebd39cfba0316d0416132e488ffc14d28893b5: 97.0ms
pex: Vendoring packaging from VendorSpec(key='packaging', requirement='packaging==19.2', rewrite=True, constrain=True) @ /usr/local/lib/python3.7/site-packages/pex/vendor/_vendored/packaging
pex: Vendoring pyparsing from VendorSpec(key='packaging', requirement='packaging==19.2', rewrite=True, constrain=True) @ /usr/local/lib/python3.7/site-packages/pex/vendor/_vendored/packaging
pex: Vendoring pkg_resources from VendorSpec(key='setuptools', requirement='setuptools==42.0.2', rewrite=True, constrain=True) @ /usr/local/lib/python3.7/site-packages/pex/vendor/_vendored/setuptools
pex: Executing: /usr/local/bin/python3.7 -s -E /tmp/tmplte70n5r
pex: PEX.run invoking /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5 --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -vvv --cache-dir /root/.pex download --dest /tmp/tmpt3ltlh3v/usr.local.bin.python3.7 --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --cert /ca-cert.crt black setuptools
pex: Installed VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Installed VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='compiler', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='distribution_target', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='resolver', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='variables', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='executor', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='version', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_info', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='tracer', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='environment', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='testing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='compatibility', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='bootstrap', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='network_configuration', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='__main__', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='util', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_bootstrapper', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='requirements', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_builder', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='jobs', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='typing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pip', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter_constraints', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='finders', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='platforms', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='orderedset', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='common', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_warnings', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='dist_metadata', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='vendor', is_pkg=True, path='pex', prefix='_pex'), _Importable(module='third_party', is_pkg=True, path='pex', prefix='_pex')))
pex: pex.third_party.packaging imported via _Loader(module_name='pex.third_party.packaging', vendor_module_name='pex.vendor._vendored.packaging.packaging')
pex: pex.third_party.pyparsing imported via _Loader(module_name='pex.third_party.pyparsing', vendor_module_name='pex.vendor._vendored.packaging.pyparsing')
pex: pex.third_party.pkg_resources imported via _Loader(module_name='pex.third_party.pkg_resources', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources')
pex: pex.third_party.pkg_resources.extern imported via _Loader(module_name='pex.third_party.pkg_resources.extern', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern')
pex: pex.third_party.pkg_resources.extern.six imported via _Loader(module_name='pex.third_party.pkg_resources.extern.six', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.six')
pex: pex.third_party.pkg_resources.extern.six.moves imported via _Loader(module_name='pex.third_party.pkg_resources.extern.six.moves', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.six.moves')
pex: pex.third_party.pkg_resources.extern.packaging imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging')
pex: pex.third_party.pkg_resources.extern.packaging.version imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.version', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.version')
pex: pex.third_party.pkg_resources.extern.packaging.specifiers imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.specifiers', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.specifiers')
pex: pex.third_party.pkg_resources.extern.pyparsing imported via _Loader(module_name='pex.third_party.pkg_resources.extern.pyparsing', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.pyparsing')
pex: pex.third_party.pkg_resources.extern.six.moves.urllib imported via _Loader(module_name='pex.third_party.pkg_resources.extern.six.moves.urllib', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.six.moves.urllib')
pex: pex.third_party.pkg_resources.extern.packaging.requirements imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.requirements', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.requirements')
pex: pex.third_party.pkg_resources.extern.packaging.markers imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.markers', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.markers')
pex: pex.third_party.pkg_resources imported via _Loader(module_name='pex.third_party.pkg_resources', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources')
pex: pex.third_party.packaging.specifiers imported via _Loader(module_name='pex.third_party.packaging.specifiers', vendor_module_name='pex.vendor._vendored.packaging.packaging.specifiers')
pex: Using the current interpreter /usr/local/bin/python3.7 since no constraints have been specified and PYTHONPATH is not set.
pex: Selecting runtime interpreter: 0.1ms
pex: Found site-library: /usr/local/lib/python3.7/site-packages
pex: Found .pth file: /usr/local/lib/python3.7/site-packages/distutils-precedence.pth
pex: Not a tainted path element: /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap
pex: Not a tainted path element: /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5
pex: Not a tainted path element: /usr/local/lib/python37.zip
pex: Not a tainted path element: /usr/local/lib/python3.7
pex: Not a tainted path element: /usr/local/lib/python3.7/lib-dynload
pex: Tainted path element: /usr/local/lib/python3.7/site-packages
pex: Scrubbing from user site: /root/.local/lib/python3.7/site-packages
pex: Scrubbing from site-packages: /usr/local/lib/python3.7/site-packages
pex: Dropping _distutils_hack
pex: Dropping setuptools
pex: Dropping _distutils_hack.override
pex: Dropping setuptools._distutils
pex: Dropping distutils
pex: Dropping distutils.core
pex: Dropping distutils.debug
pex: Dropping distutils.errors
pex: Dropping distutils.dist
pex: Dropping distutils.fancy_getopt
pex: Dropping distutils.util
pex: Dropping distutils.dep_util
pex: Dropping distutils.spawn
pex: Dropping distutils.log
pex: Dropping distutils.py35compat
pex: Dropping distutils.cmd
pex: Dropping distutils.dir_util
pex: Dropping distutils.file_util
pex: Dropping distutils.archive_util
pex: Dropping distutils.config
pex: Dropping distutils.extension
pex: Dropping setuptools._deprecation_warning
pex: Dropping setuptools.version
pex: Dropping pkg_resources
pex: Dropping pkg_resources.extern
pex: Dropping pkg_resources._vendor
pex: Dropping pkg_resources._vendor.appdirs
pex: Dropping pkg_resources.extern.appdirs
pex: Dropping pkg_resources._vendor.packaging
pex: Dropping pkg_resources._vendor.packaging.__about__
pex: Dropping pkg_resources.extern.packaging
pex: Dropping pkg_resources.extern.packaging.version
pex: Dropping pkg_resources.extern.packaging._structures
pex: Dropping pkg_resources.extern.packaging._typing
pex: Dropping pkg_resources.extern.packaging.specifiers
pex: Dropping pkg_resources.extern.packaging._compat
pex: Dropping pkg_resources.extern.packaging.utils
pex: Dropping pkg_resources.extern.packaging.requirements
pex: Dropping pkg_resources._vendor.pyparsing
pex: Dropping pkg_resources.extern.pyparsing
pex: Dropping pkg_resources.extern.packaging.markers
pex: Dropping setuptools.extension
pex: Dropping setuptools.monkey
pex: Dropping distutils.filelist
pex: Dropping setuptools.dist
pex: Dropping distutils.version
pex: Dropping setuptools.extern
pex: Dropping setuptools._vendor
pex: Dropping setuptools._vendor.packaging
pex: Dropping setuptools._vendor.packaging.__about__
pex: Dropping setuptools.extern.packaging
pex: Dropping setuptools._vendor.ordered_set
pex: Dropping setuptools.extern.ordered_set
pex: Dropping setuptools.windows_support
pex: Dropping setuptools.config
pex: Dropping setuptools.extern.packaging.version
pex: Dropping setuptools.extern.packaging._structures
pex: Dropping setuptools.extern.packaging._typing
pex: Dropping setuptools.extern.packaging.specifiers
pex: Dropping setuptools.extern.packaging._compat
pex: Dropping setuptools.extern.packaging.utils
pex: Dropping setuptools.depends
pex: Dropping setuptools._imp
pex: Dropping setuptools.py34compat
pex: Dropping setuptools.msvc
pex: Dropping distutils.ccompiler
pex: Dropping distutils.sysconfig
pex: Dropping distutils.text_file
pex: New sys.path: ['/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload']
pex: E: tags for 'manylinux2014_x86_64' x PythonInterpreter('/usr/local/bin/python3.7', PythonIdentity('/usr/local/bin/python3.7', 'cp37', 'cp37m', 'manylinux2014_x86_64', (3, 7, 9))) -> frozenset({<cp37-cp37m-manylinux1_x86_64 @ 140329298470704>, <py30-none-manylinux2014_x86_64 @ 140329298487648>, <cp37-none-manylinux2010_x86_64 @ 140329298471584>, <py30-none-any @ 140329298488928>, <cp35-abi3-manylinux2014_x86_64 @ 140329298472544>, <py34-none-linux_x86_64 @ 140329298485888>, <cp36-abi3-linux_x86_64 @ 140329298472464>, <py35-none-any @ 140329298488528>, <py35-none-linux_x86_64 @ 140329298485328>, <py33-none-linux_x86_64 @ 140329298486448>, <py37-none-manylinux1_x86_64 @ 140329298483392>, <py35-none-manylinux2014_x86_64 @ 140329298484752>, <cp36-abi3-manylinux2014_x86_64 @ 140329298471984>, <cp37-none-any @ 140329298488208>, <py32-none-manylinux1_x86_64 @ 140329298486848>, <cp37-none-manylinux2014_x86_64 @ 140329298471424>, <py36-none-manylinux1_x86_64 @ 140329298484512>, <py30-none-linux_x86_64 @ 140329298488128>, <py34-none-manylinux1_x86_64 @ 140329298485728>, <cp37-abi3-manylinux1_x86_64 @ 140329298471184>, <cp36-abi3-manylinux1_x86_64 @ 140329298472304>, <cp35-abi3-manylinux1_x86_64 @ 140329298472864>, <cp37-none-linux_x86_64 @ 140329298471904>, <py32-none-any @ 140329298488768>, <py37-none-manylinux2014_x86_64 @ 140329298483072>, <cp37-cp37m-linux_x86_64 @ 140329298470384>, <cp34-abi3-linux_x86_64 @ 140329298481872>, <py32-none-manylinux2014_x86_64 @ 140329298486528>, <py37-none-any @ 140329298488288>, <cp37-abi3-manylinux2014_x86_64 @ 140329298470864>, <py31-none-linux_x86_64 @ 140329298487568>, <py31-none-any @ 140329298488848>, <py34-none-any @ 140329298488608>, <py31-none-manylinux2014_x86_64 @ 140329298487088>, <py34-none-manylinux2014_x86_64 @ 140329298485408>, <cp37-cp37m-manylinux2010_x86_64 @ 140329298470544>, <py36-none-linux_x86_64 @ 140329298484672>, <cp37-cp37m-manylinux2014_x86_64 @ 140329298389504>, <py30-none-manylinux1_x86_64 @ 140329298487968>, <py31-none-manylinux2010_x86_64 @ 140329298487248>, <py3-none-manylinux2010_x86_64 @ 140329298483792>, <py35-none-manylinux1_x86_64 @ 140329298485072>, <py33-none-manylinux2010_x86_64 @ 140329298486128>, <py36-none-any @ 140329298488448>, <cp33-abi3-manylinux1_x86_64 @ 140329298482272>, <py33-none-any @ 140329298488688>, <cp32-abi3-manylinux2010_x86_64 @ 140329298482672>, <cp37-none-manylinux1_x86_64 @ 140329298471744>, <py36-none-manylinux2014_x86_64 @ 140329298484192>, <py33-none-manylinux2014_x86_64 @ 140329298485968>, <cp34-abi3-manylinux1_x86_64 @ 140329298481712>, <py36-none-manylinux2010_x86_64 @ 140329298484352>, <cp37-abi3-linux_x86_64 @ 140329298471344>, <cp32-abi3-linux_x86_64 @ 140329298482992>, <cp35-abi3-linux_x86_64 @ 140329298481312>, <cp32-abi3-manylinux2014_x86_64 @ 140329298482512>, <cp36-abi3-manylinux2010_x86_64 @ 140329298472144>, <cp35-abi3-manylinux2010_x86_64 @ 140329298472704>, <py37-none-manylinux2010_x86_64 @ 140329298483232>, <py3-none-manylinux1_x86_64 @ 140329298483952>, <cp33-abi3-linux_x86_64 @ 140329298482432>, <cp34-abi3-manylinux2010_x86_64 @ 140329298481552>, <py32-none-manylinux2010_x86_64 @ 140329298486688>, <cp33-abi3-manylinux2014_x86_64 @ 140329298481952>, <py3-none-any @ 140329298488368>, <py34-none-manylinux2010_x86_64 @ 140329298485568>, <py3-none-manylinux2014_x86_64 @ 140329298483632>, <py32-none-linux_x86_64 @ 140329298487008>, <cp37-abi3-manylinux2010_x86_64 @ 140329298471024>, <cp33-abi3-manylinux2010_x86_64 @ 140329298482112>, <py37-none-linux_x86_64 @ 140329298483552>, <py33-none-manylinux1_x86_64 @ 140329298486288>, <cp34-abi3-manylinux2014_x86_64 @ 140329298481392>, <cp32-abi3-manylinux1_x86_64 @ 140329298482832>, <py30-none-manylinux2010_x86_64 @ 140329298487808>, <py3-none-linux_x86_64 @ 140329298484112>, <py35-none-manylinux2010_x86_64 @ 140329298484912>, <py31-none-manylinux1_x86_64 @ 140329298487408>})
pex: Activating PEX virtual environment from /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5: 10.8ms
pex:   Searching dependency cache: /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps: 1.4ms
pex:     Adding setuptools 42.0.2: 0.1ms
pex:     Adding wheel 0.33.6: 0.1ms
pex:     Adding pip 20.0.dev0: 0.1ms
pex:   Resolving pip from [Requirement.parse('pip==20.0.dev0')]: 2.0ms
pex:   Resolving setuptools from [Requirement.parse('setuptools==42.0.2')]: 2.7ms
pex:   Resolving wheel from [Requirement.parse('wheel==0.33.6')]: 1.9ms
pex:   Activating pip 20.0.dev0: 0.2ms
pex:     Adding sitedir: 0.1ms
pex:   Activating setuptools 42.0.2: 0.3ms
pex:     Adding sitedir: 0.2ms
pex:   Activating wheel 0.33.6: 0.3ms
pex:     Adding sitedir: 0.2ms
pex: Bootstrap complete, performing final sys.path modifications...
pex: Not uninstalling VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='compiler', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='distribution_target', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='resolver', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='variables', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='executor', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='version', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_info', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='tracer', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='environment', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='testing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='compatibility', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='bootstrap', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='network_configuration', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='__main__', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='util', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_bootstrapper', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='requirements', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_builder', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='jobs', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='typing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pip', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter_constraints', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='finders', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='platforms', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='orderedset', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='common', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_warnings', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='dist_metadata', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='vendor', is_pkg=True, path='pex', prefix='_pex'), _Importable(module='third_party', is_pkg=True, path='pex', prefix='_pex')))
pex: un-imported pex.third_party.packaging
pex: un-imported pex.third_party.packaging.specifiers
pex: un-imported pex.vendor._vendored.packaging.packaging
pex: un-imported pex.vendor._vendored.packaging.packaging.__about__
pex: un-imported pex.vendor._vendored.packaging.packaging._compat
pex: un-imported pex.vendor._vendored.packaging.packaging._structures
pex: un-imported pex.vendor._vendored.packaging.packaging.markers
pex: un-imported pex.vendor._vendored.packaging.packaging.specifiers
pex: un-imported pex.vendor._vendored.packaging.packaging.tags
pex: un-imported pex.vendor._vendored.packaging.packaging.version
pex: un-imported pex.third_party.pyparsing
pex: un-imported pex.vendor._vendored.packaging.pyparsing
pex: un-imported pex.third_party.pkg_resources
pex: un-imported pex.third_party.pkg_resources.extern
pex: un-imported pex.third_party.pkg_resources.extern.packaging
pex: un-imported pex.third_party.pkg_resources.extern.packaging.markers
pex: un-imported pex.third_party.pkg_resources.extern.packaging.requirements
pex: un-imported pex.third_party.pkg_resources.extern.packaging.specifiers
pex: un-imported pex.third_party.pkg_resources.extern.packaging.version
pex: un-imported pex.third_party.pkg_resources.extern.pyparsing
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources._vendor
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources._vendor.packaging.__about__
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.appdirs
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging._compat
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging._structures
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.markers
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.requirements
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.specifiers
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.version
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.pyparsing
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.py31compat
pex: Uninstalled VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Demoting code from Bootstrap(sys_path_entry='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap')
pex: un-imported <module 'pex.version' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/version.py'>
pex: un-imported <module 'pex.vendor._vendored.setuptools' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/_vendored/setuptools/__init__.py'>
pex: un-imported <module 'pex.vendor._vendored.packaging' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/_vendored/packaging/__init__.py'>
pex: un-imported <module 'pex.vendor._vendored' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/_vendored/__init__.py'>
pex: un-imported <module 'pex.vendor' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/__init__.py'>
pex: un-imported <module 'pex.variables' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/variables.py'>
pex: un-imported <module 'pex.util' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/util.py'>
pex: un-imported <module 'pex.typing' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/typing.py'>
pex: un-imported <module 'pex.tracer' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/tracer.py'>
pex: un-imported <module 'pex.third_party' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/third_party/__init__.py'>
pex: un-imported <module 'pex.platforms' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/platforms.py'>
pex: un-imported <module 'pex.pip' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pip.py'>
pex: un-imported <module 'pex.pex_warnings' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_warnings.py'>
pex: un-imported <module 'pex.pex_info' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_info.py'>
pex: un-imported <module 'pex.pex_builder' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_builder.py'>
pex: un-imported <module 'pex.pex_bootstrapper' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_bootstrapper.py'>
pex: un-imported <module 'pex.pex' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex.py'>
pex: un-imported <module 'pex.orderedset' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/orderedset.py'>
pex: un-imported <module 'pex.network_configuration' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/network_configuration.py'>
pex: un-imported <module 'pex.jobs' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/jobs.py'>
pex: un-imported <module 'pex.interpreter_constraints' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/interpreter_constraints.py'>
pex: un-imported <module 'pex.interpreter' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/interpreter.py'>
pex: un-imported <module 'pex.finders' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/finders.py'>
pex: un-imported <module 'pex.executor' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/executor.py'>
pex: un-imported <module 'pex.environment' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/environment.py'>
pex: un-imported <module 'pex.distribution_target' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/distribution_target.py'>
pex: un-imported <module 'pex.dist_metadata' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/dist_metadata.py'>
pex: un-imported <module 'pex.compiler' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/compiler.py'>
pex: un-imported <module 'pex.compatibility' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/compatibility.py'>
pex: un-imported <module 'pex.common' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/common.py'>
pex: un-imported <module 'pex.bootstrap' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/bootstrap.py'>
pex: un-imported <module 'pex' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/__init__.py'>
pex: Re-imported pex from ['/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex']
pex: PYTHONPATH contains:
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5
pex:   * /usr/local/lib/python37.zip
pex:     /usr/local/lib/python3.7
pex:     /usr/local/lib/python3.7/lib-dynload
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/setuptools
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/wheel
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Created temporary directory: /tmp/pip-req-tracker-iad7wt7d
Initialized build tracking at /tmp/pip-req-tracker-iad7wt7d
Created build tracker: /tmp/pip-req-tracker-iad7wt7d
Entered build tracker: /tmp/pip-req-tracker-iad7wt7d
Created temporary directory: /tmp/pip-download-42q1y0wc
1 location(s) to search for versions of black:
* https://pypi.org/simple/black/
Fetching project page and analyzing links: https://pypi.org/simple/black/
Getting page https://pypi.org/simple/black/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/black/" in the cache
No cache entry available
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/black/ HTTP/1.1" 200 4035
Updating cache with response from "https://pypi.org/simple/black/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/22/5a/ebe5881cbf07435ea1ad76119ee142f42fb7a333d562c202db015b959ae7/black-18.3a0-py3-none-any.whl#sha256=7183263650ba3071034e90b40a1ea74abccbd32cf525cef6d7914479dbe7f2fb (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a0
  Found link https://files.pythonhosted.org/packages/83/a2/fae3ddb516a52a0f37e6e2ef3bd789b6d093d2c7015a41a41bff30f39897/black-18.3a0.tar.gz#sha256=0461c7a52b5beb378936bf642753dec7a45305c96c6129d540b9c53227121a5a (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a0
  Found link https://files.pythonhosted.org/packages/61/b4/3a39dcf59342f53680c59a5e44c0b172ea5d6470a5aaba8ae3cdbfccc1e7/black-18.3a1-py3-none-any.whl#sha256=04edc87b51b34a7aa093f39bf207e2ebc3612b26f2c79fdf8ff13b2d60622c9a (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a1
  Found link https://files.pythonhosted.org/packages/1b/4d/ff18ef3018083196b85c843387dfa6a1faf376e8f662f74634bad84092d5/black-18.3a1.tar.gz#sha256=c9662827051587b732ecb4994a268f218eda896137b3ea6b8af8510682438b79 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a1
  Found link https://files.pythonhosted.org/packages/83/12/450fb640290d1499ae3922f95190b66b36918656c68376517a8477c6c4c9/black-18.3a2-py36-none-any.whl#sha256=e18daeede445c8d78da14734fa87015ba94ab3d2320792fdf31d95ff5a8eb8eb (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a2
  Found link https://files.pythonhosted.org/packages/f7/2b/359a321cb63d3b4a9aecc0d0fb87dd20023ed1a6bd1a9ab44fb1db91580a/black-18.3a2.tar.gz#sha256=2343980b35f425bfb33eebc6bddc4af57ef271c503d30de89f7812c3429a2bc6 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a2
  Found link https://files.pythonhosted.org/packages/3a/ae/3e4c6b429af58b100d3cbdd1b7c0ba0d3d8b74781761961b4ede32390810/black-18.3a3-py36-none-any.whl#sha256=4d21d41c23b8bf4b063ca57353414cba77f554d3360a7d538954c1de5b8df664 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a3
  Found link https://files.pythonhosted.org/packages/0f/69/f60a77390ae2182e8370fe669632812915599163853e55d04dec3ebed605/black-18.3a3.tar.gz#sha256=ee8f84ff1e06b5c262e3bad000eaa2a12b0071a4b2f269502114aeaefbdfb198 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a3
  Found link https://files.pythonhosted.org/packages/5e/c7/a1acc4f8f6baecacefabcf79e2165255f5605c7e33108aa58f16f4e55760/black-18.3a4-py36-none-any.whl#sha256=40c7d13615f9f183b5bab52afa4b3fa9eac62549149ba698185a5365db8efffc (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a4
  Found link https://files.pythonhosted.org/packages/f1/c2/a336bc9bf176d30d9824e587fbe6f921ee51d7ab5f67f28ad7c2ee52fc8c/black-18.3a4.tar.gz#sha256=814a1f5c9cf8dd8d325a224aa989be622fbaec9325ba13d884d9cf5485b47056 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a4
  Found link https://files.pythonhosted.org/packages/b0/3d/28412b6baff122e70a1e9e8c9fdf4f50b1a17b010006a0844c49a182ce1d/black-18.4a0-py36-none-any.whl#sha256=3d0b42275fc6362f60eeaecf9168a503085490be4fd780dffa48035271dc8908 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a0
  Found link https://files.pythonhosted.org/packages/26/fc/fe16644348a849e4171d311dd4a9a6f0f4e1ff62d5525091e0965b547d8b/black-18.4a0.tar.gz#sha256=390b3b3ec035dc2ca5e9df6303e6049d800d07b89d01d985db44ffca4976ae11 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a0
  Found link https://files.pythonhosted.org/packages/9e/0f/933f8016959805328371a25899ea124590345969aac8c4822306c7452ec8/black-18.4a1-py36-none-any.whl#sha256=375582bcb6b8345d10475a9298b1357e8002996c3b31df4eba42f5b541e60f96 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a1
  Found link https://files.pythonhosted.org/packages/6b/2a/908a33ca35eb6fb5d8a57351fba24727ba3ed30191035557f57fa6d9cb83/black-18.4a1.tar.gz#sha256=e9823469b247ed5e186c21f7e94c02aafd8d0905729a5b3748b555bcc2637e5b (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a1
  Found link https://files.pythonhosted.org/packages/e1/e6/dee648e0b1c7997eafa14e51911eeacbed026b2611605bd34d0f824ab6d7/black-18.4a2-py36-none-any.whl#sha256=d5cb273f9fc36ff89f07b84efe50a73d9177158162917198ed4263cd1c40d585 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a2
  Found link https://files.pythonhosted.org/packages/e5/3d/a2204bc1f2feaf5db0105db0f7956a591552ca9ee36f91dff37b08962866/black-18.4a2.tar.gz#sha256=cecee6dc0250661cf86dced28adddf3b65bb669636638745e488e7c1d718e8be (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a2
  Found link https://files.pythonhosted.org/packages/5e/62/58d43ed2e23d853461498132945b510601f0f2e4c3b0a4835bf3ba9b94d7/black-18.4a3-py36-none-any.whl#sha256=af80d5990a4af4bd8885847fef659d59c248d694ed7d2aa076ecaeabc6320006 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a3
  Found link https://files.pythonhosted.org/packages/d1/b0/9233ecee3573cc6e9c1fc257af3c90e4d80818f141ee16d7da4a397120f7/black-18.4a3.tar.gz#sha256=b45061e041762296b0cbf8f476a3036109571e73ce6f3e2d10e1ccc4d0827d59 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a3
  Found link https://files.pythonhosted.org/packages/db/c8/c587383e6b57a07b9937ac29ff93f845aa3d344aa9242f0eba13163eaa2b/black-18.4a4-py36-none-any.whl#sha256=dfb0e344587510ef662785f4a8991addb4017840459bb6d1d53d344ab0f10300 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a4
  Found link https://files.pythonhosted.org/packages/65/c1/3d5ad1f94bbc4b233dba82d2934f5a3b15a33ddaf08165cd0c27d8cfe34b/black-18.4a4.tar.gz#sha256=58594128efb5d21f359c8680911241985e627eb3505397f3982ffbae914d8202 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a4
  Found link https://files.pythonhosted.org/packages/c6/3d/0a966ead09d4f81f5887e5126c2da22d6a0efed89de4b4e9dc98ed727efc/black-18.5b0-py36-none-any.whl#sha256=5fec0f25486046b9edb97961c946412ced96021247dd1a60ecd9f0567b68b030 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b0
  Found link https://files.pythonhosted.org/packages/50/3d/887dcd61f361167438af8002cde23376c79439954bd37cbb732be8cf2ab3/black-18.5b0.tar.gz#sha256=4fec2566f9fbbd4a58de50a168cbe3ab952713530410d227e82e4c65d1fad946 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b0
  Found link https://files.pythonhosted.org/packages/27/f8/736d91bc142221f0505008059f2eb621e4a753a1236183dc96c0d81b52da/black-18.5b1-py36-none-any.whl#sha256=fa8d4b58aeb454443d4ef02c9daa615cd9a241f0cbe5f2b38b5965209d26dd1c (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b1
  Found link https://files.pythonhosted.org/packages/e5/0b/2b5abd975559a5e4cb5cf9697ca242878b4894dc731dd133e582ce459869/black-18.5b1.tar.gz#sha256=c956857a831e508e75f0dc754b86982bad98d95c96aae39c20749209b3958ad7 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b1
  Found link https://files.pythonhosted.org/packages/fe/48/d20ab5b038ac70d834b512db343c8fd4d172b35cd541e9544ea3501e3fa6/black-18.6b0-py36-none-any.whl#sha256=8a1220b97ee524e32dcab0e5703e82af96f73b91df86ca0d634fbc4cddde9b25 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b0
  Found link https://files.pythonhosted.org/packages/21/c3/b37ce7f074c6d10ff1fad31edce423502bf50830e74b7a93a2c0ace0d1a2/black-18.6b0.tar.gz#sha256=0e4a4e8b2cae2797b90e300b3e8fa2011557e2f697367ea8a85164b60a37a812 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b0
  Found link https://files.pythonhosted.org/packages/0e/8a/b902cfa966631ee02ecf76ca4466aee800766100a7dd7b2966cf143aa93f/black-18.6b1-py36-none-any.whl#sha256=80dbe2103e937de543942b7946fe71adc387506fe1922018e2b97555513a35fc (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b1
  Found link https://files.pythonhosted.org/packages/70/64/c8257484b57ed564e82170ceced2f8ea8fe57356a0801ffec5d162d19367/black-18.6b1.tar.gz#sha256=0c07b68fc6fc4df8b09873e81893d7b77d52794fa3431d8843b590bc33956105 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b1
  Found link https://files.pythonhosted.org/packages/f5/25/0badb111a8dd98216f55d60c0981d7c64d66e26fde67cc01944da285cb0b/black-18.6b2-py36-none-any.whl#sha256=3efe92eafbde15f8ac06478de11cfb84e47504896ccdde64507e751d2f91ec3a (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b2
  Found link https://files.pythonhosted.org/packages/2c/f1/4e63113f2ee5bd47640c4884ccbed07277d738602ea6d61e0971f527e05a/black-18.6b2.tar.gz#sha256=fc26c4ab28c541fb824f59fa83d5702f75829495d5a1dee603b29bc4fbe79095 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b2
  Found link https://files.pythonhosted.org/packages/c4/53/7a5b0ac836f128a83d837c5e39ee6eb48657b17082f6fe6e21d5fa80871d/black-18.6b3-py36-none-any.whl#sha256=479cc8b3455a75b5b289276b5f47fd2bb412f2f369292989c12b891264758b5c (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b3
  Found link https://files.pythonhosted.org/packages/28/db/fc2361255cea2010fd814edc44c50d118bfcdd031ae7c47ad15b01d3e38d/black-18.6b3.tar.gz#sha256=fe3b7ac846f2a7c91d926782184826c57d2be283c57f0d6b37b85496eb5469ff (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b3
  Found link https://files.pythonhosted.org/packages/4e/b5/575cdaabf1f98ebe50c0ee85b202fcce3b8df1f21d6ed02708bc539674c3/black-18.6b4-py36-none-any.whl#sha256=4b475bbd528acce094c503a3d2dbc2d05a4075f6d0ef7d9e7514518e14cc5191 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b4
  Found link https://files.pythonhosted.org/packages/2d/43/64205493cfb4c8e1720208d73502a121ace2a195cd4d1d49bd7470e4fd92/black-18.6b4.tar.gz#sha256=22158b89c1a6b4eb333a1e65e791a3f8b998cf3b11ae094adb2570f31f769a44 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b4
  Found link https://files.pythonhosted.org/packages/2a/34/9938749f260a861cdd8427d63899e08f9a2a041159a26c2615b02828c973/black-18.9b0-py36-none-any.whl#sha256=817243426042db1d36617910df579a54f1afd659adb96fc5032fcf4b36209739 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.9b0
  Found link https://files.pythonhosted.org/packages/e8/5f/0f79fcd943ba465cbd4bf303c9794970c13a95e5456630de9f72e7f37ad4/black-18.9b0.tar.gz#sha256=e030a9a28f542debc08acceb273f228ac422798e5215ba2a791a6ddeaaca22a5 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.9b0
  Found link https://files.pythonhosted.org/packages/30/62/cf549544a5fe990bbaeca21e9c419501b2de7a701ab0afb377bc81676600/black-19.3b0-py36-none-any.whl#sha256=09a9dcb7c46ed496a9850b76e4e825d6049ecd38b611f1224857a79bd985a8cf (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.3b0
  Found link https://files.pythonhosted.org/packages/89/07/aebb10fb8f2ffbac672dfbebffa724643bc84cf012a57737a622d1dabddb/black-19.3b0.tar.gz#sha256=68950ffd4d9169716bcb8719a56c07a2f4485354fec061cdd5910aa07369731c (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.3b0
  Found link https://files.pythonhosted.org/packages/fd/bb/ad34bbc93d1bea3de086d7c59e528d4a503ac8fe318bd1fa48605584c3d2/black-19.10b0-py36-none-any.whl#sha256=1b30e59be925fafc1ee4565e5e08abef6b03fe455102883820fe5ee2e4734e0b (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.10b0
  Found link https://files.pythonhosted.org/packages/b0/dc/ecd83b973fb7b82c34d828aad621a6e5865764d52375b8ac1d7a45e23c8d/black-19.10b0.tar.gz#sha256=c2edb73a08e9e0e6f65a0e6af18b059b8b1cdd5bef997d7a0b181df93dc81539 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.10b0
  Found link https://files.pythonhosted.org/packages/7d/b4/72fb7e851376edf60822318f42143f2984b42f2d69270a115acb6d0ed1e1/black-20.8b0.tar.gz#sha256=0fc6b2f00ccd34cc444d41f24967377c2bc6d0b53f333c56376405ff6cd6f789 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 20.8b0
  Found link https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 20.8b1
Given no hashes to check 42 links for project 'black': discarding no candidates
Using version 20.8b1 (newest of versions: 18.3a0, 18.3a1, 18.3a2, 18.3a3, 18.3a4, 18.4a0, 18.4a1, 18.4a2, 18.4a3, 18.4a4, 18.5b0, 18.5b1, 18.6b0, 18.6b1, 18.6b2, 18.6b3, 18.6b4, 18.9b0, 19.3b0, 19.10b0, 20.8b0, 20.8b1)
Collecting black
  Created temporary directory: /tmp/pip-unpack-imio_gce
  Looking up "https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz HTTP/1.1" 200 1096433
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 707 kB/s eta 0:00:01  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz"
  Caching due to etag
     |████████████████████████████████| 1.1 MB 707 kB/s
  Saved /tmp/tmpt3ltlh3v/usr.local.bin.python3.7/black-20.8b1.tar.gz
  Added black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea to build tracker '/tmp/pip-req-tracker-iad7wt7d'
  Created temporary directory: /tmp/pip-build-env-f3a0lj2f
  Running command /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f3a0lj2f/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
  Non-user install by explicit request
  Created temporary directory: /tmp/pip-ephem-wheel-cache-xosyyour
  Created build tracker: /tmp/pip-req-tracker-iad7wt7d
  Entered build tracker: /tmp/pip-req-tracker-iad7wt7d
  Created temporary directory: /tmp/pip-install-zbaj_ez2
  1 location(s) to search for versions of setuptools:
  * https://pypi.org/simple/setuptools/
  Fetching project page and analyzing links: https://pypi.org/simple/setuptools/
  Getting page https://pypi.org/simple/setuptools/
  Found index url https://pypi.org/simple
  Looking up "https://pypi.org/simple/setuptools/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (2): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=3, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (3): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (4): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (5): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (6): pypi.org:443
  Could not fetch URL https://pypi.org/simple/setuptools/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/setuptools/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))) - skipping
  Given no hashes to check 0 links for project 'setuptools': discarding no candidates
  ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
  Cleaning up...
  Removed build tracker: '/tmp/pip-req-tracker-iad7wt7d'
  ERROR: No matching distribution found for setuptools>=41.0
  Exception information:
  Traceback (most recent call last):
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/cli/base_command.py", line 172, in _main
      status = self.run(options, args)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/commands/install.py", line 384, in run
      resolver.resolve(requirement_set)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 185, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 341, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 289, in _get_abstract_dist_for
      req.populate_link(self.finder, upgrade_allowed, require_hashes)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/req/req_install.py", line 252, in populate_link
      self.link = finder.find_requirement(self, upgrade)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/index/package_finder.py", line 925, in find_requirement
      'No matching distribution found for %s' % req
  pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools>=41.0
  WARNING: You are using pip version 20.0.dev0; however, version 20.2.3 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.
  Installing build dependencies ... error
Removed black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea from build tracker '/tmp/pip-req-tracker-iad7wt7d'
Removed build tracker: '/tmp/pip-req-tracker-iad7wt7d'
ERROR: Command errored out with exit status 1: /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f3a0lj2f/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/cli/base_command.py", line 172, in _main
    status = self.run(options, args)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/commands/download.py", line 146, in run
    resolver.resolve(requirement_set)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 185, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 341, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 290, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/operations/prepare.py", line 544, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/operations/prepare.py", line 97, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/distributions/source.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/distributions/source.py", line 68, in _setup_isolation
    "Installing build dependencies"
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/build_env.py", line 201, in install_requirements
    call_subprocess(args, spinner=spinner)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f3a0lj2f/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
pid 64 -> /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5 --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -vvv --cache-dir /root/.pex download --dest /tmp/tmpt3ltlh3v/usr.local.bin.python3.7 --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --cert /ca-cert.crt black setuptools exited with 1 and STDERR:
None

Note that black is successfully lokked up and downloaded through the proxy on pypi, but then setuptools fails the same sequence immediately after.
@jsirois jsirois mentioned this issue Oct 5, 2020
Merged
@jsirois
Copy link
Member Author

jsirois commented Oct 5, 2020

An extra confusing variable is that when REQUESTS_CA_BUNDLE is exported instead of passing --cert, then both black and setuptools are looked up and downloaded successfully through the proxy.

@jsirois jsirois mentioned this issue Oct 5, 2020
Merged
@jsirois
Copy link
Member Author

jsirois commented Oct 5, 2020

Ok, this is a bug in Pip PEP-517 handling. Notice:
https://github.com/psf/black/blob/235412635e91950c8ef2d9ebe777f97fffd4f01d/pyproject.toml#L36-L38

In other words, black uses pyproject.toml and specifies a custom builder:

[build-system]
requires = ["setuptools>=41.0", "setuptools-scm", "wheel"]
build-backend = "setuptools.build_meta"

Its the execution of the custom builder that fails here in the prerequisite step of downloading its requirements. That seems to point to Pip not propagating its --cert setting to subprocesses.

@jsirois
Copy link
Member Author

jsirois commented Oct 5, 2020

And the Pip issue is here: pypa/pip#5502

@jsirois
Copy link
Member Author

jsirois commented Oct 5, 2020

OK, and it looks like we can convert --cert into a PIP_CERT env var to solve this: pypa/pip#5502 (comment)

@jsirois
Copy link
Member Author

jsirois commented Oct 5, 2020

Currently we set up --cert here in _calculate_package_index_options:
https://github.com/pantsbuild/pex/blob/0a29161e3b24a1b7a884ab3acd5f2f411bac7336/pex/pip.py#L173-L175

The assumption in that method is CLI args are returned based on network configuration. We'll probably need to split that method into _calculate_package_index_flags and _calculate_package_index_env where _calculate_package_index_env will yield - for now - this one new env var.

@jsirois jsirois mentioned this issue Oct 5, 2020
Closed
5 tasks
@jsirois jsirois self-assigned this Oct 5, 2020
jsirois added a commit to jsirois/pex that referenced this issue Oct 6, 2020
@jsirois
Fix --cert for PEP-518 builds.
e88df84 
Previously `--cert` did not propagate correctly through Pip. Work around
this by passing the option via environment variable.

Fixes pex-tool#1058
@jsirois jsirois mentioned this issue Oct 6, 2020
Merged
jsirois added a commit that referenced this issue Oct 6, 2020
@jsirois
Fix --cert for PEP-518 builds. (#1060)
61a7d8b 
Previously `--cert` did not propagate correctly through Pip. Work around
this by passing the option via environment variable.

Fixes #1058
@jsirois
Copy link
Member Author

jsirois commented Oct 8, 2020

As noted in #1060 this is still not completely fixed.

@jsirois jsirois reopened this Oct 8, 2020
jsirois added a commit to jsirois/pex that referenced this issue Oct 8, 2020
@jsirois
Fix handling of --cert and --client-cert.
9606c45 
In order to work around pypa/pip#5502 fully,
we extract a `PackageIndexConfiguration` class that handles arrangement
of Pip command line flags and enviornment variables to effect the most
hermetic means possible of conveying `--cert` and `--client-cert` to Pip
and its subprocesses.

Fixes pex-tool#1058
@jsirois jsirois mentioned this issue Oct 8, 2020
Merged
jsirois added a commit that referenced this issue Oct 8, 2020
@jsirois
Fix handling of --cert and --client-cert. (#1063)
4580055 
In order to work around pypa/pip#5502 fully,
we extract a `PackageIndexConfiguration` class that handles arrangement
of Pip command line flags and enviornment variables to effect the most
hermetic means possible of conveying `--cert` and `--client-cert` to Pip
and its subprocesses.

Fixes #1058
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsirois jsirois

Labels
bug resolver
Projects
None yet
Milestone
No milestone
1 participant
@jsirois