[BUGFIX] Pin the same pip version as core st2

[cognifloyd]

The pack CI is currently pinning pip>=9.0,<9.1 in the Makefile and the dependencies script, both of which are used for pack CI (it is also pinned in the circleci config for this repo).

But, core st2 has pip==20.0.2 pinned in its Makefile, which is also used by the dependencies script for pack CI.

# Pin common pip version here across all the targets
# Note! Periodic maintenance pip upgrades are required to be up-to-date with the latest pip security fixes and updates
PIP_VERSION ?= 20.0.2

And, to make the pinned versions even more problematic, virtualenv is now installing an even newer version of pip just before we try to install the st2 requirements, and that newer pip version (20.3) uses the new pip resolver which can't handle st2 requirements. This is the output of virtualenv in a recent test run. Note how pip-9.0.3 gets installed and then virtualenv promptly installs pip-20.3.3.

Successfully installed distlib-0.3.1 pip-9.0.3 setuptools-44.1.1 virtualenv-20.4.0
created virtual environment CPython2.7.18.final.0-64 in 512ms
  creator CPython2Posix(dest=/home/circleci/virtualenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/home/circleci/.local/share/virtualenv)
    added seed packages: pip==20.3.3, setuptools==44.1.1, wheel==0.36.2
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
Installing StackStorm requirements from /tmp/st2/requirements.txt

So, the CI is flapping back and forth between pip-9.0*, pip-20.0*, and pip-20.3* which makes it very difficult to debug.

This PR fixes all that by using the same pinned version of pip that st2 has pinned. It also makes sure the correct version of pip is installed after the virtualenv is built but before the st2 requirements get installed.

[cognifloyd]

To clarify: dependencies is called before running pack tests during each pack's CI jobs with something like this (in .circleci/config.yml):

      - run:
          name: Download dependencies
          shell: /bin/bash
          command: |
            git clone -b master git://github.com/stackstorm-exchange/ci.git ~/ci
            ~/ci/.circle/dependencies

Then, dependencies script does approximately this:

1. install the pinned version of pip
2. build the virtualenv
3. installs the st2 requirements.
4. copy the Makefile into the pack
5. use the Makefile to install
   • pip (again)
   • runner requirements
   • default CI requirements (like mock)

So, I change the pinned version and install pip again after building the virtualenv, but this time IN the virtualenv.

+PIP_DEP="pip==20.0.2"
-sudo pip install -U "pip>=9.0,<9.1" setuptools virtualenv
+sudo pip install -U "${PIP_DEP}" setuptools virtualenv
 
 virtualenv ~/virtualenv
 source ~/virtualenv/bin/activate
 
+# virtualenv is updating pip. Revert back to our pinned version.
+pip install -U "${PIP_DEP}"
+pip --version
+
 # Copy over Makefile and  install StackStorm runners and register metrics drivers
 echo "Installing StackStorm runners and registering metrics drivers"
 
 if [[ -n "${ROOT_DIR}" ]]; then
     PACK_REQUIREMENTS_FILE="${ROOT_DIR}/requirements.txt"
     PACK_TESTS_REQUIREMENTS_FILE="${ROOT_DIR}/requirements-tests.txt"
 
     echo "Copying Makefile to ${ROOT_DIR}"
     cp ~/ci/.circle/Makefile ${ROOT_DIR}
     make -C requirements-ci .install-runners

[blag]

The --no-download flag to virtualenv might help, of if you create the virtualenv with Python 3 you might be able to use --without-pip (eg: python3 -m venv --without pip virtualenv), which does not install or upgrade pip in the virtual environment (since you should be able to use python -m pip instead of pip itself).

[cognifloyd]

hmm. Cool. I didn't think of looking up virtualenv flags.

Happily, I don't think that flag is restricted to python3.

Plus, looking at the two versions of virtualenv mentioned in this build I can also use --pip to specify which version it should use.
https://virtualenv.pypa.io/en/20.0.18/cli_interface.html#section-seeder
https://virtualenv.pypa.io/en/20.4.0/cli_interface.html#section-seeder

[cognifloyd]

OK. Here are a couple more failing builds:
https://app.circleci.com/pipelines/github/StackStorm-Exchange/stackstorm-activecampaign/53/workflows/303a76e2-5f14-420d-9246-4d277340cd98/jobs/311
https://app.circleci.com/pipelines/github/StackStorm-Exchange/stackstorm-acos/49/workflows/fefde9bb-567a-4abf-903f-728cce299283/jobs/284

At the end, you see pip complaining:

error: cryptography 2.8 is installed but cryptography>=3.2 is required by set(['pyOpenSSL'])

Looking farther up in the logs you see error messages coming from pip 20.3.3, which is far newer than we have tested with:

ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.
pyopenssl 20.0.1 requires cryptography>=3.2, but you'll have cryptography 2.8 which is incompatible.
requests 2.23.0 requires chardet<4,>=3.0.2, but you'll have chardet 4.0.0 which is incompatible.
futurist 1.10.0 requires PrettyTable<0.8,>=0.7.1, but you'll have prettytable 1.0.1 which is incompatible.
Successfully installed Babel-2.9.0 MarkupSafe-1.1.1 RandomWords-0.3.0 amqp-2.5.2 apscheduler-3.6.3 argcomplete-1.12.2 argparse-1.4.0 backports.functools-lru-cache-1.6.1 bcrypt-3.1.7 beautifulsoup4-4.9.3 certifi-2020.12.5 cffi-1.14.4 chardet-4.0.0 click-7.1.2 colorama-0.4.4 configparser-4.0.2 contextlib2-0.6.0.post1 cryptography-2.8 debtcollector-1.22.0 decorator-4.4.2 dnspython-1.16.0 enum34-1.1.10 eventlet-0.25.1 fasteners-0.16 flex-6.14.0 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.3.0 futurist-1.10.0 gitdb2-2.0.6 gitpython-2.1.15 greenlet-0.4.15 gunicorn-19.9.0 httplib2-0.18.1 idna-2.10 importlib-metadata-2.1.1 importlib-resources-3.3.1 ipaddr-2.2.0 ipaddress-1.0.23 iso8601-0.1.13 jinja2-2.10.3 jsonpath-rw-1.4.0 jsonpointer-2.0 jsonschema-2.6.0 kazoo-2.8.0 kombu-4.6.6 linecache2-1.0.0 lockfile-0.12.2 logshipper-0.1.1.dev111 mock-2.0.0 mongoengine-0.18.2 monotonic-1.5 more-itertools-5.0.0 msgpack-1.0.2 netaddr-0.8.0 netifaces-0.10.9 networkx-1.11 nose-1.3.7 nose-parallel-0.3.1 nose-timer-0.7.5 ntlm-auth-1.5.0 orquesta-1.2.0 oslo.config-1.12.1 oslo.i18n-3.25.1 oslo.serialization-2.29.2 oslo.utils-3.37.0 paramiko-2.7.1 passlib-1.7.1 pathlib2-2.3.5 pbr-5.5.1 pika-1.1.0 ply-3.11 prettytable-1.0.1 prompt-toolkit-1.0.15 psutil-5.6.6 pyOpenSSL-20.0.1 pycparser-2.20 pyinotify-0.9.6 pymongo-3.10.0 pynacl-1.4.0 pyparsing-2.4.7 pyrabbit-1.1.0 python-dateutil-2.8.0 python-editor-1.0.4 python-gnupg-0.4.5 python-json-logger-0.1.11 python-statsd-2.1.0 pytz-2019.1 pywinrm-0.3.0 pyyaml-5.1.2 rednose-1.3.0 repoze.lru-0.7 requests-2.23.0 requests-ntlm-1.1.0 retrying-1.3.3 rfc3987-1.3.8 routes-2.4.1 scandir-1.10.0 semver-2.9.0 simplejson-3.17.2 singledispatch-3.4.0.3 six-1.13.0 smmap-3.0.5 smmap2-3.0.1 soupsieve-1.9.6 sseclient-py-1.7 st2-auth-backend-flat-file-0.1.0 stevedore-1.30.1 strict-rfc3339-0.7 tenacity-6.3.1 termstyle-0.1.11 tooz-1.66.1 traceback2-1.4.0 typing-3.7.4.3 tzlocal-2.1 ujson-1.35 unittest2-1.1.0 urllib3-1.25.11 validate-email-1.3 vine-1.3.0 voluptuous-0.12.1 waitress-1.4.4 wcwidth-0.2.5 webob-1.8.5 webtest-2.0.35 wrapt-1.12.1 xmltodict-0.12.0 yaql-1.1.3 zake-0.2.2 zipp-1.0.0
WARNING: You are using pip version 20.3.3; however, version 20.3.4 is available.

And finally, if you look farther up in the log, you see the culprit for the bad pip version where the virtualenv is built:

Successfully installed distlib-0.3.1 pip-9.0.3 setuptools-44.1.1 virtualenv-20.4.0
created virtual environment CPython2.7.18.final.0-64 in 377ms
  creator CPython2Posix(dest=/home/circleci/virtualenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/home/circleci/.local/share/virtualenv)
    added seed packages: pip==20.3.3, setuptools==44.1.1, wheel==0.36.2
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
Installing StackStorm requirements from /tmp/st2/requirements.txt

[cognifloyd]

I just added StackStorm-Exchange/stackstorm-acos@3f7dd93 to one of the PRs to see how this PR affects the dependencies setup:
✅ python3.6 tests still pass
❌ python2.7 tests are not passing

The warnings from pip 20.3.3 are gone, but something else is still messed up. Hmm.

[cognifloyd]

💡 Oh. I get it. --pip and --without-pip need the newer virtualenv, but we need to use an older virtualenv on py2.7. virtualenv>=20 is a complete rewrite and at least one indirect dep (pipenv) says it needs the older virtualenv on py2.7

Here's the error about pipenv needing an older virtualenv:

ERROR: poetry 1.0.5 has requirement virtualenv<17.0.0,>=16.7.9; python_version >= "2.7" and python_version < "2.8", but you'll have virtualenv 20.4.0 which is incompatible.

So, now we need to create handle virtualenv install + create separately for py27 and py3.

[cognifloyd]

OK. Lowering virtualenv version resolved the pipenv error, but something else is still broken.
https://app.circleci.com/pipelines/github/StackStorm-Exchange/stackstorm-acos/52/workflows/07b6c206-0cf1-4add-b357-0be898806be4/jobs/289

[cognifloyd]

I temporarily added a pyOpenSSL dep fix here. I'm submitting that change in StackStorm/st2#5131

Once we get that working and merged, then this PR should be good to go as well.

[cognifloyd]

I dropped python2 support and rebased on master. I think this is ready to merge.

[cognifloyd]

rebased on master

[Kami]

LGTM 👍