Problem: pip problems when deploying SS with ansible

[mamedin]

Expected behaviour

The SS production pip requirements should be installed with no errors.

Current behaviour

When deploying SS with ansible I have the error at " Install pip requirements for production" steps:

Running setup.py install for brotli: started\n Running setup.py install for brotli: finished with status 'error'\n Complete output from command /usr/share/archivematica/virtualenvs/archivematica-storage-service/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-iRkl8D/brotli/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-8O_Y1W/install-record.txt --single-version-externally-managed --compile --install-headers /usr/share/archivematica/virtualenvs/archivematica-storage-service/include/site/python2.7/brotli:\n running install\n running build\n running build_ext\n Traceback (most recent call last):\n File "", line 1, in \n File "/tmp/pip-install-iRkl8D/brotli/setup.py", line 234, in \n 'test': TestCommand\n File "/usr/lib/python2.7/distutils/core.py", line 151, in setup\n dist.run_commands()\n File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands\n self.run_command(cmd)\n File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command\n cmd_obj.run()\n File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/local/lib/python2.7/site-packages/setuptools/command/install.py", line 61, in run\n return orig.install.run(self)\n File "/usr/lib/python2.7/distutils/command/install.py", line 601, in run\n self.run_command('build')\n File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command\n self.distribution.run_command(command)\n File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command\n cmd_obj.run()\n File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run\n self.run_command(cmd_name)\n File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command\n self.distribution.run_command(command)\n File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command\n cmd_obj.run()\n File "/usr/lib/python2.7/distutils/command/build_ext.py", line 340, in run\n self.build_extensions()\n File "/usr/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions\n self.build_extension(ext)\n File "/tmp/pip-install-iRkl8D/brotli/setup.py", line 94, in build_extension\n depends=ext.depends)\n File "/usr/lib/python2.7/distutils/ccompiler.py", line 566, in compile\n depends, extra_postargs)\n File "/usr/lib/python2.7/distutils/ccompiler.py", line 329, in _setup_compile\n raise TypeError, "'output_dir' must be a string or None"\n TypeError: 'output_dir' must be a string or None\n \n ----------------------------------------\n\n:stderr: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.\n Failed building wheel for brotli\noslo-config 6.8.0 has requirement requests>=2.18.0, but you'll have requests 2.14.2 which is incompatible.\nbotocore 1.9.23 has requirement python-dateutil<2.7.0,>=2.1, but you'll have python-dateutil 2.7.5 which is incompatible.\nCommand "/usr/share/archivematica/virtualenvs/archivematica-storage-service/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-iRkl8D/brotli/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-8O_Y1W/install-record.txt --single-version-externally-managed --compile --install-headers /usr/share/archivematica/virtualenvs/archivematica-storage-service/include/site/python2.7/brotli" failed with error code 1 in /tmp/pip-install-iRkl8D/brotli/\n"

Steps to reproduce

Deploy a new archivematica with ansible as described on deploy-pub or deploying standard storage service with deploy-priv.

Your environment (version of Archivematica, OS version, etc)

Tested with deploy-pub and local vagrant deploy for CentOS and Ubuntu.

Tested with a bionic deploy with deploy-private.

It doesn't fail on installed AM.

---

For Artefactual use:
Please make sure these steps are taken before moving this issue from Review to Verified in Waffle:

• All PRs related to this issue are properly linked 👍
• All PRs related to this issue have been merged 👍
• Test plan for this issue has been implemented and passed 👍
• Documentation regarding this issue has been written and it has been added to the release notes, if needed 👍

[mamedin]

Seems related to pip latest releases:

https://github.com/pypa/pip/releases

The ansible-archivematica-src role uses latest pip version: https://github.com/artefactual-labs/ansible-archivematica-src/blob/stable/1.8.x/tasks/common.yml#L15

[scollazo]

The problem is introduced by setuptools v40.7.0, released yesterday. After downgrading setuptools to 0.46.3, I was able to install brotli==0.5.2

[scollazo]

@sevein found the upstream bug: pypa/setuptools#1653

[scollazo]

There is a new version of setuptools that fixes this: https://github.com/pypa/setuptools/releases/tag/v40.7.1 , I was able to run a clean install of qa/1.x without problems