Test with Python 3.9 - look for deprecation

[terriko]

Python 3.9 is coming, and some of the things we're using specifically in the tests may have been deprecated. To do this, you'll probably want to install python 3.9 and set up a separate venv to test it. Report anythign found back into this issue.

[Nirali0029]

I will like to work on this.

[terriko]

@Nirali0029 Feel free! Let us know if you hit any snags.

[chazuttu]

Python 3.9 is the last version providing backward compatibility layers.
Opening GzipFile file for writing without specifying the mode argument is deprecated

[terriko]

@chazuttu Ooh, good find. I'll open a separate issue to get that fixed.

[Molkree]

I have tried to test on Ubuntu 20.04 with Python 3.9 in venv (in WSL :D) and encountered some problems. When doing pip install -r requirements.txt pycares fails to install. Dependency chain is cve-bin-tool -> aiohttp -> aiodns -> pycares.

Error

Collecting isort
  Using cached isort-5.8.0-py3-none-any.whl (103 kB)
Collecting rich
  Using cached rich-10.1.0-py3-none-any.whl (201 kB)
Collecting plotly
  Using cached plotly-4.14.3-py2.py3-none-any.whl (13.2 MB)
Collecting jinja2>=2.11.3
  Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
Collecting beautifulsoup4
  Using cached beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
Collecting aiohttp[speedups]>=3.7.4
  Using cached aiohttp-3.7.4.post0-cp39-cp39-manylinux2014_x86_64.whl (1.4 MB)
Requirement already satisfied: toml in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from -r requirements.txt (line 7)) (0.10.2)
Collecting pyyaml>=5.4
  Using cached PyYAML-5.4.1-cp39-cp39-manylinux1_x86_64.whl (630 kB)
Collecting jsonschema>=3.0.2
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Requirement already satisfied: py>=1.10.0 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from -r requirements.txt (line 10)) (1.10.0)
Requirement already satisfied: pytest in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from -r requirements.txt (line 11)) (6.2.3)
Collecting pytest-xdist
  Using cached pytest_xdist-2.2.1-py3-none-any.whl (37 kB)
Collecting pytest-cov
  Using cached pytest_cov-2.11.1-py2.py3-none-any.whl (20 kB)
Collecting pytest-asyncio
  Using cached pytest_asyncio-0.14.0-py3-none-any.whl (11 kB)
Collecting rpmfile>=1.0.6
  Using cached rpmfile-1.0.8-py3-none-any.whl (19 kB)
Collecting zstandard
  Using cached zstandard-0.15.2-cp39-cp39-manylinux2014_x86_64.whl (2.2 MB)
Requirement already satisfied: attrs>=17.3.0 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]>=3.7.4->-r requirements.txt (line 6)) (20.3.0)
Requirement already satisfied: typing-extensions>=3.6.5 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]>=3.7.4->-r requirements.txt (line 6)) (3.7.4.3)
Collecting chardet<5.0,>=2.0
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting async-timeout<4.0,>=3.0
  Using cached async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Requirement already satisfied: yarl<2.0,>=1.0 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]>=3.7.4->-r requirements.txt (line 6)) (1.6.3)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]>=3.7.4->-r requirements.txt (line 6)) (5.1.0)
Collecting aiodns
  Using cached aiodns-2.0.0-py2.py3-none-any.whl (4.8 kB)
Collecting cchardet
  Using cached cchardet-2.1.7-cp39-cp39-manylinux2010_x86_64.whl (265 kB)
Collecting brotlipy
  Using cached brotlipy-0.7.0-cp39-cp39-manylinux1_x86_64.whl (1.1 MB)
Collecting MarkupSafe>=0.23
  Using cached MarkupSafe-1.1.1-cp39-cp39-manylinux2010_x86_64.whl (32 kB)
Requirement already satisfied: setuptools in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from jsonschema>=3.0.2->-r requirements.txt (line 9)) (44.0.0)
Collecting pyrsistent>=0.14.0
  Using cached pyrsistent-0.17.3-cp39-cp39-linux_x86_64.whl
Requirement already satisfied: six>=1.11.0 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from jsonschema>=3.0.2->-r requirements.txt (line 9)) (1.15.0)
Requirement already satisfied: idna>=2.0 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from yarl<2.0,>=1.0->aiohttp[speedups]>=3.7.4->-r requirements.txt (line 6)) (3.1)
Collecting soupsieve>1.2
  Using cached soupsieve-2.2.1-py3-none-any.whl (33 kB)
Collecting retrying>=1.3.3
  Using cached retrying-1.3.3-py3-none-any.whl
Requirement already satisfied: pluggy<1.0.0a1,>=0.12 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from pytest->-r requirements.txt (line 11)) (0.13.1)
Requirement already satisfied: packaging in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from pytest->-r requirements.txt (line 11)) (20.9)
Requirement already satisfied: iniconfig in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from pytest->-r requirements.txt (line 11)) (1.1.1)
Collecting coverage>=5.2.1
  Using cached coverage-5.5-cp39-cp39-manylinux2010_x86_64.whl (243 kB)
Collecting execnet>=1.1
  Using cached execnet-1.8.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
  Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting apipkg>=1.4
  Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting commonmark<0.10.0,>=0.9.0
  Using cached commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
Collecting colorama<0.5.0,>=0.4.0
  Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting pygments<3.0.0,>=2.6.0
  Using cached Pygments-2.8.1-py3-none-any.whl (983 kB)
Collecting pycares>=3.0.0
  Using cached pycares-3.1.1.tar.gz (233 kB)
Requirement already satisfied: cffi>=1.5.0 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from pycares>=3.0.0->aiodns->aiohttp[speedups]>=3.7.4->-r requirements.txt (line 6)) (1.14.5)
Requirement already satisfied: pycparser in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from cffi>=1.5.0->pycares>=3.0.0->aiodns->aiohttp[speedups]>=3.7.4->-r requirements.txt (line 6)) (2.20)
Requirement already satisfied: pyparsing>=2.0.2 in /home/molkree/cve-bin-tool-venv-py3.9/lib/python3.9/site-packages (from packaging->pytest->-r requirements.txt (line 11)) (2.4.7)
Building wheels for collected packages: pycares
  Building wheel for pycares (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/molkree/cve-bin-tool-venv-py3.9/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3cyqzo9f/pycares_30ca30041fe448f5bee2049e2b879f8b/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3cyqzo9f/pycares_30ca30041fe448f5bee2049e2b879f8b/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-p48tuhwk
       cwd: /tmp/pip-install-3cyqzo9f/pycares_30ca30041fe448f5bee2049e2b879f8b/
  Complete output (22 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/pycares
  copying src/pycares/__main__.py -> build/lib.linux-x86_64-3.9/pycares
  copying src/pycares/utils.py -> build/lib.linux-x86_64-3.9/pycares
  copying src/pycares/_version.py -> build/lib.linux-x86_64-3.9/pycares
  copying src/pycares/errno.py -> build/lib.linux-x86_64-3.9/pycares
  copying src/pycares/__init__.py -> build/lib.linux-x86_64-3.9/pycares
  running build_ext
  generating cffi module 'build/temp.linux-x86_64-3.9/_cares.c'
  creating build/temp.linux-x86_64-3.9
  building '_cares' extension
  creating build/temp.linux-x86_64-3.9/build
  creating build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9
  creating build/temp.linux-x86_64-3.9/deps
  creating build/temp.linux-x86_64-3.9/deps/c-ares
  creating build/temp.linux-x86_64-3.9/deps/c-ares/src
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DHAVE_CONFIG_H=1 -D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64 -I/home/molkree/cve-bin-tool-venv-py3.9/include -I/usr/include/python3.9 -Ideps/c-ares/src -Ideps/c-ares/src/config_linux -c build/temp.linux-x86_64-3.9/_cares.c -o build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9/_cares.o
  error: [Errno 2] No such file or directory: 'x86_64-linux-gnu-gcc'
  ----------------------------------------
  ERROR: Failed building wheel for pycares
  Running setup.py clean for pycares
Failed to build pycares
Installing collected packages: pycares, chardet, async-timeout, apipkg, soupsieve, retrying, pytest-forked, pyrsistent, pygments, MarkupSafe, execnet, coverage, commonmark, colorama, cchardet, brotlipy, aiohttp, aiodns, zstandard, rpmfile, rich, pyyaml, pytest-xdist, pytest-cov, pytest-asyncio, plotly, jsonschema, jinja2, isort, beautifulsoup4
    Running setup.py install for pycares ... error
    ERROR: Command errored out with exit status 1:
     command: /home/molkree/cve-bin-tool-venv-py3.9/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3cyqzo9f/pycares_30ca30041fe448f5bee2049e2b879f8b/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3cyqzo9f/pycares_30ca30041fe448f5bee2049e2b879f8b/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-v6_wp3u9/install-record.txt --single-version-externally-managed --compile --install-headers /home/molkree/cve-bin-tool-venv-py3.9/include/site/python3.9/pycares
         cwd: /tmp/pip-install-3cyqzo9f/pycares_30ca30041fe448f5bee2049e2b879f8b/
    Complete output (22 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/pycares
    copying src/pycares/__main__.py -> build/lib.linux-x86_64-3.9/pycares
    copying src/pycares/utils.py -> build/lib.linux-x86_64-3.9/pycares
    copying src/pycares/_version.py -> build/lib.linux-x86_64-3.9/pycares
    copying src/pycares/errno.py -> build/lib.linux-x86_64-3.9/pycares
    copying src/pycares/__init__.py -> build/lib.linux-x86_64-3.9/pycares
    running build_ext
    generating cffi module 'build/temp.linux-x86_64-3.9/_cares.c'
    creating build/temp.linux-x86_64-3.9
    building '_cares' extension
    creating build/temp.linux-x86_64-3.9/build
    creating build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9
    creating build/temp.linux-x86_64-3.9/deps
    creating build/temp.linux-x86_64-3.9/deps/c-ares
    creating build/temp.linux-x86_64-3.9/deps/c-ares/src
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DHAVE_CONFIG_H=1 -D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64 -I/home/molkree/cve-bin-tool-venv-py3.9/include -I/usr/include/python3.9 -Ideps/c-ares/src -Ideps/c-ares/src/config_linux -c build/temp.linux-x86_64-3.9/_cares.c -o build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9/_cares.o
    error: [Errno 2] No such file or directory: 'x86_64-linux-gnu-gcc'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/molkree/cve-bin-tool-venv-py3.9/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3cyqzo9f/pycares_30ca30041fe448f5bee2049e2b879f8b/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3cyqzo9f/pycares_30ca30041fe448f5bee2049e2b879f8b/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-v6_wp3u9/install-record.txt --single-version-externally-managed --compile --install-headers /home/molkree/cve-bin-tool-venv-py3.9/include/site/python3.9/pycares Check the logs for full command output.

There are other people facing this on Python 3.9 (one, two, three). My error is not exactly the same and most of the others were on macOS but I'm pretty sure it's Python 3.9 problem. pycares and aiodns have same maintainer and it looks like they no longer have time for those projects (pycares, aiodns), so it is unclear when this will be resolved.

It's possible that a relatively easy solution might be found (seems like macOS folks were able to fix it tinkering with c-ares) but I haven't looked into it.

[Molkree]

In a follow up to my previous message: installing gcc and libpython3.9-dev helped to build pycares properly and install cve-bin-tool. Note that I did not need them to install on Python 3.8 on the same machine.

All necessary bits are already present in GitHub-hosted runners so Python 3.9 should work in CI here. I've added it to the workflow (both Ubuntu and Windows) and it passed all tests (haven't run long tests though). See here.

P.S. Also tested on Python 3.10, Ubuntu tests fail inside pytest (should be fixed with next release) and Windows tests fail because something tries to install Pillow unsuccessfully 🤷‍♂️

[terriko]

I think with the addition of pyupgrade in #1147 that it's safe to close this. I expect to do another round of double checking before we start officially supporting 3.9 but that's not on the radar for the 2.1 release at this time.