add script to find dependencies of Python packages #3839

Flamefire · 2021-09-16T12:57:18Z

Example usage with EasyBuild:

eb TensorFlow-2.3.4.eb --dump-env
source TensorFlow-2.3.4.env
findPythonDeps.py tensorflow==2.3.4

Or now: findPythonDeps.py tensorflow==2.3.4 --ec TensorFlow-2.3.4.eb

Example usage with EasyBuild: ``` eb TensorFlow-2.3.4.eb --dump-env source TensorFlow-2.3.4.env findPythonDeps.py tensorflow==2.3.4 ```

easybuild/scripts/findPythonDeps.py

Improves the error messages generated.

smoors · 2022-06-30T12:34:23Z

I just tested this with an existing easyconfig.
seems to work nicely, except for 2 dependency conflicts (see output below). I guess that's unavoidable, because updated versions of pythran and beniget have to be added manually?
however, strange that pip first says that I have gast 0.5.2, and later it says I have gast 0.4.0...

findPythonDeps.py --ec TensorFlow-2.7.1-foss-2021b.eb tensorflow==2.7.1
...
  Attempting uninstall: gast
    Found existing installation: gast 0.5.2
    Not uninstalling gast at /vscmnt/brussel_pixiu_apps/_apps_brussel/CO7/skylake/software/SciPy-bundle/2021.10-foss-2021b/lib/python3.9/site-packages, outside environment /tmp/tmp8a98rj97tensorflow-deps/venv
    Can't uninstall 'gast'. No files were found to uninstall.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pythran 0.10.0 requires gast~=0.5.0, but you have gast 0.4.0 which is incompatible.
beniget 0.4.1 requires gast~=0.5.0, but you have gast 0.4.0 which is incompatible.
Successfully installed astunparse-1.6.3 cachetools-5.2.0 gast-0.4.0 google-auth-2.9.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.47.0 keras-2.7.0 keras-preprocessing-1.1.2 libclang-14.0.1 markdown-3.
3.7 oauthlib-3.2.0 opt-einsum-3.3.0 pyasn1-modules-0.2.8 requests-oauthlib-1.3.1 rsa-4.8 tensorboard-2.9.1 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.7.1 tensorflow-estimator-2.7.0 t
ensorflow-io-gcs-filesystem-0.26.0 termcolor-1.1.0 werkzeug-2.1.2 wrapt-1.14.1
WARNING: You are using pip version 21.2.2; however, version 22.1.2 is available.
You should consider upgrading via the '/tmp/tmp8a98rj97tensorflow-deps/venv/bin/python -m pip install --upgrade pip' command.

List of dependencies in (likely) install order:
[   ('wrapt', '1.14.1'),
    ('termcolor', '1.1.0'),
    ('tensorflow-io-gcs-filesystem', '0.26.0'),
    ('tensorflow-estimator', '2.7.0'),
    ('Werkzeug', '2.1.2'),
    ('tensorboard-plugin-wit', '1.8.1'),
    ('tensorboard-data-server', '0.6.1'),
    ('Markdown', '3.3.7'),
    ('grpcio', '1.47.0'),
    ('oauthlib', '3.2.0'),
    ('requests-oauthlib', '1.3.1'),
    ('rsa', '4.8'),
    ('pyasn1-modules', '0.2.8'),
    ('cachetools', '5.2.0'),
    ('google-auth', '2.9.0'),
    ('google-auth-oauthlib', '0.4.6'),
    ('tensorboard', '2.9.1'),
    ('opt-einsum', '3.3.0'),
    ('libclang', '14.0.1'),
    ('Keras-Preprocessing', '1.1.2'),
    ('keras', '2.7.0'),
    ('google-pasta', '0.2.0'),
    ('astunparse', '1.6.3'),
    ('tensorflow', '2.7.1')]
Sorted list of dependencies:
[   ('Keras-Preprocessing', '1.1.2'),
    ('Markdown', '3.3.7'),
    ('Werkzeug', '2.1.2'),
    ('astunparse', '1.6.3'),
    ('cachetools', '5.2.0'),
    ('google-auth', '2.9.0'),
    ('google-auth-oauthlib', '0.4.6'),
    ('google-pasta', '0.2.0'),
    ('grpcio', '1.47.0'),
    ('keras', '2.7.0'),
    ('libclang', '14.0.1'),
    ('oauthlib', '3.2.0'),
    ('opt-einsum', '3.3.0'),
    ('pyasn1-modules', '0.2.8'),
    ('requests-oauthlib', '1.3.1'),
    ('rsa', '4.8'),
    ('tensorboard', '2.9.1'),
    ('tensorboard-data-server', '0.6.1'),
    ('tensorboard-plugin-wit', '1.8.1'),
    ('tensorflow', '2.7.1'),
    ('tensorflow-estimator', '2.7.0'),
    ('tensorflow-io-gcs-filesystem', '0.26.0'),
    ('termcolor', '1.1.0'),
    ('wrapt', '1.14.1')]

ocaisa · 2022-06-30T12:38:00Z

Would the pip upgrade in the virtualenv help with that I wonder...probably not a bad idea to be using the latest and greatest for the use case here.

smoors · 2022-06-30T12:57:28Z

Would the pip upgrade in the virtualenv help with that I wonder...probably not a bad idea to be using the latest and greatest for the use case here.

I agree it's a good idea, but it didn't help in this case.

Flamefire · 2022-07-01T10:13:55Z

Would the pip upgrade in the virtualenv help with that I wonder...probably not a bad idea to be using the latest and greatest for the use case here.

Ok, done.

@smoors

however, strange that pip first says that I have gast 0.5.2, and later it says I have gast 0.4.0...

gast 0.5.2 is in SciPy-Bundle and 0.4.0 in TF, that's likely what confuses pip. Anyway I don't understand the comment in the TF 2.7.1 EC:

# gast deps beniget and pythran require different version from Scipy-bundle

Does gast depend on those or the other way round? The install order in TF and Scipy-bundle is different.

ocaisa · 2022-07-01T13:24:01Z

That comment indeed doesn't read right.

pythran relies on beniget and gast (https://github.com/serge-sans-paille/pythran/blob/master/requirements.txt)
beniget relies on gast (https://github.com/serge-sans-paille/beniget/blame/master/README.rst#L7)

ocaisa · 2022-07-01T13:49:46Z

Apart from the issue around gast, when comparing the output to what is actually in the easyconfig, there seems to be quite a few extra packages in the easyconfig (from https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.7.1-foss-2021b-CUDA-11.4.1.eb#L142) is that because we enable optional stuff in TensorFlow?

easybuild/scripts/findPythonDeps.py

ocaisa · 2022-07-01T14:45:29Z

I tested this out with a set of read-only EB installations with

./findPythonDeps.py --ec TensorFlow-2.3.1-foss-2020a-Python-3.8.2.eb tensorflow==2.3.1

and got this output

[ocaisa@gpunode1 ~]$ ./findPythonDeps.py --ec TensorFlow-2.3.1-foss-2020a-Python-3.8.2.eb tensorflow==2.3.1
This is EasyBuild 4.5.5 (framework: 4.5.5, easyblocks: 4.5.5) on host gpunode1.int.eessi-gpu.learnhpc.eu.
Traceback (most recent call last):
  File "/home/ocaisa/./findPythonDeps.py", line 162, in <module>
    out = run_cmd(cmd, action_desc='Run in new environment', shell=True, executable='/bin/bash')
  File "/home/ocaisa/./findPythonDeps.py", line 50, in run_cmd
    raise RuntimeError('Failed to %s: %s' % (action_desc, out))
RuntimeError: Failed to Run in new environment: virtualenv 20.0.18 from /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages/virtualenv/__init__.py
Traceback (most recent call last):
  File "./findPythonDeps.py", line 169, in <module>
    print_deps(args.package, args.verbose)
  File "./findPythonDeps.py", line 104, in print_deps
    dep_tree = get_dep_tree(package, verbose)
  File "./findPythonDeps.py", line 75, in get_dep_tree
    out = run_in_venv('pip install "%s"' % package_spec, venv_dir, action_desc='install ' + package_spec)
  File "./findPythonDeps.py", line 57, in run_in_venv
    return run_cmd(cmd, action_desc, shell=True, executable='/bin/bash')
  File "./findPythonDeps.py", line 50, in run_cmd
    raise RuntimeError('Failed to %s: %s' % (action_desc, out))
RuntimeError: Failed to install tensorflow==2.3.1: Collecting tensorflow==2.3.1
  Downloading tensorflow-2.3.1-cp38-cp38-manylinux2010_x86_64.whl (320.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 320.5/320.5 MB 4.0 MB/s eta 0:00:00
Collecting keras-preprocessing<1.2,>=1.1.1
  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.6/42.6 kB 8.7 MB/s eta 0:00:00
Collecting gast==0.3.3
  Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Requirement already satisfied: six>=1.12.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.14.0)
Collecting astunparse==1.6.3
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting absl-py>=0.7.0
  Downloading absl_py-1.1.0-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.7/123.7 kB 26.1 MB/s eta 0:00:00
Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/h5py/2.10.0-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (2.10.0)
Collecting opt-einsum>=2.3.2
  Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.5/65.5 kB 14.1 MB/s eta 0:00:00
Collecting tensorflow-estimator<2.4.0,>=2.3.0
  Downloading tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 459.0/459.0 kB 5.1 MB/s eta 0:00:00
Collecting grpcio>=1.8.6
  Downloading grpcio-1.47.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 26.8 MB/s eta 0:00:00
Requirement already satisfied: numpy<1.19.0,>=1.16.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/SciPy-bundle/2020.03-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.18.3)
Requirement already satisfied: wheel>=0.26 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (0.34.2)
Collecting google-pasta>=0.1.8
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.5/57.5 kB 12.1 MB/s eta 0:00:00
Collecting tensorboard<3,>=2.3.0
  Downloading tensorboard-2.9.1-py3-none-any.whl (5.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 28.5 MB/s eta 0:00:00
Collecting wrapt>=1.11.1
  Downloading wrapt-1.14.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (81 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.0/81.0 kB 12.8 MB/s eta 0:00:00
Collecting termcolor>=1.1.0
  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: protobuf>=3.9.2 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/protobuf-python/3.13.0-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (3.13.0)
Requirement already satisfied: setuptools in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from protobuf>=3.9.2->tensorflow==2.3.1) (45.2.0)
Requirement already satisfied: requests<3,>=2.21.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.23.0)
Collecting google-auth<3,>=1.6.3
  Downloading google_auth-2.9.0-py2.py3-none-any.whl (167 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.8/167.8 kB 27.3 MB/s eta 0:00:00
Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Downloading tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 32.3 MB/s eta 0:00:00
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 781.3/781.3 kB 10.8 MB/s eta 0:00:00
Collecting werkzeug>=1.0.1
  Downloading Werkzeug-2.1.2-py3-none-any.whl (224 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 224.9/224.9 kB 37.7 MB/s eta 0:00:00
Collecting markdown>=2.6.8
  Downloading Markdown-3.3.7-py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.8/97.8 kB 20.8 MB/s eta 0:00:00
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting cachetools<6.0,>=2.0.0
  Downloading cachetools-5.2.0-py3-none-any.whl (9.3 kB)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 28.6 MB/s eta 0:00:00
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata>=4.4
  Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.25.9)
Requirement already satisfied: chardet<4,>=3.0.2 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.9)
Requirement already satisfied: certifi>=2017.4.17 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2020.4.5.1)
Requirement already satisfied: zipp>=0.5 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.2.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.4.8)
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.5/151.5 kB 28.3 MB/s eta 0:00:00
Building wheels for collected packages: termcolor
  Building wheel for termcolor (setup.py): started
  Building wheel for termcolor (setup.py): finished with status 'done'
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=1d8e72fb021465f11e53dd21ca36eda7ce1b1498f9d6ba676f15427b2cd6340d
  Stored in directory: /tmp/tmpa3h8_phetensorflow-deps/pip-cache/pip/wheels/a0/16/9c/5473df82468f958445479c59e784896fa24f4a5fc024b0f501
Successfully built termcolor
Installing collected packages: termcolor, tensorflow-estimator, tensorboard-plugin-wit, wrapt, werkzeug, tensorboard-data-server, rsa, pyasn1-modules, opt-einsum, oauthlib, keras-preprocessing, importlib-metadata, grpcio, google-pasta, gast, cachetools, astunparse, absl-py, requests-oauthlib, markdown, google-auth, google-auth-oauthlib, tensorboard, tensorflow
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 1.6.0
    Uninstalling importlib-metadata-1.6.0:
ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: 'INSTALLER'

so it seems to be trying to uninstall modules...

Flamefire · 2022-07-07T08:59:55Z

so it seems to be trying to uninstall modules...

It may try but not actually do due to using a virtualenv. Try to run in verbose mode.

boegelbot · 2022-07-07T10:10:21Z

@Flamefire: Tests failed in GitHub Actions, see https://github.com/easybuilders/easybuild-framework/actions/runs/2628385631
Output from first failing test suite run:

FAIL: test_pypi_source_urls (test.framework.filetools.FileToolsTest)
Test pypi_source_urls() function.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/runner/4bc18a62debab69380a4fd9777e2e69a55bda18f/lib/python2.7/site-packages/test/framework/filetools.py", line 1601, in test_pypi_source_urls
    self.assertTrue(isinstance(res, list) and res)
AssertionError: [] is not true

----------------------------------------------------------------------
Ran 811 tests in 794.137s

FAILED (failures=1)
ERROR: Not all tests were successful.

bleep, bloop, I'm just a bot (boegelbot v20200716.01)
Please talk to my owner @boegel if you notice you me acting stupid),
or submit a pull request to https://github.com/boegel/boegelbot fix the problem.

ocaisa · 2022-07-07T14:24:41Z

Seems that if I run it three times I get three different results, the first fails, the second succeeds (but doesn't report all deps), the third fails again.

output of first run, which fails

[ocaisa@gpunode1 ~]$ ./findPythonDeps.py  --ec TensorFlow-2.3.1-foss-2020a-Python-3.8.2.eb tensorflow==2.3.1
This is EasyBuild 4.5.5 (framework: 4.5.5, easyblocks: 4.5.5) on host gpunode1.int.eessi-gpu.learnhpc.eu.
^[[ATraceback (most recent call last):
  File "/home/ocaisa/./findPythonDeps.py", line 162, in <module>
    out = run_cmd(cmd, action_desc='Run in new environment', shell=True, executable='/bin/bash')
  File "/home/ocaisa/./findPythonDeps.py", line 50, in run_cmd
    raise RuntimeError('Failed to %s: %s' % (action_desc, out))
RuntimeError: Failed to Run in new environment: virtualenv 20.0.18 from /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages/virtualenv/__init__.py
Traceback (most recent call last):
  File "./findPythonDeps.py", line 169, in <module>
    print_deps(args.package, args.verbose)
  File "./findPythonDeps.py", line 104, in print_deps
    dep_tree = get_dep_tree(package, verbose)
  File "./findPythonDeps.py", line 75, in get_dep_tree
    out = run_in_venv('pip install "%s"' % package_spec, venv_dir, action_desc='install ' + package_spec)
  File "./findPythonDeps.py", line 57, in run_in_venv
    return run_cmd(cmd, action_desc, shell=True, executable='/bin/bash')
  File "./findPythonDeps.py", line 50, in run_cmd
    raise RuntimeError('Failed to %s: %s' % (action_desc, out))
RuntimeError: Failed to install tensorflow==2.3.1: Collecting tensorflow==2.3.1
  Downloading tensorflow-2.3.1-cp38-cp38-manylinux2010_x86_64.whl (320.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 320.5/320.5 MB 5.1 MB/s eta 0:00:00
Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/h5py/2.10.0-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (2.10.0)
Requirement already satisfied: wheel>=0.26 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (0.34.2)
Requirement already satisfied: numpy<1.19.0,>=1.16.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/SciPy-bundle/2020.03-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.18.3)
Requirement already satisfied: six>=1.12.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.14.0)
Collecting tensorboard<3,>=2.3.0
  Downloading tensorboard-2.9.1-py3-none-any.whl (5.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 18.1 MB/s eta 0:00:00
Collecting grpcio>=1.8.6
  Downloading grpcio-1.47.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 20.3 MB/s eta 0:00:00
Collecting keras-preprocessing<1.2,>=1.1.1
  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.6/42.6 kB 7.5 MB/s eta 0:00:00
Collecting opt-einsum>=2.3.2
  Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.5/65.5 kB 12.7 MB/s eta 0:00:00
Collecting wrapt>=1.11.1
  Downloading wrapt-1.14.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (81 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.0/81.0 kB 11.4 MB/s eta 0:00:00
Collecting google-pasta>=0.1.8
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.5/57.5 kB 11.2 MB/s eta 0:00:00
Collecting termcolor>=1.1.0
  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting tensorflow-estimator<2.4.0,>=2.3.0
  Downloading tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 459.0/459.0 kB 38.6 MB/s eta 0:00:00
Collecting absl-py>=0.7.0
  Downloading absl_py-1.1.0-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.7/123.7 kB 22.6 MB/s eta 0:00:00
Requirement already satisfied: protobuf>=3.9.2 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/protobuf-python/3.13.0-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (3.13.0)
Collecting gast==0.3.3
  Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Collecting astunparse==1.6.3
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: setuptools in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from protobuf>=3.9.2->tensorflow==2.3.1) (45.2.0)
Collecting markdown>=2.6.8
  Downloading Markdown-3.3.7-py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.8/97.8 kB 14.6 MB/s eta 0:00:00
Requirement already satisfied: requests<3,>=2.21.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.23.0)
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 781.3/781.3 kB 25.9 MB/s eta 0:00:00
Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Downloading tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 32.9 MB/s eta 0:00:00
Collecting google-auth<3,>=1.6.3
  Downloading google_auth-2.9.0-py2.py3-none-any.whl (167 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.8/167.8 kB 19.3 MB/s eta 0:00:00
Collecting werkzeug>=1.0.1
  Downloading Werkzeug-2.1.2-py3-none-any.whl (224 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 224.9/224.9 kB 33.5 MB/s eta 0:00:00
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting cachetools<6.0,>=2.0.0
  Downloading cachetools-5.2.0-py3-none-any.whl (9.3 kB)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 29.9 MB/s eta 0:00:00
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata>=4.4
  Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.25.9)
Requirement already satisfied: certifi>=2017.4.17 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2020.4.5.1)
Requirement already satisfied: idna<3,>=2.5 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.9)
Requirement already satisfied: chardet<4,>=3.0.2 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.0.4)
Requirement already satisfied: zipp>=0.5 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.2.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.4.8)
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.5/151.5 kB 24.8 MB/s eta 0:00:00
Building wheels for collected packages: termcolor
  Building wheel for termcolor (setup.py): started
  Building wheel for termcolor (setup.py): finished with status 'done'
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=e55a8bea5999f27d285984df4928f76dc8540114e865fd6721789b9d28ea5187
  Stored in directory: /tmp/tmpbc58rvigtensorflow-deps/pip-cache/pip/wheels/a0/16/9c/5473df82468f958445479c59e784896fa24f4a5fc024b0f501
Successfully built termcolor
Installing collected packages: termcolor, tensorflow-estimator, tensorboard-plugin-wit, wrapt, werkzeug, tensorboard-data-server, rsa, pyasn1-modules, opt-einsum, oauthlib, keras-preprocessing, importlib-metadata, grpcio, google-pasta, gast, cachetools, astunparse, absl-py, requests-oauthlib, markdown, google-auth, google-auth-oauthlib, tensorboard, tensorflow
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 1.6.0
    Uninstalling importlib-metadata-1.6.0:
ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: 'INSTALLER'

successful output for second run

[ocaisa@gpunode1 ~]$ ./findPythonDeps.py  --ec TensorFlow-2.3.1-foss-2020a-Python-3.8.2.eb tensorflow==2.3.1
This is EasyBuild 4.5.5 (framework: 4.5.5, easyblocks: 4.5.5) on host gpunode1.int.eessi-gpu.learnhpc.eu.
virtualenv 20.0.18 from /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages/virtualenv/__init__.py
tensorflow==2.3.1 installed: Collecting tensorflow==2.3.1
  Downloading tensorflow-2.3.1-cp38-cp38-manylinux2010_x86_64.whl (320.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 320.5/320.5 MB 4.6 MB/s eta 0:00:00
Requirement already satisfied: numpy<1.19.0,>=1.16.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/SciPy-bundle/2020.03-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.18.3)
Collecting grpcio>=1.8.6
  Downloading grpcio-1.47.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 31.3 MB/s eta 0:00:00
Requirement already satisfied: termcolor>=1.1.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.1.0)
Requirement already satisfied: wheel>=0.26 in /tmp/tmpps_a5t96tensorflow-deps/venv/lib/python3.8/site-packages (from tensorflow==2.3.1) (0.34.2)
Collecting astunparse==1.6.3
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: tensorflow-estimator<2.4.0,>=2.3.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (2.3.0)
Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/h5py/2.10.0-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (2.10.0)
Requirement already satisfied: six>=1.12.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.14.0)
Collecting tensorboard<3,>=2.3.0
  Downloading tensorboard-2.9.1-py3-none-any.whl (5.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 31.4 MB/s eta 0:00:00
Requirement already satisfied: opt-einsum>=2.3.2 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (3.3.0)
Collecting gast==0.3.3
  Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Collecting absl-py>=0.7.0
  Downloading absl_py-1.1.0-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.7/123.7 kB 20.2 MB/s eta 0:00:00
Requirement already satisfied: protobuf>=3.9.2 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/protobuf-python/3.13.0-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (3.13.0)
Collecting google-pasta>=0.1.8
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.5/57.5 kB 12.1 MB/s eta 0:00:00
Requirement already satisfied: wrapt>=1.11.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.14.1)
Requirement already satisfied: keras-preprocessing<1.2,>=1.1.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.1.2)
Requirement already satisfied: setuptools in /tmp/tmpps_a5t96tensorflow-deps/venv/lib/python3.8/site-packages (from protobuf>=3.9.2->tensorflow==2.3.1) (46.1.3)
Collecting google-auth<3,>=1.6.3
  Downloading google_auth-2.9.0-py2.py3-none-any.whl (167 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.8/167.8 kB 28.1 MB/s eta 0:00:00
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.6.1)
Collecting markdown>=2.6.8
  Downloading Markdown-3.3.7-py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.8/97.8 kB 18.5 MB/s eta 0:00:00
Requirement already satisfied: requests<3,>=2.21.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.23.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.8.1)
Requirement already satisfied: werkzeug>=1.0.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.1.2)
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (4.8)
Collecting cachetools<6.0,>=2.0.0
  Downloading cachetools-5.2.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata>=4.4
  Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Requirement already satisfied: certifi>=2017.4.17 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2020.4.5.1)
Requirement already satisfied: chardet<4,>=3.0.2 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.25.9)
Requirement already satisfied: zipp>=0.5 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.2.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.2.0)
Installing collected packages: importlib-metadata, grpcio, google-pasta, gast, cachetools, astunparse, absl-py, requests-oauthlib, markdown, google-auth, google-auth-oauthlib, tensorboard, tensorflow
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 1.6.0
    Not uninstalling importlib-metadata at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages, outside environment /tmp/tmpps_a5t96tensorflow-deps/venv
    Can't uninstall 'importlib-metadata'. No files were found to uninstall.
Successfully installed absl-py-1.1.0 astunparse-1.6.3 cachetools-5.2.0 gast-0.3.3 google-auth-2.9.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.47.0 importlib-metadata-4.12.0 markdown-3.3.7 requests-oauthlib-1.3.1 tensorboard-2.9.1 tensorflow-2.3.1

List of dependencies in (likely) install order:
[   ('Markdown', '3.3.7'),
    ('grpcio', '1.47.0'),
    ('requests-oauthlib', '1.3.1'),
    ('cachetools', '5.2.0'),
    ('google-auth', '2.9.0'),
    ('google-auth-oauthlib', '0.4.6'),
    ('absl-py', '1.1.0'),
    ('tensorboard', '2.9.1'),
    ('google-pasta', '0.2.0'),
    ('gast', '0.3.3'),
    ('astunparse', '1.6.3'),
    ('tensorflow', '2.3.1')]
Sorted list of dependencies:
[   ('Markdown', '3.3.7'),
    ('absl-py', '1.1.0'),
    ('astunparse', '1.6.3'),
    ('cachetools', '5.2.0'),
    ('gast', '0.3.3'),
    ('google-auth', '2.9.0'),
    ('google-auth-oauthlib', '0.4.6'),
    ('google-pasta', '0.2.0'),
    ('grpcio', '1.47.0'),
    ('requests-oauthlib', '1.3.1'),
    ('tensorboard', '2.9.1'),
    ('tensorflow', '2.3.1')]

failed output for third run

[ocaisa@gpunode1 ~]$ ./findPythonDeps.py  --ec TensorFlow-2.3.1-foss-2020a-Python-3.8.2.eb tensorflow==2.3.1
This is EasyBuild 4.5.5 (framework: 4.5.5, easyblocks: 4.5.5) on host gpunode1.int.eessi-gpu.learnhpc.eu.
Traceback (most recent call last):
  File "/home/ocaisa/./findPythonDeps.py", line 162, in <module>
    out = run_cmd(cmd, action_desc='Run in new environment', shell=True, executable='/bin/bash')
  File "/home/ocaisa/./findPythonDeps.py", line 50, in run_cmd
    raise RuntimeError('Failed to %s: %s' % (action_desc, out))
RuntimeError: Failed to Run in new environment: virtualenv 20.0.18 from /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages/virtualenv/__init__.py
tensorflow==2.3.1 installed: Requirement already satisfied: tensorflow==2.3.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (2.3.1)
Requirement already satisfied: gast==0.3.3 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (0.3.3)
Requirement already satisfied: grpcio>=1.8.6 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.47.0)
Requirement already satisfied: absl-py>=0.7.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.1.0)
Requirement already satisfied: tensorflow-estimator<2.4.0,>=2.3.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (2.3.0)
Requirement already satisfied: astunparse==1.6.3 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.6.3)
Requirement already satisfied: google-pasta>=0.1.8 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (0.2.0)
Requirement already satisfied: protobuf>=3.9.2 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/protobuf-python/3.13.0-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (3.13.0)
Requirement already satisfied: termcolor>=1.1.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.1.0)
Requirement already satisfied: numpy<1.19.0,>=1.16.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/SciPy-bundle/2020.03-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.18.3)
Requirement already satisfied: tensorboard<3,>=2.3.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (2.9.1)
Requirement already satisfied: wrapt>=1.11.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.14.1)
Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/h5py/2.10.0-foss-2020a-Python-3.8.2/lib/python3.8/site-packages (from tensorflow==2.3.1) (2.10.0)
Requirement already satisfied: wheel>=0.26 in /tmp/tmpcz8or1pjtensorflow-deps/venv/lib/python3.8/site-packages (from tensorflow==2.3.1) (0.34.2)
Requirement already satisfied: keras-preprocessing<1.2,>=1.1.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.1.2)
Requirement already satisfied: six>=1.12.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (1.14.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorflow==2.3.1) (3.3.0)
Requirement already satisfied: setuptools in /tmp/tmpcz8or1pjtensorflow-deps/venv/lib/python3.8/site-packages (from protobuf>=3.9.2->tensorflow==2.3.1) (46.1.3)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.8.1)
Requirement already satisfied: werkzeug>=1.0.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.1.2)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.4.6)
Requirement already satisfied: markdown>=2.6.8 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.3.7)
Requirement already satisfied: requests<3,>=2.21.0 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.23.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.9.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.6.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (5.2.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (4.8)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (4.12.0)
Requirement already satisfied: chardet<4,>=3.0.2 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.9)
Requirement already satisfied: certifi>=2017.4.17 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2020.4.5.1)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.25.9)
Requirement already satisfied: zipp>=0.5 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.2.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /localscratch/ocaisa.357.0/lib/python3.8/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.2.0)

Traceback (most recent call last):
  File "./findPythonDeps.py", line 169, in <module>
    print_deps(args.package, args.verbose)
  File "./findPythonDeps.py", line 104, in print_deps
    dep_tree = get_dep_tree(package, verbose)
  File "./findPythonDeps.py", line 79, in get_dep_tree
    dep_tree = run_in_venv('pipdeptree -j -p "%s"' % package_name,
  File "./findPythonDeps.py", line 57, in run_in_venv
    return run_cmd(cmd, action_desc, shell=True, executable='/bin/bash')
  File "./findPythonDeps.py", line 50, in run_cmd
    raise RuntimeError('Failed to %s: %s' % (action_desc, out))
RuntimeError: Failed to collect dependencies: /bin/bash: /localscratch/ocaisa.357.0/bin/pipdeptree: /tmp/tmpps_a5t96tensorflow-deps/venv/bin/python: bad interpreter: No such file or directory

ocaisa · 2022-07-07T14:28:11Z

In my environment PIP_PREFIX is set, and I think that is affecting things.

ocaisa · 2022-07-07T14:31:18Z

Confirmed, after unsetting that var it worked out of the box (and for numerous iterations)

Flamefire · 2022-07-08T09:38:22Z

Confirmed, after unsetting that var it worked out of the box (and for numerous iterations)

Then I think it makes sense to do that in the script. Can you test the new version?

ocaisa · 2022-07-08T10:11:33Z

This works great now. One small niggle with your dependency checking, it throws false positives if the user has non-allowed modules loaded:

WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
* GCCcore/10.3.0
* GCCcore/10.3.0

The warning is in stderr, so if you only parse stdout you could avoid it (or you could make sure the line includes a easyconfig filename)

Check only stdout not stderr. The latter may contain lines like > WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment: > * GCCcore/10.3.0

Flamefire · 2022-07-08T14:11:30Z

The warning is in stderr, so if you only parse stdout you could avoid it (or you could make sure the line includes a easyconfig filename)

Good catch. Added a toggle to NOT capture stderr to the run-wrapper. Can you retest?

ocaisa · 2022-07-11T08:39:12Z

This works in all my scenarios now.

I think you need to add it to the list at https://github.com/easybuilders/easybuild-framework/blob/develop/setup.py#L99 so that it actually get's installed with a new release.

ocaisa · 2022-07-11T08:40:34Z

Ah, maybe not, I guess it is already covered in https://github.com/easybuilders/easybuild-framework/blob/develop/setup.py#L102

Flamefire · 2022-07-14T16:17:15Z

We may even want to combine both into data_files. No need to add them to scripts which may modify the shebang

ocaisa · 2022-07-15T12:59:40Z

Changing the installation will just delay this PR. I'm going to merge as is, and then if you feel strongly about moving things to data_files you can open another PR (and someone more familiar with Python packaging could comment)

Flamefire added 4 commits September 20, 2021 08:55

Add script to find dependencies of Python packages

51df040

Example usage with EasyBuild: ``` eb TensorFlow-2.3.4.eb --dump-env source TensorFlow-2.3.4.env findPythonDeps.py tensorflow==2.3.4 ```

Add --ec option

c19f7f1

Import pkg_resources with error if not found

3cd3596

Better output

cfa1929

Flamefire force-pushed the findPythonDeps branch from dfeb2f8 to cfa1929 Compare September 20, 2021 06:55

easybuilders deleted a comment from boegelbot Dec 4, 2021

boegel added the enhancement label Dec 4, 2021

boegel added this to the 4.x milestone Dec 4, 2021

ocaisa reviewed Jun 30, 2022

View reviewed changes

easybuild/scripts/findPythonDeps.py Outdated Show resolved Hide resolved

ocaisa reviewed Jun 30, 2022

View reviewed changes

easybuild/scripts/findPythonDeps.py Show resolved Hide resolved

Flamefire added 3 commits June 30, 2022 13:40

Fix typo

5323e82

Check availability of EasyBuild and virtualenv

766df6f

Improves the error messages generated.

Formatting

aae7002

Flamefire added 2 commits July 1, 2022 12:06

Be a bit more verbose for --verbose

70c2afc

Update PIP in virtualenv first

03ef150

ocaisa reviewed Jul 1, 2022

View reviewed changes

easybuild/scripts/findPythonDeps.py Outdated Show resolved Hide resolved

ocaisa reviewed Jul 1, 2022

View reviewed changes

easybuild/scripts/findPythonDeps.py Outdated Show resolved Hide resolved

Flamefire added 2 commits July 7, 2022 10:59

Improve help text for --ec

5c1c503

Don't assume script is executable

ca22095

Flamefire added 4 commits July 8, 2022 11:15

Improve help message

83cfffe

Suppress output of call

b1c6a42

Check and report missing modules when using --ec

b6d8423

Unset $PIP_PREFIX if set

9a3eb24

Avoid false positives in dependency checking

7f8ab8d

Check only stdout not stderr. The latter may contain lines like > WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment: > * GCCcore/10.3.0

ocaisa approved these changes Jul 15, 2022

View reviewed changes

ocaisa merged commit f0686d1 into easybuilders:develop Jul 15, 2022

Flamefire deleted the findPythonDeps branch July 15, 2022 18:12

branfosj modified the milestones: 4.x, next release (4.6.1?) Jul 20, 2022

boegel changed the title ~~Add script to find dependencies of Python packages~~ add script to find dependencies of Python packages Aug 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add script to find dependencies of Python packages #3839

add script to find dependencies of Python packages #3839

Flamefire commented Sep 16, 2021 •

edited

Loading

smoors commented Jun 30, 2022

ocaisa commented Jun 30, 2022

smoors commented Jun 30, 2022

Flamefire commented Jul 1, 2022 •

edited

Loading

ocaisa commented Jul 1, 2022

ocaisa commented Jul 1, 2022

ocaisa commented Jul 1, 2022 •

edited

Loading

Flamefire commented Jul 7, 2022

boegelbot commented Jul 7, 2022

ocaisa commented Jul 7, 2022 •

edited

Loading

ocaisa commented Jul 7, 2022

ocaisa commented Jul 7, 2022 •

edited

Loading

Flamefire commented Jul 8, 2022

ocaisa commented Jul 8, 2022 •

edited

Loading

Flamefire commented Jul 8, 2022

ocaisa commented Jul 11, 2022

ocaisa commented Jul 11, 2022

Flamefire commented Jul 14, 2022

ocaisa commented Jul 15, 2022

add script to find dependencies of Python packages #3839

add script to find dependencies of Python packages #3839

Conversation

Flamefire commented Sep 16, 2021 • edited Loading

smoors commented Jun 30, 2022

ocaisa commented Jun 30, 2022

smoors commented Jun 30, 2022

Flamefire commented Jul 1, 2022 • edited Loading

ocaisa commented Jul 1, 2022

ocaisa commented Jul 1, 2022

ocaisa commented Jul 1, 2022 • edited Loading

Flamefire commented Jul 7, 2022

boegelbot commented Jul 7, 2022

ocaisa commented Jul 7, 2022 • edited Loading

ocaisa commented Jul 7, 2022

ocaisa commented Jul 7, 2022 • edited Loading

Flamefire commented Jul 8, 2022

ocaisa commented Jul 8, 2022 • edited Loading

Flamefire commented Jul 8, 2022

ocaisa commented Jul 11, 2022

ocaisa commented Jul 11, 2022

Flamefire commented Jul 14, 2022

ocaisa commented Jul 15, 2022

Flamefire commented Sep 16, 2021 •

edited

Loading

Flamefire commented Jul 1, 2022 •

edited

Loading

ocaisa commented Jul 1, 2022 •

edited

Loading

ocaisa commented Jul 7, 2022 •

edited

Loading

ocaisa commented Jul 7, 2022 •

edited

Loading

ocaisa commented Jul 8, 2022 •

edited

Loading