update TensorFlow 2.1.0 easyconfigs to provide more dependencies via EasyBuild #11109

Flamefire · 2020-08-13T14:22:53Z

Requires Make TensorFlow use system/EasyBuild installed libraries easybuild-easyblocks#2117
Requires create symlink from lib64 to lib subdir in installation directories to avoid that GCC prefers /lib64 system directories easybuild-framework#3401 and ICU rebuild
Requires automatically add required -G Ninja when using CMakeNinja easyblock + add 'generator' custom easyconfig parameter for CMakeMake easybuild-easyblocks#2120
Requires ~~Use pip to install protobuf-python in 2019b toolchain (requires re-downloading source tarball!) #11143~~ (use pip for protobuf-python)
Requires ~~{math}[fosscuda/2019b] scipy v1.4.1 w/ Python 3.7.4 #11216~~ (scipy for fosscuda/2019b)
Requires ~~{math}[foss/2019b] scipy v1.4.1 w/ Python 3.7.4 #11218~~ (scipy 1.4.1 for foss/2019b)

This is basically a test run. It makes the installation faster and smaller and avoids some downloads at build time.

This approach will be required for TF 2.3.0 and I'll add PRs for the other 2.x ECs once this is merged

Reason why it is required is the TF 2.3 bundles cURL but builds it with Bazel side-stepping the (correct) configuration from cURL and introducing a bug on some systems making parts of TF unusable: tensorflow/tensorflow#40065 (comment)

Flamefire · 2020-08-14T12:20:39Z

@boegel Whats going on with the tests? I didn't add neither hwloc nor ICU

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb

Flamefire · 2020-08-31T13:36:53Z

Test report by @Flamefire
FAILED
Build succeeded for 3 out of 5 (5 easyconfigs in this PR)
taurusi5018.taurus.hrsk.tu-dresden.de - Linux RHEL 7.8, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/5cb1a2273ad5422115b2db1b55190799 for a full test report.

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb

easybuild/easyconfigs/n/nsync/nsync-1.24.0-GCCcore-8.3.0.eb

boegel

also stick to hwloc 1.x in foss easyconfig

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb

boegel · 2020-09-04T12:33:10Z

@boegelbot please test @ generoso
EB_ARGS="--include-easyblocks-from-pr 2117"

boegelbot · 2020-09-04T12:40:10Z

@boegel: Request for testing this PR well received on generoso

PR test command 'EB_PR=11109 EB_ARGS="--include-easyblocks-from-pr 2117" /apps/slurm/default/bin/sbatch --job-name test_PR_11109 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 5257

Test results coming soon (I hope)...

- notification for comment with ID 687114655 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

Flamefire · 2020-09-04T13:48:31Z

Test report by @Flamefire
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
taurusml24 - Linux RHEL 7.6, POWER, 8335-GTX, Python 2.7.5
See https://gist.github.com/5718090247881e2ea7707f1e3e6dec33 for a full test report.

boegel · 2020-09-04T13:51:46Z

Test report by @boegel
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
node3404.kirlia.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz (cascadelake), Python 2.7.5
See https://gist.github.com/ff4737a4818fc66d2af5a080b54aaa88 for a full test report.

boegel · 2020-09-04T13:54:27Z

Test report by @boegel
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
node3106.skitty.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/6ebd5b6de8b4089942268534c3f19d4e for a full test report.

boegel · 2020-09-04T14:04:26Z

Test report by @boegel
FAILED
Build succeeded for 7 out of 9 (5 easyconfigs in this PR)
node3300.joltik.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), Python 3.6.8
See https://gist.github.com/e9d830106381a7b54475758b41617c91 for a full test report.

edit: failed because of old double-conversion installation (see fix in #9659), will re-submit test

Flamefire · 2020-09-04T15:31:10Z

Test report by @Flamefire
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
taurusi5186.taurus.hrsk.tu-dresden.de - Linux RHEL 7.8, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/8b24a73ad25d361a97ebe5a720ec452d for a full test report.

boegelbot · 2020-09-04T15:46:17Z

Test report by @boegelbot
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
generoso-x-1 - Linux centos linux 8.2.2004, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/5493bd67e169c13dd654efbb4629cf5e for a full test report.

boegel · 2020-09-04T15:48:07Z

Test report by @boegel
FAILED
Build succeeded for 4 out of 6 (5 easyconfigs in this PR)
node2411.golett.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (haswell), Python 2.7.5
See https://gist.github.com/2fb41216076a30e81e72c7c106455281 for a full test report.

edit: ignore this one, not enough memory available (gcc: internal compiler error: Bus error signal terminated program cc1plus)

boegel · 2020-09-04T18:56:41Z

Test report by @boegel
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
node3309.joltik.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), Python 3.6.8
See https://gist.github.com/5938623f83e0e896a69e3b12437e3acb for a full test report.

boegel · 2020-09-04T22:36:32Z

Test report by @boegel
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in this PR)
node3415.kirlia.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz (cascadelake), Python 2.7.5
See https://gist.github.com/8b1db00b08f61e9552006ec0d18ea2f6 for a full test report.

edit: tested on top of easybuilders/easybuild-easyblocks#2163

boegel · 2020-09-05T05:39:20Z

@boegelbot please test @ generoso
EB_ARGS="--include-easyblocks-from-pr 2163"

boegelbot · 2020-09-05T05:40:08Z

@boegel: Request for testing this PR well received on generoso

PR test command 'EB_PR=11109 EB_ARGS="--include-easyblocks-from-pr 2163" /apps/slurm/default/bin/sbatch --job-name test_PR_11109 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 5260

Test results coming soon (I hope)...

- notification for comment with ID 687555263 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2020-09-05T08:46:29Z

Test report by @boegelbot
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in this PR)
generoso-x-1 - Linux centos linux 8.2.2004, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/1acb3029ad4bc23e5483b7977393d686 for a full test report.

boegel · 2020-09-06T07:41:52Z

Going in, thanks @Flamefire!

Flamefire mentioned this pull request Aug 13, 2020

Make TensorFlow use system/EasyBuild installed libraries easybuilders/easybuild-easyblocks#2117

Merged

Flamefire mentioned this pull request Aug 17, 2020

{lib}[fosscuda/2019b] TensorFlow v2.3.0 w/ Python 3.7.4 #11040

Merged

2 tasks

Flamefire force-pushed the tensorflow_syslibs branch 4 times, most recently from fad4551 to 3b6d8ab Compare August 17, 2020 14:37

easybuilders deleted a comment from boegelbot Aug 18, 2020

boegel added the enhancement label Aug 18, 2020

boegel added this to the next release (4.2.3?) milestone Aug 18, 2020

boegel requested changes Aug 18, 2020

View reviewed changes

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb Outdated Show resolved Hide resolved

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb Outdated Show resolved Hide resolved

Flamefire force-pushed the tensorflow_syslibs branch from 94b8c8a to 1ede6f9 Compare August 19, 2020 14:59

Flamefire mentioned this pull request Sep 1, 2020

make sure self.start_dir is set in ExtensionEasyBlock easybuilders/easybuild-framework#3426

Merged

Flamefire force-pushed the tensorflow_syslibs branch from 1ede6f9 to c6ea62a Compare September 1, 2020 12:23

boegel requested changes Sep 4, 2020

View reviewed changes

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb Outdated Show resolved Hide resolved

mstud reviewed Sep 4, 2020

View reviewed changes

easybuild/easyconfigs/n/nsync/nsync-1.24.0-GCCcore-8.3.0.eb Outdated Show resolved Hide resolved

Flamefire force-pushed the tensorflow_syslibs branch from 64ad00a to 99be36f Compare September 4, 2020 08:33

easybuilders deleted a comment from boegelbot Sep 4, 2020

boegel requested changes Sep 4, 2020

View reviewed changes

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb Outdated Show resolved Hide resolved

easybuilders deleted a comment from boegelbot Sep 4, 2020

boegel changed the title ~~[TensorFlow 2.1.0] Use system libs~~ update TensorFlow 2.1.0 easyconfigs to provide more dependencies via EasyBuild Sep 4, 2020

Flamefire added 2 commits September 4, 2020 14:22

[TensorFlow 2.1.0] Use system libs

7a78560

Use SciPy from EC to reduce build times

098dadc

Flamefire force-pushed the tensorflow_syslibs branch from 991c12f to 098dadc Compare September 4, 2020 12:23

boegel added change new update labels Sep 4, 2020

boegel approved these changes Sep 6, 2020

View reviewed changes

boegel merged commit 4eec88a into easybuilders:develop Sep 6, 2020

Flamefire deleted the tensorflow_syslibs branch September 7, 2020 07:55

Flamefire mentioned this pull request Sep 7, 2020

use more EasyBuild installed dependencies for TensorFlow 2.2.0 #11224

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update TensorFlow 2.1.0 easyconfigs to provide more dependencies via EasyBuild #11109

update TensorFlow 2.1.0 easyconfigs to provide more dependencies via EasyBuild #11109

Flamefire commented Aug 13, 2020 •

edited by boegel

Loading

Flamefire commented Aug 14, 2020

Flamefire commented Aug 31, 2020

boegel left a comment

boegel commented Sep 4, 2020

boegelbot commented Sep 4, 2020

Flamefire commented Sep 4, 2020

boegel commented Sep 4, 2020

boegel commented Sep 4, 2020

boegel commented Sep 4, 2020 •

edited

Loading

Flamefire commented Sep 4, 2020

boegelbot commented Sep 4, 2020

boegel commented Sep 4, 2020 •

edited

Loading

boegel commented Sep 4, 2020

boegel commented Sep 4, 2020 •

edited

Loading

boegel commented Sep 5, 2020

boegelbot commented Sep 5, 2020

boegelbot commented Sep 5, 2020

boegel commented Sep 6, 2020

update TensorFlow 2.1.0 easyconfigs to provide more dependencies via EasyBuild #11109

update TensorFlow 2.1.0 easyconfigs to provide more dependencies via EasyBuild #11109

Conversation

Flamefire commented Aug 13, 2020 • edited by boegel Loading

Flamefire commented Aug 14, 2020

Flamefire commented Aug 31, 2020

boegel left a comment

Choose a reason for hiding this comment

boegel commented Sep 4, 2020

boegelbot commented Sep 4, 2020

Flamefire commented Sep 4, 2020

boegel commented Sep 4, 2020

boegel commented Sep 4, 2020

boegel commented Sep 4, 2020 • edited Loading

Flamefire commented Sep 4, 2020

boegelbot commented Sep 4, 2020

boegel commented Sep 4, 2020 • edited Loading

boegel commented Sep 4, 2020

boegel commented Sep 4, 2020 • edited Loading

boegel commented Sep 5, 2020

boegelbot commented Sep 5, 2020

boegelbot commented Sep 5, 2020

boegel commented Sep 6, 2020

Flamefire commented Aug 13, 2020 •

edited by boegel

Loading

boegel commented Sep 4, 2020 •

edited

Loading

boegel commented Sep 4, 2020 •

edited

Loading

boegel commented Sep 4, 2020 •

edited

Loading