Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update scipy easyblock for scipy >= 1.9.0 to use meson/ninja #2862

Merged
merged 15 commits into from
Feb 25, 2023

Conversation

jfgrimm
Copy link
Member

@jfgrimm jfgrimm commented Jan 17, 2023

(created using eb --new-pr)

@jfgrimm
Copy link
Member Author

jfgrimm commented Jan 18, 2023

figuring out the best way to ensure all the tests are run in a way they don't fail silently

@boegel boegel marked this pull request as draft January 18, 2023 16:36
@jfgrimm jfgrimm marked this pull request as ready for review January 19, 2023 15:30
@jfgrimm
Copy link
Member Author

jfgrimm commented Jan 19, 2023

Test report by @jfgrimm
Using easyblocks from PR(s) #2861, #2862
SUCCESS
Build succeeded for 6 out of 6 (6 easyconfigs in total)
himem06.pri.viking.alces.network - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/d4871a9ec3d7bf99984c4dc547e674e9 for a full test report.

@jfgrimm

This comment was marked as outdated.

@jfgrimm
Copy link
Member Author

jfgrimm commented Jan 19, 2023

Test report by @jfgrimm

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 2 (2 easyconfigs in total)
himem06.pri.viking.alces.network - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/09121aae431fc0f1327daf30c8066f1a for a full test report.

@jfgrimm
Copy link
Member Author

jfgrimm commented Jan 19, 2023

test failures actually are present in old builds, they were just ignored (see #2237)

@jfgrimm
Copy link
Member Author

jfgrimm commented Jan 20, 2023

Test report by @jfgrimm

Overview of tested easyconfigs (in order)

  • SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
  • SUCCESS SciPy-bundle-2021.05-foss-2021a.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
himem06.pri.viking.alces.network - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/7e697701e89c433ca89242d6515ea421 for a full test report.

extra_vars = ({
'enable_slow_tests': [False, "Run scipy test suite, including tests marked as slow", CUSTOM],
# ignore tests by default to maintain behaviour in older easyconfigs
'ignore_test_result': [True, "Run scipy test suite, but ignore result (only log)", CUSTOM],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should set this to None, and auto-enable for scipy versions >= 1.9.0 unless it's explicitly set to False or True in easyconfig?



class EB_scipy(FortranPythonPackage):
class EB_scipy(FortranPythonPackage, MesonNinja):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit surprised that using MesonNinja works to install scipy as an extension, since that generic easyblock doesn't derive from ExtensionEasyBlock (which is sort of required for stuff you want to install both as extension and stand-alone).

That's probably because MesonNinja is only used as "secondary" easyblock, next to FortranPythonPackage, but to make this future-proof (there will come a time where deriving from FortranPythonPackage is useless since the scipy versions we still support are all >= 1.9.0) we may need to be a bit more careful here (which could be as simple as changing MesonNinja to derive from ExtensionEasyBlock, rather than from EasyBlock like it does now).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so I had a bit of a closer look, and the key thing that makes this work as-is seems to be calling an __init__ (e.g. PythonPackages.__init__ ) which derives from ExtensionEasyBlock
With the new changes, the meson build still makes use of PythonPackage anyway (to get the python command, and for the sanity check), so now removing FortranPythonPackage would not break anything.

easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
@boegel
Copy link
Member

boegel commented Jan 22, 2023

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS SciPy-bundle-2020.11-foss-2020b.eb
  • SUCCESS SciPy-bundle-2021.05-foss-2021a.eb
  • SUCCESS SciPy-bundle-2021.05-intel-2021a.eb
  • SUCCESS SciPy-bundle-2021.10-foss-2021b.eb
  • SUCCESS SciPy-bundle-2021.10-foss-2021b-Python-2.7.18.eb
  • SUCCESS SciPy-bundle-2021.10-intel-2021b.eb
  • SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
  • SUCCESS SciPy-bundle-2022.05-intel-2022a.eb

Build succeeded for 8 out of 8 (8 easyconfigs in total)
node3102.skitty.os - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/b5dd170a8e3d16664a9a10723b1a6411 for a full test report.

easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
if self.cfg['ignore_test_result'] is None:
self.cfg['ignore_test_result'] = True
# ignore installopts inherited from FortranPythonPackage
self.cfg['installopts'] = ""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should at least also log the original installopts value if we're going to do a hard reset here...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to strip the ones inherited from PythonPackage, perhaps that's better?

easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Show resolved Hide resolved
easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
@branfosj
Copy link
Member

== 2023-02-16 11:27:10,457 build_log.py:171 ERROR EasyBuild crashed with an error (at easybuild/src/easybuild-framework/easybuild/base/exceptions.py:126 in __init__): cmd " unset LDFLAGS &&   ninja -j 20  --no-deps  --ignore-installed  --no-index  install" exited with exit code 1 and output:
ninja: unrecognized option '--no-deps'
usage: ninja [options] [targets...]

if targets are unspecified, builds the 'default' target (see manual).

options:
  --version      print ninja version ("1.11.1")
  -v, --verbose  show all command lines while building
  --quiet        don't show progress status, just command output

  -C DIR   change to DIR before doing anything else
  -f FILE  specify input build file [default=build.ninja]

  -j N     run N jobs in parallel (0 means infinity) [default=20 on this system]
  -k N     keep going until N jobs fail (0 means infinity) [default=1]
  -l N     do not start new jobs if the load average is greater than N
  -n       dry run (don't run commands but act like they succeeded)

  -d MODE  enable debugging (use '-d list' to list modes)
  -t TOOL  run a subtool (use '-t list' to list subtools)
    terminates toplevel options; further flags are passed to the tool
  -w FLAG  adjust warnings (use '-w list' to list warnings)

From a build of easybuilders/easybuild-easyconfigs#16912 started at 11:19:59 - so with 016118a

easybuild/easyblocks/s/scipy.py Outdated Show resolved Hide resolved
simplify logic to set default value for ignore_test_result in scipy easyblock
@jfgrimm
Copy link
Member Author

jfgrimm commented Feb 25, 2023

Test report by @jfgrimm

Overview of tested easyconfigs (in order)

  • SUCCESS SciPy-bundle-2019.10-foss-2019b-Python-3.7.4.eb
  • SUCCESS SciPy-bundle-2019.10-fosscuda-2019b-Python-3.7.4.eb
  • SUCCESS SciPy-bundle-2020.11-foss-2020b.eb
  • SUCCESS SciPy-bundle-2020.11-fosscuda-2020b.eb
  • SUCCESS SciPy-bundle-2021.05-foss-2021a.eb
  • SUCCESS SciPy-bundle-2021.05-intel-2021a.eb
  • SUCCESS SciPy-bundle-2021.10-foss-2021b.eb
  • SUCCESS SciPy-bundle-2021.10-intel-2021b.eb
  • SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
  • SUCCESS SciPy-bundle-2022.05-intel-2022a.eb
  • SUCCESS Python-2.7.18-GCCcore-11.2.0.eb
  • SUCCESS hypothesis-4.57.1-GCCcore-11.2.0-Python-2.7.18.eb
  • SUCCESS pybind11-2.7.1-GCCcore-11.2.0-Python-2.7.18.eb
  • SUCCESS SciPy-bundle-2021.10-foss-2021b-Python-2.7.18.eb

Build succeeded for 14 out of 14 (11 easyconfigs in total)
node098.pri.viking.alces.network - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/d1cc0932ecfb0d55b8151f53461eb279 for a full test report.

@boegel
Copy link
Member

boegel commented Feb 25, 2023

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS SciPy-bundle-2019.10-foss-2019b-Python-3.7.4.eb
  • SUCCESS SciPy-bundle-2019.10-fosscuda-2019b-Python-3.7.4.eb
  • SUCCESS SciPy-bundle-2020.11-foss-2020b.eb
  • SUCCESS SciPy-bundle-2020.11-fosscuda-2020b.eb
  • SUCCESS SciPy-bundle-2021.05-foss-2021a.eb
  • SUCCESS SciPy-bundle-2021.05-intel-2021a.eb
  • SUCCESS SciPy-bundle-2021.10-foss-2021b.eb
  • SUCCESS SciPy-bundle-2021.10-foss-2021b-Python-2.7.18.eb
  • SUCCESS SciPy-bundle-2021.10-intel-2021b.eb
  • SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
  • SUCCESS SciPy-bundle-2022.05-intel-2022a.eb

Build succeeded for 11 out of 11 (11 easyconfigs in total)
node3125.skitty.os - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/fcb67fa6b23e8e779e23ebc5ec3b7402 for a full test report.

@boegel
Copy link
Member

boegel commented Feb 25, 2023

Thanks a lot for all the effort on this @jfgrimm and @branfosj!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants