Fujitsu toolchain TODO list for EB v4.4.0

[migueldiascosta]

Framework

• toolchain definitions: add toolchain definition for Fujitsu ARM toolchain easybuild-framework#3677
  • hierarchy: FCC, ffmpi, Fujitsu (even though MPI and SSL are provided by the lang module)
  • make sure fcc is always called with -Nclang, even when CFLAGS is not used (e.g. _ctypes/libffi configure script in Python2)
  • -Kfp_precise doesn't work in clang mode, closest we can get seems to be -Knoeval,nofast_matmul,nofp_contract,nofp_relaxed,noilfunc (and at least the tests that require precise in GMP do pass with these flags)
  • -KNOSVE also doesn't work in clang mode (use -Knosimd instead)
  • -Kfast is actually -O3 plus a bunch of other stuff, so to play it safe keep default level at -O2 and leave -Kfast to opt and specific easyblocks
  • including arm_sve.h fails in clang mode when including the top level include dir of the lang module (in clang mode it should be the one in e.g. $EBROOTLANG/clang-comp/lib64/clang/7.1.0/include, but there is no need to add it), skip it when adding include flags for dependency prefixes
  • test with RPATH
  • make sure top level lib64 dir of lang module is found and added to rpath by wrapper
    • use LDFLAGS instead of LIBRARY_PATH, so that it is added to the buildenv module: set LDFLAGS instead of LIBRARY_PATH and add new required keyword to _get_software_root for Fujitsu toolchain easybuild-framework#3704
  • add required keyword argument (introduced in Performance improvements for easyconfig parsing and eb startup easybuild-framework#3555) to overriden _get_software_root: set LDFLAGS instead of LIBRARY_PATH and add new required keyword to _get_software_root for Fujitsu toolchain easybuild-framework#3704
• w.r.t. easybuilders/easybuild-framework@2138971, we need a better solution for passing -Nclang: it was moved to CC and CXX in compiler toolchain definition because some configure scripts (e.g. _ctypes/libffi) don't use C[XX]FLAGS when detecting the compiler, but on the other hand some scripts try to find an executable named fcc -Nclang and fail, including easybuild itself: WARNING Could not find command 'fcc -Nclang' (...) DEBUG Not installing RPATH wrapper for non-existing command 'fcc -Nclang', so this is not right; moved back to C[XX]FLAGS in move -Nclang back to $C[XX]FLAGS in toolchain support for Fujitsu compiler easybuild-framework#3713
• w.r.t. set LDFLAGS instead of LIBRARY_PATH and add new required keyword to _get_software_root for Fujitsu toolchain easybuild-framework#3704, we probably need to set both LIBRARY_PATH and LDFLAGS, some builds don't use LDFLAGS so the RPATH wrapper misses it (without RPATH it's not a problem because the compiler always looks at this path for dynamic libraries); done in add Fujitsu compiler lib* subdir to both $LIBRARY_PATH and $LDFLAGS easybuild-framework#3714
  • only add libdir to LIBRARY_PATH if it's not already there: ~~only add fujitsu libdir to LIBRARY_PATH if it's not already there easybuild-framework#3717
• fcc only accepts gnuXX, FCC only accepts gnu++XX, will need to parse CFLAGS and CXXFLAGS accordingly in the framework toolchain definition: parse C standard flags in CFLAGS for Fujitsu compiler easybuild-framework#3731
• needs COMPILER_MODULE_NAME = ['FCC'] (or simply inherit from ffmpi) in toolchains/fujitsu.py, which will require explicitly adding FCC as a dependency in the Fujitsu easyconfig: don't override COMPILER_MODULE_NAME, inherited from Ffmpi, in Fujitsu toolchain definition easybuild-framework#3721

Easyblocks

• update fftw easyblock: enhance FFTW easyblock to support SVE CPU feature and building with Fujitsu compiler easybuild-easyblocks#2425

Easyconfigs

• don't use --try-toolchain --copy-ec to generate easyconfigs to contribute back, use cp/mv/sed scripts instead
• FCC: {toolchain}[system/system] FCC v21.05 easybuild-easyconfigs#12863, change FCC's moduleclass to 'compiler' and add explicit subtoolchain dependencies to Fujitsu easyconfig easybuild-easyconfigs#13007
  • version pinning
  • binutils osdep
  • use the compiler version, FCC-4.5.0, instead of FCC-21.05 (currently the modules available are lang/tcsds-1.2.29 with Fujitsu Compiler 4.3.1, lang/tcsds-1.2.30a with Fujitsu Compiler 4.4.0a and lang/tcsds-1.2.31 with Fujitsu Compiler 4.5.0, so it does seem to be one compiler version per module version): done in use version of Fujitsu compiler instead of our version for the Fujitsu toolchain easybuild-easyconfigs#12995
• binutils and dependencies: {lang,tools}[FCC/4.5.0] binutils v2.36.1, Bison v3.7.6, flex v2.6.4, ... easybuild-easyconfigs#12996
  • configure script seems to be confused about the presence of limits.h and fcntl.h when using the Fujitsu compiler, fixed with prebuildopts = 'export CFLAGS="-DHAVE_LIMITS_H -DHAVE_FCNTL_H $CFLAGS" && '
  • M4: building was hitting this, https://bugs.llvm.org/show_bug.cgi?id=16404
    • fixed by adding --rtlib=compiler-rt to $LDFLAGS (this may be necessary in other places...)
• Perl and dependencies: {tools}[FCC/4.5.0] Perl v5.32.1, DB v18.1.40, expat v2.2.9, groff v1.22.4, ... easybuild-easyconfigs#12997
  • Perl extension XML-Bare has hardcoded CC=gcc, SetInterval tries to use clang++, used exts_default_options = {'buildopts': 'CC="$CC"'} but may be worth addressing more generically in easyblock?
• Autotools and dependencies: {devel}[FCC/4.5.0] Autoconf v2.71, Automake v1.16.3, Autotools v20210128, ... easybuild-easyconfigs#12998
• FFTW: {numlib}[FCC/4.5.0] FFTW v1.0.0-fujitsu easybuild-easyconfigs#12999
• ffmpi + OSU-Micro-Benchmarks: {perf,toolchain}[ffmpi/4.5.0,system/system] ffmpi v4.5.0, OSU-Micro-Benchmarks v5.7.1 easybuild-easyconfigs#13000
• Full toolchain + HPL: {devel,toolchain,tools}[Fujitsu/21.05] Fujitsu v21.05, HPL v2.3, buildenv vdefault easybuild-easyconfigs#13001, change FCC's moduleclass to 'compiler' and add explicit subtoolchain dependencies to Fujitsu easyconfig easybuild-easyconfigs#13007
  • MPI doesn't seem to be supported in Fujitsu's FFTW yet, so it's disabled in the easyconfig and moving it to FCC instead of ffmpi to make that clear
  • decide name (and versionsuffix?) of FFTW module ({toolchain}[system/system] FCC v21.05 easybuild-easyconfigs#12863 (comment)): done in don't use a different module name for FFTW in the Fujitsu toolchain easybuild-framework#3712

[boegel]

@migueldiascosta

w.r.t. easybuilders/easybuild-framework@2138971, we need a better solution for passing -Nclang: it was moved to CC and CXX in compiler toolchain definition because some configure scripts (e.g. _ctypes/libffi) don't use C[XX]FLAGS when detecting the compiler, but on the other hand some scripts try to find an executable named fcc -Nclang and fail, including easybuild itself: WARNING Could not find command 'fcc -Nclang' (...) DEBUG Not installing RPATH wrapper for non-existing command 'fcc -Nclang', so this is not right

• one option would be to enforce RPATH and make sure -Nclang is set in the RPATH wrapper (?)

We can enhance framework to strip off flags from the $CXX value before it tries to locate the command, but even then you may run into surprises in some places. The best way to fix may be to ensure that $CXXFLAGS is picked up consistently, but that seems painful?
I'm not sure that abusing the RPATH wrappers makes sense. Perhaps we need another compiler wrapper script that can be used to inject specific flags (and then have the RPATH wrapper call that wrapper rather than the actual compiler)?

w.r.t. easybuilders/easybuild-framework#3704, we probably need to set both LIBRARY_PATH and LDFLAGS, some builds don't use LDFLAGS so the RPATH wrapper misses it (without RPATH it's not the problem because the compiler always looks at this path for dynamic libraries)

Same solution, a separate wrapper to inject what's needed?

[migueldiascosta]

split into what went into EB v4.4.0, this issue, and what's still open, #704