Update sanity tests for OpenFOAM, add logging #1205

olesenm · 2017-07-05T11:17:22Z

this allows redirection of the Allwmake output to a log file so that
it is possible to debug any errors with the build process and have a
record of the configured build options.

boegel · 2017-07-06T07:44:43Z

problem with the tests should be fixed now that easybuilders/easybuild-framework#2258 is merged; I have retriggered the tests...

- this allows redirection of the Allwmake output to a log file so that it is possible to debug any errors with the build process and have a record of the configured build options.

- The 'wdot' utility has been deprecated in favour of 'postProcess', so don't wdot for the sanity check. - For scripts, only 'paraFoam' is relatively safe to check. The foamExec script may not last since it uses some outdated FOAM_INST_DIR semantics.

olesenm · 2017-07-11T10:48:02Z

Hello @easybuilders/easybuild-easyblocks-maintainers - please review

boegel · 2017-08-09T12:09:58Z

easybuild/easyblocks/o/openfoam.py

@@ -281,6 +281,9 @@ def build_step(self):
                "Cleaning .*",
            ]
            run_cmd_qa(cmd_tmpl % 'Allwmake.firstInstall', qa, no_qa=noqa, log_all=True, simple=True)
+        elif LooseVersion(self.version) >= LooseVersion('1600'):


Hmm, this version check looks wrong? OpenFOAM versions are like 2.3.1, 3.0.1, etc.

So, the condition here would never be met?

Actually we now have versions following YYMM, so 1606, 1612, 1706 thus far with 1706 having been released at the end of June-2017.
http://openfoam.com/releases/openfoam-v1706/

Ah, OK, I overlooked that, my apologies.

So, the -log is only supported since the 1600 version?

I'd also suggest to cleaning this up a bit, feels too copy-paste as it is now:

else: cmd = 'Allwmake' if LooseVersion(self.version) >= LooseVersion('1600'): # instruct to redirect output of Allwmake to a log file, for easier debugging & future reference cmd += ' -log' run_cmd(cmd_tmpl % cmd, log_all=True, simple=True, log_output=True)

It was added in 1606. How would you clean this up? (Too much of the package looks like cut-n-paste already).

Then you should modify the version check to >= LooseVersion('1606')?

See code snippet above for a suggestion on how to avoid copy-pasting.

boegel · 2017-08-09T12:10:12Z

easybuild/easyblocks/o/openfoam.py

-               [os.path.join(toolsdir, x) for x in ["deformedGeom", "engineSwirl", "modifyMesh",
-                                                    "refineMesh", "wdot"]]
+               [os.path.join(toolsdir, x) for x in ["checkMesh", "deformedGeom", "engineSwirl",
+                                                    "modifyMesh", "refineMesh"]]


Why these changes to the sanity check, they are unrelated to the -log change?

Among other things, the wdot utility has been deprecated. There is a shell script placeholder (in a different location) that now only produces a message about it being redundant. This means that the previous sanity checks will never work.
Since the comments in the sanity check claim to have some arbitrarily chosen applications, I simply replaced wdot with checkMesh which should continue to exist for a while.
These sanity checks could actually be further reduced to blockMesh and checkMesh and be done with it.

OK, thanks for clarifying the removal of wdot, makes sense.

I'd rather keep the others around next to blockMesh & checkMesh, since they may help in catching problems with the build, unless you have a good reason for getting rid of those too?

What about the removal of foamExec, has that been deprecated/removed in recent versions too?

Sure, no harm keeping around others. If foamExec hasn't been removed, then I agree that it makes a bad sanity test. Probably the only really safe script to check is paraFoam, the others may be too much of a moving target.

Hmm, not sure what you are saying w.r.t. foamExec... Was it removed in recent OpenFOAM versions?

The original design of foamExec was to help starting up with identical paths, versions on various machines when not using a scheduler. While this generally still works, the upgrades to OpenFOAM configuration files means that the foamExec provision for starting an alternative OpenFOAM version and/or an alternative prefix location is quite broken. So I think that it (and foamJob) are in need of an overhaul or removal. In either case, a perhaps slightly dubious future.

boegel · 2017-08-10T13:38:54Z

easybuild/easyblocks/o/openfoam.py

@@ -281,6 +281,9 @@ def build_step(self):
                "Cleaning .*",
            ]
            run_cmd_qa(cmd_tmpl % 'Allwmake.firstInstall', qa, no_qa=noqa, log_all=True, simple=True)
+        elif LooseVersion(self.version) >= LooseVersion('1600'):


Ah, OK, I overlooked that, my apologies.

So, the -log is only supported since the 1600 version?

I'd also suggest to cleaning this up a bit, feels too copy-paste as it is now:

else: cmd = 'Allwmake' if LooseVersion(self.version) >= LooseVersion('1600'): # instruct to redirect output of Allwmake to a log file, for easier debugging & future reference cmd += ' -log' run_cmd(cmd_tmpl % cmd, log_all=True, simple=True, log_output=True)

boegel · 2017-09-05T12:55:04Z

@olesenm ping on following up on this? It would be nice to get this merged in soon...

akesandgren · 2017-09-07T05:32:41Z

Just a note on the version thingie. Remember that there are versions of openfoam from both openfoam.com and openfoam.org. The openfoam.org versions are still at 5.x which is approx the same as the 17xx version from openfoam.com. Some users want the openform.org version and the easyblock needs to handle both.

boegel · 2017-09-07T06:41:37Z

For now, I don't think there's a problem, but that may change in the future if the .org and .com versions go further apart...

@olesenm Can we think of a way to discriminate between the .org vs the .com versions (preferably without relying on checking the format of the version number)?

If needed, we can consider renaming the .com versions to something else than OpenFOAM, like we did with OpenFOAM-Extend.

olesenm · 2017-09-11T13:18:04Z

@boegel : first day back from holidays (thus unresponsive for a period).

For spack, we have openfoam as a virtual package and openfoam-com, openfoam-org and foam-extend as individual packages that are each a provider of the openfoam virtual package.
I think that this removes all ambiguity and doesn't try to take any position about what is considered upstream or fork etc.

The configuration files for openfoam-com and openfoam-org are very similar, but probably not enough to use the same diffs for them. In spack, we generate the new content directly (instead of patching), but don't know how well that fits into the easybuild philosophy, nor if it works well with the rest of the easybuild setup. The foam-extend configuration files look quite different, so need some separate treatment. Most of the commonly used core routines I tucked into the openfoam-com package and just import them as required for the others.

olesenm · 2017-09-11T13:43:47Z

For openfoam-com, the wmake/rules/General/general file defines the following line:

WM_VERSION = OPENFOAM_PLUS=1706

where OPENFOAM_PLUS comes from the internal name for the development line, but is also used to define the release as an integer (YYMM) for use as a compiler define. In retrospect, it might have been cleaner to have defined OPENFOAM_COM, but not important enough to go change it now.

The following snippet should work to catch openfoam-com, but does not provide any distinction between development and release branches (just the major API level).

# openfoam.com (development or release branch)
#
# parse wmake/rules/General/general/rules for this type of content
#    WM_VERSION = OPENFOAM_PLUS=1706
# can be used instead of parsing $WM_PROJECT_VERSION at all
foamType  = "openfoam-com"
reVersion = re.compile(r'OPENFOAM_PLUS\s*=\s*([\d.]+)')
try:
    for line in open(environ["WM_DIR"] + '/rules/General/general', 'r'):
        m = re.search(reVersion, line)
        if m:
            foamType  = "openfoam-com"
            foamVer   = m.group(1)
            break
except (KeyError, IOError):
    pass

# OR a fallback:
m = re.match(r'v?(\d{4})\+?', verString)
# openfoam.com (release version) - eg, v1606+, v1612, 1706 etc
...

For foam-extend a WM_FORK=extend variable is defined in the etc/bashrc file.

For openfoam-org there is nothing particular to help users distinguish the source.

boegel · 2017-10-17T19:44:02Z

@olesenm Please take a look at the updates in olesenm#1, which should make this good to go, sorry about the holdup...

boegel · 2017-10-19T08:32:01Z

Merged via #1272, thanks @olesenm!

easybuilders deleted a comment from boegelbot Jul 6, 2017

olesenm added 2 commits July 11, 2017 11:53

ENH: add Allwmake logging for OpenFOAM build

2d16417

- this allows redirection of the Allwmake output to a log file so that it is possible to debug any errors with the build process and have a record of the configured build options.

olesenm force-pushed the openfoam-log-option branch from 13b9ad1 to 4fddf72 Compare July 11, 2017 09:53

olesenm changed the title ~~ENH: add Allwmake logging for OpenFOAM build~~ Update sanity tests for OpenFOAM, add logging Jul 11, 2017

wpoely86 approved these changes Jul 11, 2017

View reviewed changes

wpoely86 added this to the 3.4.0 milestone Jul 11, 2017

boegel requested changes Aug 9, 2017

View reviewed changes

boegel requested changes Aug 10, 2017

View reviewed changes

boegel modified the milestones: 3.4.0, 3.5.0 Sep 4, 2017

olesenm mentioned this pull request Sep 6, 2017

new OpenFOAM-1612, OpenFOAM-1706 versions easybuilders/easybuild-easyconfigs#4814

Closed

This was referenced Oct 17, 2017

{cae}[intel/2017a] OpenFOAM v5.0 easybuilders/easybuild-easyconfigs#5233

Merged

fix code style issues + add blockMesh to sanity check in OpenFOAM easyblock olesenm/easybuild-easyblocks#1

Closed

boegel mentioned this pull request Oct 19, 2017

remove foamExec & wdot from sanity checks, add blockMesh & checkMesh + enable logging for recent OpenFOAM versions #1272

Merged

wpoely86 merged commit 4fddf72 into easybuilders:develop Oct 19, 2017

olesenm deleted the openfoam-log-option branch December 11, 2017 15:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update sanity tests for OpenFOAM, add logging #1205

Update sanity tests for OpenFOAM, add logging #1205

olesenm commented Jul 5, 2017

boegel commented Jul 6, 2017

olesenm commented Jul 11, 2017

boegel Aug 9, 2017

olesenm Aug 9, 2017

boegel Aug 10, 2017

olesenm Aug 12, 2017

boegel Aug 12, 2017

boegel Aug 9, 2017

olesenm Aug 9, 2017 •

edited

Loading

boegel Aug 10, 2017

olesenm Aug 12, 2017

boegel Aug 12, 2017

olesenm Sep 11, 2017

boegel Aug 10, 2017

boegel commented Sep 5, 2017

akesandgren commented Sep 7, 2017

boegel commented Sep 7, 2017

olesenm commented Sep 11, 2017

olesenm commented Sep 11, 2017

boegel commented Oct 17, 2017

boegel commented Oct 19, 2017

Update sanity tests for OpenFOAM, add logging #1205

Update sanity tests for OpenFOAM, add logging #1205

Conversation

olesenm commented Jul 5, 2017

boegel commented Jul 6, 2017

olesenm commented Jul 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olesenm Aug 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boegel commented Sep 5, 2017

akesandgren commented Sep 7, 2017

boegel commented Sep 7, 2017

olesenm commented Sep 11, 2017

olesenm commented Sep 11, 2017

boegel commented Oct 17, 2017

boegel commented Oct 19, 2017

olesenm Aug 9, 2017 •

edited

Loading