WW3/SCHSIM integration and CoastalApp pre-requirements #102

saeed-moghimi-noaa · 2022-03-01T19:04:07Z

From meeting on 3/1/2022
@platipodium @awest-noaa @josephzhang8 @zacharyburnettNOAA

Next steps on CoastalApp repo side:

Feature/ww3 branch to merge with develop
- WW3 changes by @awest-noaa #103
Joseph/Carsten push to schism-dev:feature/schism, then Zach merges into noaa:feature/schism
- merge schism-dev/feature/schism into noaa-ocs-modeling/feature/schism #104
Merge feature/schism to develop
- add SCHISM submodule and build files #44
Test by Andre
Test Joseph and Carsten
Tag
(Joseph/Carsten) fork develop to schism-dev and drop feature/schism

Andre will provide the test case to Joseph.

Joseph tests testcase with SCHISM on VIMS infrastructure
Carsten/Joseph test WW3/SCHISM on VIMS

The text was updated successfully, but these errors were encountered:

platipodium · 2022-03-28T03:24:26Z

Joseph/Carsten push to schism-dev:feature/schism

ghost · 2022-03-30T11:35:32Z

merged #44 with commit 9b47de5

ghost · 2022-03-31T19:29:52Z

@platipodium I apologize for that, I have fixed the issue in build.sh in ba54b23, but I now get

[100%] Built target pschism
make[3]: Leaving directory `/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM/build'
/apps/cmake/3.20.1/bin/cmake -E cmake_progress_start /scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM/build/CMakeFiles 0
make[2]: Leaving directory `/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM/build'
make[1]: Leaving directory `/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM/build'
### Compile the SCHISM cap, this uses the DESTDIR and SCHISM_BUILD_DIR exported variables
make -C  /scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM DESTDIR=/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM_INSTALL \
SCHISM_BUILD_DIR=/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM/build install-nuopc
make[1]: Entering directory `/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM'
make[1]: *** No rule to make target `install-nuopc'.  Stop.
make[1]: Leaving directory `/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM'
make: *** [/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM_INSTALL/schism.mk] Error 2

I will continue to look for the change that broke this, again sorry!

platipodium · 2022-04-07T13:32:48Z

./build.sh does not build WW3:

[INFO] list of the programs selected : ww3_prnc

Processing ww3_prnc
---------------------
make[2]: Entering directory `/sciclone/home20/clemmen/devel/noaa/CoastalApp/WW3/model/ftn'
ad3 : processing w3nmlprncmd
mv: cannot stat 'w3nmlprncmd.o': No such file or directory
ad3 : processing ww3_prnc
mv: cannot stat 'ww3_prnc.o': No such file or directory
      Linking ww3_prnc
      *** file ww3_prnc.o not found ***
      *** file w3nmlprncmd.o not found ***
*** Missing object files ***
make[2]: *** [/sciclone/home20/clemmen/devel/noaa/CoastalApp/WW3/model/exe/ww3_prnc] Error 3
make[2]: Leaving directory `/sciclone/home20/clemmen/devel/noaa/CoastalApp/WW3/model/ftn'
make[1]: *** [gout] Error 1
make[1]: Leaving directory `/sciclone/home20/clemmen/devel/noaa/CoastalApp/WW3/model/esmf'
make: *** [/sciclone/home20/clemmen/devel/noaa/CoastalApp/WW3_INSTALL/nuopc.mk] Error 2

platipodium · 2022-04-07T13:56:31Z

Side note for NEMS:

rm -rf NEMS
git clone -b feature/schism [email protected]:schism-dev/NEMS.git NEMS
(cd NEMS; git checkout feature/schism; git submodule update --init --recursive)

still issues with ww3_prnc and "SCHISM not supported" remain

platipodium · 2022-04-07T14:01:43Z

 ./build.sh -plat femto -component "WW3DATA ATMESH SCHISM"
The modeling component "SCHISM" is not supported.

ghost · 2022-04-07T15:00:25Z

here are the differences between the feature/schism branch before and after synchronizing with develop (i.e., these are all the NEW commits and changes that came FROM develop INTO feature/schism):
https://github.com/schism-dev/CoastalApp/compare/a559471ae09b9364f6859b54c03c7980388d03e3...noaa-ocs-modeling:feature/schism?diff=split&w=1

ghost · 2022-04-07T16:38:11Z

we found a few incompatibilities when building, so I will revert develop back to before the merge, and then @platipodium will make some fixes and make a PR

ghost · 2022-04-07T16:39:41Z

build error:

schism_esmf_util.F90(164): error #6796: The variable must have the TARGET attribute or be a subobject of an object with the TARGET attribute, or it must have the POINTER attribute.[ETA2]
isPtr%wrap%ptrMap(4)%farrayPtr1 => eta2
-------------------------------------^
schism_esmf_util.F90(1603): error #6633: The type of the actual argument differs from the type of the dummy argument.   [TRIM]
call ESMF_TimeSet(startTime, trim(message), rc=localrc)
-------------------------------^
compilation aborted for schism_esmf_util.F90 (code 1)
make[2]: *** [schism_esmf_util.o] Error 1
make[2]: Leaving directory `/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM/schism-esmf/src/schism'
make[1]: *** [src/schism/schism_bmi.o] Error 2
make[1]: Leaving directory `/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM/schism-esmf'
make: *** [/scratch2/COASTAL/coastal/save/shared/repositories/CoastalApp_schism/SCHISM_INSTALL/schism.mk] Error 2

platipodium · 2022-04-07T16:40:09Z

I will work on this on schism-dev/features schism and PR

ghost · 2022-04-07T16:47:47Z

building WW3 on Hera seems to correctly compile the ww3_prnc component:

 ./build.sh -plat hera -component "WW3" --compiler intel

platipodium · 2022-04-07T17:10:58Z

Work required on schism-esmf clean target, such that it does not depend on schism library being present @platipodium

ghost · 2022-04-07T17:16:11Z

NEMS/src/incmake/component_SCHISM.mk

# Rule for cleaning the SRCDIR and BINDIR:

clean_SCHISM:
	+cd $(SCHISM_ROOTDIR)/schism-esmf; exec $(MAKE) DESTDIR=$(SCHISM_BINDIR) SCHISM_BUILD_DIR=$(SCHISM_BLDDIR) -k clean
	+cd $(SCHISM_BLDDIR) ; exec $(MAKE) -k clean
	@echo ""

distclean_SCHISM: clean_SCHISM
	+cd $(SCHISM_ROOTDIR)/schism-esmf; exec $(MAKE) DESTDIR=$(SCHISM_BINDIR) SCHISM_BUILD_DIR=$(SCHISM_BLDDIR) -k distclean
	+cd $(SCHISM_BLDDIR) ; exec $(MAKE) -k distclean
	rm -rf $(SCHISM_BINDIR)
	@echo ""

https://github.com/schism-dev/NEMS/blob/d8ebe07d41e584888f6e8e20d26e4f6399161520/src/incmake/component_SCHISM.mk#L69-L80

platipodium · 2022-04-08T17:43:04Z

With the updates in NEMS and schism-esmf the clean and distclean targets have been fixed @pvelissariou1 . Please check. Changes have been made to schism-dev forks of CostalApp (updated pointers) and NEMS (schism makefile fragment). @zacharyburnettNOAA Please accept existing PRs upstream into NEMS feature/schism and CoastalApp feature/schim

ghost · 2022-04-08T19:15:29Z

@zacharyburnettNOAA Please accept existing PRs upstream into NEMS feature/schism and CoastalApp feature/schim

done, see below:

platipodium · 2022-04-08T20:31:42Z

I need help to make progress on femto with compilation of WW3. Still no idea how we get the wrong compiler env in the build system, nor how to get rid of it @pvelissariou1 and @awest-noaa

platipodium · 2022-04-08T20:34:09Z

I fixed mpiifort -> mpifort locally for femto (intel/openmpi toolchain). But still the wrong ifort is selected:

WW3/model/tmp/w3nmlprncmd.l:/opt/spack/lib/spack/env/intel/ifort -c -module /sciclone/home20/clemmen/devel/noaa/CoastalApp/WW3/model/mod -no-fma -ip -p -g -i4 -real-size 32 -fp-model precise -assume byterecl -convert big_endian -fno-alias -fno-fnalias -O3 -xhost -I/usr/local/skylake/linux-centos7-x86_64/intel-18.0.5/netcdf-4.4.1.1-mtecjavmtxm5pcn3zqjhkm57xzk2byqw/include -I/sciclone/home20/clemmen/devel/noaa/CoastalApp/WW3/model/ftn
WW3/model/tmp/w3nmlprncmd.l:/sciclone/home20/clemmen/devel/noaa/CoastalApp/WW3/model/bin/comp: Zeile 96: /opt/spack/lib/spack/env/intel/ifort: Keine Berechtigung


/sciclone/home20/clemmen/devel/noaa/CoastalApp% which ifort
/usr/local/intel-2018/compilers_and_libraries_2018.5.274/linux/bin/intel64/ifort

platipodium · 2022-05-27T17:27:37Z

WW3 compiles fine on strand (intel/intelmpi) but not on femto (intel/openmpi) nor levante (gcc/openmpi)

platipodium · 2022-05-27T18:14:35Z

levante:~/devel/noaa/CoastalApp/WW3/model/tmp/ww3_prnc.err
ww3_prnc.F90:190:11:

190 | USE NETCDF
| 1
Fatal Error: Cannot open module file 'netcdf.mod' for reading at (1): No such file or directory
compilation terminated.

platipodium · 2022-05-27T18:17:52Z

femto:~/devel/noaa/CoastalApp/WW3/model/tmp/ww3_prnc.err
/sciclone/home20/clemmen/devel/noaa/CoastalApp/WW3/model/bin/comp: Zeile 96: /opt/spack/lib/spack/env/intel/ifort: Keine Berechtigung

platipodium · 2022-05-27T18:46:18Z

setenv WWATCH3_NETCDF NC4
setenv NETCDF_CONFIG /apps/netcdf/4.6.1/intel/16.1.150/bin/nc-config

.ww3.env

setenv WWATCH3_NETCDF NC4
setenv NETCDF_CONFIG /apps/netcdf/4.6.1/intel/16.1.150/bin/nc-config
.wwatch3.env

Environment variables for wavewatch III

---------------------------------------

WWATCH3_LPR printer
WWATCH3_F77 ifort
WWATCH3_CC icc
WWATCH3_DIR /scratch4/COASTAL/coastal/save/Andre.VanderWesthuysen/ww3_git/EMC_ww3/model
WWATCH3_TMP /scratch4/COASTAL/coastal/save/Andre.VanderWesthuysen/ww3_git/EMC_ww3/model/tmp
WWATCH3_SOURCE no
WWATCH3_LIST no

platipodium · 2022-05-27T18:53:00Z

WWATCH3_ENV := $(WW3_BINDIR)/wwatch3.env
WWATCH3_NETCDF := NC4
NETCDF_CONFIG := $(shell which nc-config)
export WWATCH3_ENV WWATCH3_NETCDF NETCDF_CONFIG

This file will overwrite the contents of previously set variables.

platipodium · 2022-05-27T18:54:13Z

I think these should be obtained from ESMFMKFILE (my 2ct)... Takis will work on more consistency in the near future.

platipodium · 2022-05-27T18:55:15Z

# ESMF_NETCDF_INCLUDE:    /opt/homebrew/anaconda3/envs/esmf/include
# ESMF_NETCDF_LIBS:       -lnetcdff -lnetcdf
# ESMF_NETCDF_LIBPATH:    /opt/homebrew/anaconda3/envs/esmf/lib

This info is also contained in ESMF's CMake provided

acrosby · 2022-06-08T15:44:12Z

@platipodium Not sure if this is helpful to you...

We did the following for our ADCIRC/WW3/ATMESH Docker image build to resolve the WW3 gnu/linux build issues with the netcdf dependent WW3 components:

sed 's/`$NETCDF_CONFIG --fc`/gfortran/g' -i WW3/model/bin/comp.tmpl
sed 's/`$NETCDF_CONFIG --includedir`/\/usr\/local\/include/g' -i WW3/model/bin/comp.tmpl
sed 's/`$NETCDF_CONFIG --fc`/gfortran/g' -i WW3/model/bin/link.tmpl
sed 's/`$NETCDF_CONFIG --includedir`/\/usr\/local\/include/g' -i WW3/model/bin/link.tmpl
sed 's/`$NETCDF_CONFIG --flibs`/-L\/usr\/local\/lib -lnetcdff -lnetcdf /g' -i WW3/model/bin/link.tmpl

Not sure why the backticks are not getting processed correctly, but this brute forces the required outputs for our cases, where we have manually built netcdf4, netcdf fortran and hdf5 into the /usr/local prefix.

platipodium · 2022-06-10T15:00:59Z

Thanks @acrosby the workaround is fine. So somehow the shell script is not portable to gnu systems, maybe we need the $( ) command or similar.. @pvelissariou1

platipodium · 2022-12-29T15:28:25Z

New problem #152 different compile issue on femto

platipodium · 2023-02-02T07:38:14Z

Problems persists on femto:

CoastalApp Compilation fails on PAHM on femto #157 with PAHM
--thirdparty libparmetis is not found on femto #158 with third party parmetis

saeed-moghimi-noaa assigned awest-noaa, josephzhang8, ghost and saeed-moghimi-noaa Mar 1, 2022

This was linked to pull requests Mar 2, 2022

add SCHISM submodule and build files #44

Merged

WW3 changes by @awest-noaa #103

Merged

ghost mentioned this issue Mar 2, 2022

merge schism-dev/feature/schism into noaa-ocs-modeling/feature/schism #104

Merged

ghost linked a pull request Mar 2, 2022 that will close this issue

merge schism-dev/feature/schism into noaa-ocs-modeling/feature/schism #104

Merged

platipodium mentioned this issue Sep 7, 2022

WW3 does not compile on femto/intel with wrong compiler selection #129

Open

pvelissariou1 mentioned this issue Oct 11, 2022

No specific subroutine for the generic ‘nuopc_updatetimestamp’ #89

Closed

platipodium mentioned this issue Dec 29, 2022

WW3 no-compile on femto/intel/openmpi, netcdf, workaround #152

Open

platipodium mentioned this issue Feb 15, 2023

building -tp parmetis fails on multiple platforms (Levante, femto) #169

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WW3/SCHSIM integration and CoastalApp pre-requirements #102

WW3/SCHSIM integration and CoastalApp pre-requirements #102

saeed-moghimi-noaa commented Mar 1, 2022 •

edited by ghost

Loading

platipodium commented Mar 28, 2022

ghost commented Mar 30, 2022

ghost commented Mar 31, 2022

platipodium commented Apr 7, 2022 •

edited by ghost

Loading

platipodium commented Apr 7, 2022 •

edited

Loading

platipodium commented Apr 7, 2022

ghost commented Apr 7, 2022 •

edited by ghost

Loading

ghost commented Apr 7, 2022

ghost commented Apr 7, 2022

platipodium commented Apr 7, 2022

ghost commented Apr 7, 2022 •

edited by ghost

Loading

platipodium commented Apr 7, 2022

ghost commented Apr 7, 2022

platipodium commented Apr 8, 2022 •

edited

Loading

ghost commented Apr 8, 2022 •

edited by ghost

Loading

platipodium commented Apr 8, 2022

platipodium commented Apr 8, 2022

platipodium commented May 27, 2022

platipodium commented May 27, 2022 •

edited

Loading

platipodium commented May 27, 2022

platipodium commented May 27, 2022

platipodium commented May 27, 2022

platipodium commented May 27, 2022

platipodium commented May 27, 2022

acrosby commented Jun 8, 2022 •

edited

Loading

platipodium commented Jun 10, 2022

platipodium commented Dec 29, 2022

platipodium commented Feb 2, 2023

WW3/SCHSIM integration and CoastalApp pre-requirements #102

WW3/SCHSIM integration and CoastalApp pre-requirements #102

Comments

saeed-moghimi-noaa commented Mar 1, 2022 • edited by ghost Loading

platipodium commented Mar 28, 2022

ghost commented Mar 30, 2022

ghost commented Mar 31, 2022

platipodium commented Apr 7, 2022 • edited by ghost Loading

platipodium commented Apr 7, 2022 • edited Loading

platipodium commented Apr 7, 2022

ghost commented Apr 7, 2022 • edited by ghost Loading

ghost commented Apr 7, 2022

ghost commented Apr 7, 2022

platipodium commented Apr 7, 2022

ghost commented Apr 7, 2022 • edited by ghost Loading

platipodium commented Apr 7, 2022

ghost commented Apr 7, 2022

platipodium commented Apr 8, 2022 • edited Loading

ghost commented Apr 8, 2022 • edited by ghost Loading

platipodium commented Apr 8, 2022

platipodium commented Apr 8, 2022

platipodium commented May 27, 2022

platipodium commented May 27, 2022 • edited Loading

platipodium commented May 27, 2022

platipodium commented May 27, 2022

Environment variables for wavewatch III

---------------------------------------

platipodium commented May 27, 2022

platipodium commented May 27, 2022

platipodium commented May 27, 2022

acrosby commented Jun 8, 2022 • edited Loading

platipodium commented Jun 10, 2022

platipodium commented Dec 29, 2022

platipodium commented Feb 2, 2023

saeed-moghimi-noaa commented Mar 1, 2022 •

edited by ghost

Loading

platipodium commented Apr 7, 2022 •

edited by ghost

Loading

platipodium commented Apr 7, 2022 •

edited

Loading

ghost commented Apr 7, 2022 •

edited by ghost

Loading

ghost commented Apr 7, 2022 •

edited by ghost

Loading

platipodium commented Apr 8, 2022 •

edited

Loading

ghost commented Apr 8, 2022 •

edited by ghost

Loading

platipodium commented May 27, 2022 •

edited

Loading

acrosby commented Jun 8, 2022 •

edited

Loading