CoastalApp test suite #150
Replies: 136 comments 39 replies
-
|
@pvelissariou1 @saeed-moghimi-noaa Hi Takis, I have downloaded the package of CoastalApp (include CoastalApp-testsuite, CoastalApp code and the required data to run the "hsofs" tests cases). On our cluster, we cannot run the command below to compile CoastalApp with PDLIB: PARMETISHOME=/usr/local/src/ANDRE/COMPILED/INTEL19/parmetis-4.0.3 ./build.sh --compiler intel --platform hydra --component "atmesh ww3" -y Because there is already 'setenv PARMETISHOME ...' in file .cshrc for our cluster, I can compile CoastalApp with command below: ./build.sh --compiler intel --platform hydra --component "atmesh ww3" -y After that, went to the run test phase. I only kept the case 'florence_shinnecock.atm2ww3' in regtest_list.dat. Following the command to run 'run_all.sh' like bellow but faced errors: [jqi@hydra CoastalApp-testsuite]$ setenv ROOTDIR /data4/jqi1/CoastalApp-testsuite
My question is: Can I reorganize the files in folder florence_shinnecock.atm2ww3/run and run this case with the regular command on our cluster? Are there something I should notice? Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Hi There Great news that you passed the compilation phase. Seems like your job submission system is not using Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Discussed with Dr. Changsheng Chen, we are going to test CoastalApp-testsuite on Frontera. Could I ask the questions about Frontera and CoastalApp here? Thanks |
Beta Was this translation helpful? Give feedback.
-
|
@pvelissariou1 @saeed-moghimi-noaa Hi, May I know if there is CoastalApp developer or user who applied CoastalApp on TACC-FRONTERA? I have downloaded the whole package of CoastalApp-testsuite including CoastalApp source code on Frontera. When attempting to build CoastalApp, it failed. (the parmetis was successfully installed in thirdparty_open) When compiling CoastalApp on our cluster Hydra, I created a file envmodules_intel.hydra in folder modulefiles to set the environment for CoastalApp. I don't know how to create a similar one on Frontera. I think this is the first and critical issue to compile CoastalApp on Frontera. Do you know how to find the paths of libraries for CoastalApp compilation on Frontera or give me suggestion who I can contact with ? I only found the environment list like below: login2.frontera(81)$ module list Currently Loaded Modules:
Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
On TACC there is not a system wide module for the required ESMF library,
and so CostalApp compilation will fail.
I have compiled an older version in my own work filesystem. I assume, that
the modulefile envmodules_intel.stampede
should work on frontera as well. Have you tried to compile using
--platform=stampede to see if it compiles on frontera?
I'll check myself later today for a fix.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Wed, Jan 11, 2023 at 10:14 AM FVCOM-GitHub ***@***.***> wrote:
@pvelissariou1 <https://github.com/pvelissariou1> @saeed-moghimi-noaa
<https://github.com/saeed-moghimi-noaa>
Hi, May I know if there is CoastalApp developer or user who applied
CoastalApp on TACC-FRONTERA?
I have downloaded the whole package of CoastalApp-testsuite including
CoastalApp source code on Frontera. When attempting to build CoastalApp, it
failed. (the parmetis was successfully installed in thirdparty_open)
When compiling CoastalApp on our cluster Hydra, I created a file
envmodules_intel.hydra in folder modulefiles to set the environment for
CoastalApp. I don't know how to create a similar one on Frontera. I think
this is the first and critical issue to compile CoastalApp on Frontera.
Do you know how to find the paths of libraries for CoastalApp compilation
on Frontera or give me suggestion who I can contact with ?
I only found the environment list like below:
login2.frontera(81)$ module list
Currently Loaded Modules:
1. intel/19.1.1 3) git/2.24.1 5) python3/3.7.0 7) pmix/3.1.4 9)
xalt/2.10.34
2. impi/19.0.9 4) autotools/1.2 6) cmake/3.24.2 8) hwloc/1.11.12 10)
TACC
Thanks.
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TP2CXQRVJSXL5JB44QTWR3L5PANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Takis, thanks. I have passed some compilation of build.sh (it seems about parmetis). The error appeared are: make[1]: Entering directory `/work2/08819/jqi_capp/frontera/CoastalApp-testsuite/CoastalApp/ATMESH' c -I. -I/include/ -I/opt/apps/intel19/hdf5/1.10.4/x86_64/include/ -L. -L/lib/ -L/opt/apps/intel19/hdf5/1.10.4/x86_64/lib/ -DWITHPETLISTS_on -DESMF_VERSION_MAJOR= atmesh_mod.F90 c -I. -I/include/ -I/opt/apps/intel19/hdf5/1.10.4/x86_64/include/ -L. -L/lib/ -L/opt/apps/intel19/hdf5/1.10.4/x86_64/lib/ -DWITHPETLISTS_on -DESMF_VERSION_MAJOR= atmesh.F90 It seems there is no ESMF linked to environment like you said. Do I need to install ESMF on my local folder for CoastalApp compilation? Thanks again, |
Beta Was this translation helpful? Give feedback.
-
|
Did you see the modulefile for stampede? I have the path there for the
installed esmf library.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Wed, Jan 11, 2023 at 1:22 PM FVCOM-GitHub ***@***.***> wrote:
Takis, thanks.
I have passed some compilation of build.sh (it seems about parmetis). The
error appeared are:
make[1]: Entering directory
`/work2/08819/jqi_capp/frontera/CoastalApp-testsuite/CoastalApp/ATMESH'
c -I. -I/include/ -I/opt/apps/intel19/hdf5/1.10.4/x86_64/include/ -L.
-L/lib/ -L/opt/apps/intel19/hdf5/1.10.4/x86_64/lib/ -DWITHPETLISTS_on
-DESMF_VERSION_MAJOR= atmesh_mod.F90
make[1]: c: Command not found
make[1]: [atmesh_mod.o] Error 127 (ignored)
c -I. -I/include/ -I/opt/apps/intel19/hdf5/1.10.4/x86_64/include/ -L.
-L/lib/ -L/opt/apps/intel19/hdf5/1.10.4/x86_64/lib/ -DWITHPETLISTS_on
-DESMF_VERSION_MAJOR= atmesh.F90
make[1]: c: Command not found
make[1]: [atmesh.o] Error 127 (ignored)
It seems there is no ESMF linked to environment like you said.
Do I need to install ESMF on my local folder for CoastalApp compilation?
Thanks again,
Jianhua
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TP4XWSBZQLHTAMDE2QDWR4B5XANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, I used it: ./build.sh --compiler intel --platform stampede --component "atmesh ww3" --tp parmetis -y |
Beta Was this translation helpful? Give feedback.
-
|
and changed the line below only: module use /work/07380/panvel/Modules/modulefiles |
Beta Was this translation helpful? Give feedback.
-
|
ok, so the path is not exported to frontera?
My account on TACC is not active, I'll try to activate it and check
CoastalApp.
You might want to compile ESMF yourself.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Wed, Jan 11, 2023 at 1:59 PM FVCOM-GitHub ***@***.***> wrote:
Yes, I used it:
./build.sh --compiler intel --platform stampede --component "atmesh ww3"
--tp parmetis -y
—
Reply to this email directly, view it on GitHub
<#150 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TP2FJLN2STC3WXNM5N3WR4GLLANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis, May I know if you have records about the environment variables setup in .bashrc for building ESMF on Frontera? Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Jianhua,
No, I don't. CoastalApp defines all variables internally.
Have you tried to build it using the stampede2 modulefile?
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Fri, Jan 13, 2023 at 8:48 AM FVCOM-GitHub ***@***.***> wrote:
Hi Takis,
May I know if you have records about the environment variables setup in
.bashrc for building ESMF on Frontera?
Thanks,
Jianhua
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TP3IGF6TDL437FUXQWTWSFTLHANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis, When I tried to build ESMF on Frontera, I have to add 'export ESMF_DIR=...' in .bashrc. Otherwise the system choose gfortran as default. There are other environment variables I don't know if and how we should setup them in .bashrc. It seems there is no file envmodules_intel.stampede2 in modulefile. There is only envmodules_intel.stampede there. I have tried envmodules_intel.stampede yesterday but it doesn't work correctly. Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Sorry. 'export ESMF_COMPILER'. |
Beta Was this translation helpful? Give feedback.
-
|
There are other variables to set. Let me check on frontera in a while.
I need to deal with some other issues first here.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Fri, Jan 13, 2023 at 9:37 AM FVCOM-GitHub ***@***.***> wrote:
Sorry. 'export ESMF_COMPILER'.
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TPZEIUUPB32FXEQJBFDWSFZCTANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Jianhua hi,
Please use "stampede" as your platform in frontera. Bothe "frontera" and
"stampede"
are identically configured in terms of modules and software. In the next
update I will
rename stampede to tacc to avoid confusions.
There is though a bug in the file *modulefiles/PlatformFuncs*,
before you run the build script please fix it as:
FROM:
export netcdf="${_netcdf_env_name}"
export netcdfHOME="${_netcdf_env_name}"
export netcdf_DIR="${_netcdf_env_name}"
export netcdf_PATH="${_netcdf_env_name}"
export netcdf_ROOT="${_netcdf_env_name}"
TO:
export NETCDF="${_netcdf_env_name}"
export NETCDFHOME="${_netcdf_env_name}"
export NETCDF_DIR="${_netcdf_env_name}"
export NETCDF_PATH="${_netcdf_env_name}"
export NETCDF_ROOT="${_netcdf_env_name}"
Also add the missing cmake module in modulefiles/envmodules_intel.stampede
as:
FROM:
module purge
module load intel impi
TO:
module purge
module load cmake
module load intel impi
I compiled CoastalApp on frontera using stampede as my platform just fine.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
On Fri, Jan 13, 2023 at 9:47 AM Panagiotis Velissariou - NOAA Affiliate <
***@***.***> wrote:
… There are other variables to set. Let me check on frontera in a while.
I need to deal with some other issues first here.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
On Fri, Jan 13, 2023 at 9:37 AM FVCOM-GitHub ***@***.***>
wrote:
> Sorry. 'export ESMF_COMPILER'.
>
> —
> Reply to this email directly, view it on GitHub
> <#150 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/APC7TPZEIUUPB32FXEQJBFDWSFZCTANCNFSM6AAAAAATGEJADM>
> .
> You are receiving this because you were mentioned.Message ID:
> <noaa-ocs-modeling/CoastalApp/repo-discussions/150/comments/4679077@
> github.com>
>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis, Following your message above, I updated the files PlatformFuncs and envmodules_intel.stampede. After that, when run build.sh like bellow: ./build.sh --compiler intel --platform stampede --component "atmesh ww3" --tp parmetis -y After building parmetis, the error message appeared: compileNems :: Compiling: make -j 1 -f GNUmakefile build COMPONENTS="ATMESH WW3" c -I. -I/opt/apps/intel19/netcdf/4.6.2/x86_64/include/ -I/opt/apps/intel19/hdf5/1.10.4/x86_64/include/ -L. -L/opt/apps/intel19/netcdf/4.6.2/x86_64/lib/ -L/opt/apps/intel19/hdf5/1.10.4/x86_64/lib/ -DWITHPETLISTS_on -DESMF_VERSION_MAJOR= atmesh_mod.F90 c -I. -I/opt/apps/intel19/netcdf/4.6.2/x86_64/include/ -I/opt/apps/intel19/hdf5/1.10.4/x86_64/include/ -L. -L/opt/apps/intel19/netcdf/4.6.2/x86_64/lib/ -L/opt/apps/intel19/hdf5/1.10.4/x86_64/lib/ -DWITHPETLISTS_on -DESMF_VERSION_MAJOR= atmesh.F90 ar rv libatmesh_cap.a atmesh.o atmesh_mod.o It seems the ESMF environment setup not correct, am I right? The path of ESMF in envmodules_intel.stampede is: module load impi-intel/esmf-8.0.1. Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Jianhua,
Yes, you are not loading ESMF.
You should have these two lines in your stampede modulefile:
module use /work/07380/panvel/Modules/modulefiles
module load impi-intel/esmf-8.0.1
in order to load and use esmf.
Also, you need to modify the file modulefiles/PlatformFuncs as we discussed.
I'll send you a google invitation in a couple of minutes to try to resolve
the issues on tacc.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Wed, Jan 18, 2023 at 9:19 AM FVCOM-GitHub ***@***.***> wrote:
Hi Takis,
Following your message above, I updated the files PlatformFuncs and
envmodules_intel.stampede. After that, when run build.sh like bellow:
./build.sh --compiler intel --platform stampede --component "atmesh ww3"
--tp parmetis -y
After building parmetis, the error message appeared:
compileNems :: Compiling: make -j 1 -f GNUmakefile build
COMPONENTS="ATMESH WW3"
NOTE: Skipping appbuilder.mk creation; no appbuilder file in use.
echo 'ATMESH WW3' >
"/work2/08819/jqi_capp/frontera/CoastalApp-testsuite/CoastalApp/NEMS/src/conf/
components.mk"
NEMS_BUILDOPT IS
echo No module selected. ; cd
/work2/08819/jqi_capp/frontera/CoastalApp-testsuite/CoastalApp/ATMESH ;
exec make -f makefile.atmesh.nuopc nuopc
No module selected.
make[1]: Entering directory
`/work2/08819/jqi_capp/frontera/CoastalApp-testsuite/CoastalApp/ATMESH'
c -I. -I/opt/apps/intel19/netcdf/4.6.2/x86_64/include/
-I/opt/apps/intel19/hdf5/1.10.4/x86_64/include/ -L.
-L/opt/apps/intel19/netcdf/4.6.2/x86_64/lib/
-L/opt/apps/intel19/hdf5/1.10.4/x86_64/lib/ -DWITHPETLISTS_on
-DESMF_VERSION_MAJOR= atmesh_mod.F90
make[1]: c: Command not found
make[1]: [atmesh_mod.o] Error 127 (ignored)
c -I. -I/opt/apps/intel19/netcdf/4.6.2/x86_64/include/
-I/opt/apps/intel19/hdf5/1.10.4/x86_64/include/ -L.
-L/opt/apps/intel19/netcdf/4.6.2/x86_64/lib/
-L/opt/apps/intel19/hdf5/1.10.4/x86_64/lib/ -DWITHPETLISTS_on
-DESMF_VERSION_MAJOR= atmesh.F90
make[1]: c: Command not found
make[1]: [atmesh.o] Error 127 (ignored)
ar rv libatmesh_cap.a atmesh.o atmesh_mod.o
ar: creating libatmesh_cap.a
ar: atmesh.o: No such file or directory
make[1]: *** [libatmesh_cap.a] Error 1
make[1]: Leaving directory
`/work2/08819/jqi_capp/frontera/CoastalApp-testsuite/CoastalApp/ATMESH'
make: ***
[/work2/08819/jqi_capp/frontera/CoastalApp-testsuite/CoastalApp/ATMESH_INSTALL/
atmesh.mk] Error 2
It seems the ESMF environment setup not correct, am I right?
The path of ESMF in envmodules_intel.stampede is:
module load impi-intel/esmf-8.0.1.
Thanks,
Jianhua
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TPYUO4MZGRF5WUYRSX3WTACV5ANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Do you mean both two lines? module use /work/07380/panvel/Modules/modulefiles I'll try. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, please join the meeting, I am logged in frontera
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Wed, Jan 18, 2023 at 9:42 AM FVCOM-GitHub ***@***.***> wrote:
Do you mean both two lines?
module use /work/07380/panvel/Modules/modulefiles
module load impi-intel/esmf-8.0.1
I'll try. Thanks.
—
Reply to this email directly, view it on GitHub
<#150 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TP6Z5GVLM5DMU44U6TDWTAFMDANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
To find out quota on FRON:
To display a summary of your TACC project balances and disk quotas at any time, execute: /usr/local/etc/taccinfo
-Joseph
Y. Joseph Zhang
Web: schism.wiki
Office: 804 684 7466
From: Panagiotis Velissariou ***@***.***>
Sent: Wednesday, May 3, 2023 10:31 AM
To: noaa-ocs-modeling/CoastalApp ***@***.***>
Cc: Y. Joseph Zhang ***@***.***>; Mention ***@***.***>
Subject: Re: [noaa-ocs-modeling/CoastalApp] CoastalApp test suite (Discussion #150)
[EXTERNAL to VIMS received message]
Hi Jianhua,
I am not aware of any disk limits for the individual user.
My work directory is located at /work2/07380/panvel (plenty of space left).
I'll investigate a little more. Let me know how it goes from your side.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***<mailto:***@***.***>
On Wed, May 3, 2023 at 9:23 AM FVCOM-GitHub ***@***.***<mailto:***@***.***>> wrote:
Hi Takis,
Thanks for your information.
I run the job on the work directory. Is there space limit for development
user?
I'll try to remove files and then test it again.
Thanks again,
Jianhua
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TPYZSDCBTDUYAYYP7W3XEJS5RANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***<mailto:***@***.***>
.com>
—
Reply to this email directly, view it on GitHub<#150 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFBKNZZ5B7P564LU73IOF23XEJTY7ANCNFSM6AAAAAATGEJADM>.
You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much Joseph.
Jianhua, please try the command.
For me it reports:
Disk Usage (GB) Limit %Used File Usage Limit %Used
/work2 451.1 1024.0 44.05 467258 3000000 15.58
Again, thanks Joseph.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
On Wed, May 3, 2023 at 9:36 AM Joseph Zhang ***@***.***>
wrote:
… To find out quota on FRON:
To display a summary of your TACC project balances and disk quotas at any
time, execute: /usr/local/etc/taccinfo
-Joseph
Y. Joseph Zhang
Web: schism.wiki
Office: 804 684 7466
From: Panagiotis Velissariou ***@***.***>
Sent: Wednesday, May 3, 2023 10:31 AM
To: noaa-ocs-modeling/CoastalApp ***@***.***>
Cc: Y. Joseph Zhang ***@***.***>; Mention ***@***.***>
Subject: Re: [noaa-ocs-modeling/CoastalApp] CoastalApp test suite
(Discussion #150)
[EXTERNAL to VIMS received message]
Hi Jianhua,
I am not aware of any disk limits for the individual user.
My work directory is located at /work2/07380/panvel (plenty of space
left).
I'll investigate a little more. Let me know how it goes from your side.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***<mailto:***@***.***>
On Wed, May 3, 2023 at 9:23 AM FVCOM-GitHub ***@***.***<mailto:***@***.***>>
wrote:
> Hi Takis,
>
> Thanks for your information.
>
> I run the job on the work directory. Is there space limit for
development
> user?
>
> I'll try to remove files and then test it again.
>
> Thanks again,
> Jianhua
>
> —
> Reply to this email directly, view it on GitHub
> <
#150 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/APC7TPYZSDCBTDUYAYYP7W3XEJS5RANCNFSM6AAAAAATGEJADM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***<mailto:***@***.***>
> .com>
>
—
Reply to this email directly, view it on GitHub<
#150 (comment)>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AFBKNZZ5B7P564LU73IOF23XEJTY7ANCNFSM6AAAAAATGEJADM>.
You are receiving this because you were mentioned.Message ID:
***@***.******@***.***>>
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TP3NJFLX4HSLU27PV33XEJUQDANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis and Joseph, Thank you. The balances for my account look like: login1.frontera(1001)$ /usr/local/etc/taccinfo
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Jianhua,
15% is for the number of files limit not the storage.
As for the storage you are using 56.23 % (/work2)
Did you delete any files before getting the quota?
It seems there is more than enough space to run the simulations.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Wed, May 3, 2023 at 11:00 AM FVCOM-GitHub ***@***.***> wrote:
Hi Takis and Joseph,
Thank you. The balances for my account look like:
login1.frontera(1001)$ /usr/local/etc/taccinfo
--------------------- Project balances for user jqi_capp
----------------------
| Name Avail SUs Expires | |
| OCE22003 167606 2023-08-31 | |
------------------------ Disk quotas for user jqi_capp
------------------------
| Disk Usage (GB) Limit %Used File Usage Limit %Used |
| /home1 0.2 25.0 0.69 10223 200000 5.11 |
| /work2 575.8 1024.0 56.23 451674 3000000 15.06 |
| /scratch1 0.0 0.0 0.00 1 0 0.00 |
| /scratch2 0.0 0.0 0.00 1 0 0.00 |
| /scratch3 0.0 0.0 0.00 1 0 0.00 |
Only 15% used.
Best,
Jianhua
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TPYD3C3LWDCAVST4N7DXEJ6HVANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis, I didn't remove any file yet. Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Jianhua,
Later today, I will run an FVCOM test case on Frontera and I'll let you
know how it went.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Wed, May 3, 2023 at 11:33 AM FVCOM-GitHub ***@***.***> wrote:
Hi Takis,
I didn't remove any file yet.
Thanks,
Jianhua
—
Reply to this email directly, view it on GitHub
<#150 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TP677XPAVLXWY5H4UBDXEKCEFANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis, I didn't include FVCOM yet. Could you help to first run a case with component = "atmesh ww3" and send to me for reference? Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Jianhua,
Yes, I will do this.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Wed, May 3, 2023 at 11:41 AM FVCOM-GitHub ***@***.***> wrote:
Hi Takis,
I didn't include FVCOM yet. Could you help to first run a case with
component = "atmesh ww3" and send to me for reference?
Thanks,
Jianhua
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TPYZQJ5K2ALFK3J37XTXEKDDLANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
@jianhua Qi ***@***.***> Jianhua, we have a new feature/FVCOM branch in
CoastalApp (I deleted the old one). This branch
contains the basic infrastructure to compile and run FVCOM through
CoastalApp. It is still in development, so I'll
keep all posted as its development progresses.
Soon this branch will be merged with develop (feature/FVCOM was created
based on the latest develop branch).
Tomorrow, I'll try to run the atm+ww3 case on Frontera to see if I get the
error messages you are getting.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Wed, May 3, 2023 at 11:41 AM FVCOM-GitHub ***@***.***> wrote:
Hi Takis,
I didn't include FVCOM yet. Could you help to first run a case with
component = "atmesh ww3" and send to me for reference?
Thanks,
Jianhua
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TPYZQJ5K2ALFK3J37XTXEKDDLANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
@jianhua Qi ***@***.***> Hi Jianhua. The issues you faced in frontera
using the updated CoastalApp module system is contributed to two factors:
a) trying to run a simulation (e.g., florence_hsofs.atm2adc
or florence_hsofs.atm2adc2ww3) that requires a lot of cores that span more
than
10 compute nodes in most cases will fail (I checked this with intel
=19). For example, in florence_hsofs.atm2adc test case
the slurm run script requires 500 cores that can be supplied in theory
by 9 nodes (9 nodes x 56 cores/node = 504 cores total). In this case the
test
case will run. Most often these 500 cores will span more than 10 nodes
(node core sharing) and the simulation may run or crash.
I have tested using a minimum of 10 nodes, with mixed results (runs,
crashes). Using a minimum of 12 nodes the simulation always crashes.
If I remember correctly, we didn't have this issue with Intel v18
compilers.
I don't know if this is due to cluster misconfiguration, or
intentional to limit users utilize extended resources without explicitly
requesting so from IT.
On "hera" we don't have this problem (we have run all the test cases
successfully, but I will check again).
Anyway, these are my findings on this issue.
b) The issue you faced with the "No space left on device" error message, is
contributed to the ucx communication library, contained
in the new system of modules and libraries CoastalApp is now using on
Frontera and other systems (the version of ucx I am using is 1.13).
On "hera" and on "orion" it works but not on Frontera. I have fixed
that by removing this library and using the system's ucx library.
I have run the test cases on Frontera and all run just fine now. I am in
the process of updating CoastalApp-testsuite to use the proper number of
cores/nodes
in the test cases.
@y. Joseph Zhang ***@***.***> Joseph, have you faced this issue ((a))
with SCHISM?
Thanks
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
On Wed, May 3, 2023 at 11:44 PM Panagiotis Velissariou - NOAA Affiliate <
***@***.***> wrote:
… @jianhua Qi ***@***.***> Jianhua, we have a new feature/FVCOM branch
in CoastalApp (I deleted the old one). This branch
contains the basic infrastructure to compile and run FVCOM through
CoastalApp. It is still in development, so I'll
keep all posted as its development progresses.
Soon this branch will be merged with develop (feature/FVCOM was created
based on the latest develop branch).
Tomorrow, I'll try to run the atm+ww3 case on Frontera to see if I get the
error messages you are getting.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
On Wed, May 3, 2023 at 11:41 AM FVCOM-GitHub ***@***.***>
wrote:
> Hi Takis,
>
> I didn't include FVCOM yet. Could you help to first run a case with
> component = "atmesh ww3" and send to me for reference?
>
> Thanks,
> Jianhua
>
> —
> Reply to this email directly, view it on GitHub
> <#150 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/APC7TPYZQJ5K2ALFK3J37XTXEKDDLANCNFSM6AAAAAATGEJADM>
> .
> You are receiving this because you were mentioned.Message ID:
> <noaa-ocs-modeling/CoastalApp/repo-discussions/150/comments/5793463@
> github.com>
>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis, Thanks for your information. I'll try again and let you know soon. Jianhua |
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis, Today I cloned CoastalApp and CoastalApp-testsuite on FRONTERA. The CoastalApp can be built using component = "atmesh ww3" without error. But when trying to submit job, it seems the job didn't on the queue. Please see the screen output bellow. My question is: Is this the final code or I have to wait for your upload? Thanks, login1.frontera(1091)$ run_all.sh --mod CoastalApp/modulefiles/envmodules_intel.tacc --accnt jqi_capp --queue normal The following variables are defined:
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Jianhua,
On Frontera you need to use ibrun instead of srun. Please set BATCH_RUNEXE
= ibrun, either from the command line or from the environment file. Run
run_all.sh --help to get all options available to the script. Let me know
how it goes.
Takis
On Friday, May 19, 2023, FVCOM-GitHub ***@***.***> wrote:
Hi Takis,
Today I cloned CoastalApp and CoastalApp-testsuite on FRONTERA. The
CoastalApp can be built using component = "atmesh ww3" without error. But
when trying to submit job, it seems the job didn't on the queue. Please see
the screen output bellow.
My question is: Is this the final code or I have to wait for your upload?
Thanks,
Jianhua
login1.frontera(1091)$ run_all.sh --mod
CoastalApp/modulefiles/envmodules_intel.tacc --accnt jqi_capp --queue normal
The following variables are defined:
ACCEPT_ALL = 0
MOD_FILE = CoastalApp/modulefiles/envmodules_intel.tacc
BIN_DIR =
/work2/08819/jqi_capp/frontera/CoastalApp_20230519/CoastalApp-testsuite/CoastalApp/ALLBIN_INSTALL
COM_DIR =
/work2/08819/jqi_capp/frontera/CoastalApp_20230519/CoastalApp-testsuite/comm
WORK_DIR =
/work2/08819/jqi_capp/frontera/CoastalApp_20230519/CoastalApp-testsuite/work
TESTS_FILE =
/work2/08819/jqi_capp/frontera/CoastalApp_20230519/CoastalApp-testsuite/regtest_list.dat
BATCH_SYSTEM = slurm
BATCH_ACCOUNT = jqi_capp
BATCH_QUEUE = normal
BATCH_SLURM_OPTS =
BATCH_PBS_OPTS =
BATCH_RUNEXE = srun
Are these values correct? [y/n]: y
run_all.sh: Running all tests in:
/work2/08819/jqi_capp/frontera/CoastalApp_20230519/CoastalApp-testsuite/regtest_list.dat
Running case: ike_shinnecock.atm2ww3
login1.frontera(1092)$ squeue -u jqi_capp
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
login1.frontera(1093)$
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.<
https://ci6.googleusercontent.com/proxy/uEwggxp_rd6UF_-cfTzmzrFdxWY7DMPxMrGVrIFA0X8KRDrQRiqxxGsGDkuxo6D50v5Nb8U3cHKYfhQYVWu6j94C9p9mZxo_Cy2db-Bu8J8Dng85Q_e8w9YhTndCVkCo_8p_7-hskvnrSzaiIYSg-v-DATk44clMCTZ0wn0jBy_E4wuqHRXslurFwL1e5ggjio072b4z3SY2ONkeup3eqKCVjBhyo-aF1QSAn3xUSBQY8EPkoKyAT1bmHGH7=s0-d-e1-ft#https://github.com/notifications/beacon/APC7TPYZQZMQUADOEHGRXR3XG66N5A5CNFSM6AAAAAATGEJADOWGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQALLIMG.gif>Message
ID: <noaa-ocs-modeling/CoastalApp/repo-discussions/150/comments/
***@***.***>
…--
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis, I followed your suggestion to set BATCH_RUNEXE = ibrun. The model still cannot run. The case I tested is ike_shinnecock.atm2ww3. The model should first run model_setup_slurm.job to get the file mod_def.ww3 and nest.ww3 but I didn't find them in work/ike_shinnecock.atm2ww3/run. Hope to get your help. Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Takis, Following your instruction, I have successfully repeated the test suite cases ike_shinnecock.atm2ww3, scituateharbor.fvc and scituateharbor.atm2fvc2ww3 on FRONTERA. The only I modified is in file scituateharbor.atm2fvc2ww3/run/config.rc: Changed: I noticed the file sciww3_forcing.nc I created from WRF contains some variables for baroclinc model of FVCOM but they are not included in ATMESH source code (see atmesh_mod.F90). Only uwnd, vwnd and P are used. I'll try to modify FVCOM cap to use ATMESH wind forcing in FVCOM. Thanks a lot, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Jianhua,
Thank you for testing. From our side testing on these tests was all
successful.
Yes, in config.rc we may use for the FVCOM case(s):
atm_dir: input
atm_nam: sciww3_forcing.nc
ATMESH expects three fields to be present in the input data file (NetCDF):
uwnd, vwnd, and P
Also ATMESH reads "ADCIRC" type of meshed data, this should not be a
problem as the input NetCDF
file is created properly.
Ufuk will start working on FVCOM/ufs-coastal shortly, that is why he
requested this test case.
Takis
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell: (205) 227-9141
email: ***@***.***
…On Mon, Jun 5, 2023 at 3:38 PM FVCOM-GitHub ***@***.***> wrote:
Hi Takis,
Following your instruction, I have successfully repeated the test suite
cases ike_shinnecock.atm2ww3, scituateharbor.fvc and
scituateharbor.atm2fvc2ww3 on FRONTERA.
The only I modified is in file scituateharbor.atm2fvc2ww3/run/config.rc:
Changed:
atm_nam: sciww3_forcing.nc
with:
atm_nam: ./input/sciww3_forcing.nc
I noticed the file sciww3_forcing.nc I created from WRF contains some
variables for baroclinc model of FVCOM but they are not included in ATMESH
source code (see atmesh_mod.F90). Only uwnd, vwnd and P are used.
I'll try to modify FVCOM cap to use ATMESH wind forcing in FVCOM.
Thanks a lot,
Jianhua
—
Reply to this email directly, view it on GitHub
<#150 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APC7TPYN76TAJFEGPOZEEOLXJY7S7ANCNFSM6AAAAAATGEJADM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
CoastalApp test suite
@pvelissariou1 @aliabdolali @platipodium @FVCOM-GitHub @GM-Ling @josephzhang8 @sarriegi
This discussion will be dedicated to CoastalApp test suite moving forward. The current repo at https://github.com/noaa-ocs-modeling/CoastalApp-testsuite just got updated. Here are what included:
in this current version of the test cases the main goal is to address the multi-node execution of CoastalApp mainly for ADCIRC and WW3 with forcing from ATMesh and PaHM.
SCHISM and FVCOM components forced by ATMesh need to be tested for similar test cases. Please follow the similar test folder structure and naming. Please submit pull request to be included in the https://github.com/noaa-ocs-modeling/CoastalApp-testsuite repo asap.
Self contained Shinnecock inlet test cases:
SHINNECOCK: FLORENCE/SANDY
For these test cases you don't need any data files if you follow Readme, you should be all set.
For HSOFS 1.8M node mesh, you need to download data and pass the env variable for the location following this Readme. Here is the list of HSOFS tests.
HSOFS: FLORENCE/SANDY
Remember to comment out the tests you do not want to run in this file.
Beta Was this translation helpful? Give feedback.
All reactions