Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

120 m and 250 m meshes running WITH ADCIRCPY #56

Closed
saeed-moghimi-noaa opened this issue Mar 29, 2021 · 37 comments
Closed

120 m and 250 m meshes running WITH ADCIRCPY #56

saeed-moghimi-noaa opened this issue Mar 29, 2021 · 37 comments

Comments

@saeed-moghimi-noaa
Copy link

@JaimeCalzadaNOAA @zacharyburnettNOAA
I suggest to get both 120 m and 250 m meshes running with ADCIRCPY and perhaps best track. Then switch back to nems. For this purpose and to make sure we isolate things. I suggest using the original f14 and f13.
Any refactoring for mesh object then happen after the basic run went through.

@ghost ghost self-assigned this Mar 31, 2021
@ghost
Copy link

ghost commented Mar 31, 2021

ADCIRC-only configuration working folder:

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc

@ghost
Copy link

ghost commented Mar 31, 2021

I've built a system to store / read configurations to JSON files

@saeed-moghimi-noaa
Copy link
Author

@JaimeCalzadaNOAA @zacharyburnettNOAA

Hi Zach and Jaime,

As the next step lets do:
Run a stand alone ADCIRC using adcircpy python client files for both hsofs_250m and hsofs_120m using sandy or Florence best track.

run cases:

  • spin up
  • tide only
  • best track run

@ghost
Copy link

ghost commented Mar 31, 2021

I will be out Thursday and Friday, but https://github.com/noaa-ocs-modeling/CoupledModelDriver/blob/main/README.md#usage has information on how to use the new JSON configuration system if you'd like to try it out

@saeed-moghimi-noaa
Copy link
Author

@zacharyburnettNOAA @JaimeCalzadaNOAA
For 120m testing use this mesh:
https://drive.google.com/drive/folders/1B3ian2KUQlVPeVjnVV73MCsusJhLjXV8?usp=sharing

Make sure to include:
fort.15
fort.14
fort.13
fort.24

Thanks,

@saeed-moghimi-noaa
Copy link
Author

@JaimeCalzadaNOAA @zacharyburnettNOAA

image

@ghost ghost mentioned this issue Apr 7, 2021
@ghost ghost added the adcirc-v55-hsofs label Apr 8, 2021
@ghost
Copy link

ghost commented Apr 8, 2021

updated to adcircpy v1.0.29 and generated configurations for the HSOFS 120m and 250m meshes for running ADCIRC with NEMS coupled with ATMESH, TPXO, and WW3DATA, in the following directories:

/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210408_hsofs_120m
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210408_hsofs_250m

both are currently running coldstart (have completed the mesh decomposition step)

@ghost
Copy link

ghost commented Apr 8, 2021

MESH DECOMPOSITION

NEMS HSOFS 120m mesh run

mesh decomposition seems to have completed successfully (there are 598 PE* folders, as requested). However, the error file (/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210408_hsofs_120m/coldstart/ADC_MESH_DECOMP.err.log) has the following:

srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 17760852.1 ON h8c54 CANCELLED AT 2021-04-08T16:58:52 ***
slurmstepd: error: *** JOB 17760852 ON h8c54 CANCELLED AT 2021-04-08T16:58:52 ***
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
adcprep            00000000006A051E  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B33B326D630  Unknown               Unknown  Unknown
adcprep            000000000061EFFD  decomp_                   225  decomp.F
adcprep            000000000042DE16  prepinput_                624  adcprep.F
adcprep            000000000042BB2B  MAIN__                    239  adcprep.F
adcprep            000000000040B65E  Unknown               Unknown  Unknown
libc-2.17.so       00002B33B36A0555  __libc_start_main     Unknown  Unknown
adcprep            000000000040B569  Unknown               Unknown  Unknown

NEMS HSOFS 250m mesh run

mesh decomposition completed successfully (there are 598 PE* folders, as requested) with no errors in /scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210408_hsofs_250m/coldstart/ADC_MESH_DECOMP.err.log

ADCIRC-only HSOFS 250m mesh run

mesh decomposition failed (no PE* folders created), and the output log (/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210408_hsofs_250m/coldstart/ADC_MESH_DECOMP.out.log) has the following at the end (line 47):

ERROR: NetCDF input or output files were specified.
 but adcprep was not compiled with NetCDF support.
 Please recompile adcprep with NetCDF libraries.

I will look into rebuilding adcprep correctly.

@saeed-moghimi-noaa
Copy link
Author

saeed-moghimi-noaa commented Apr 8, 2021

(coupledmodeldriver) Saeed.Moghimi@hfe08:/scratch2/COASTAL/coastal/save/shared/saeed/adcircpy$ python example_3_250m.py 
Traceback (most recent call last):
  File "/scratch2/COASTAL/coastal/save/shared/saeed/adcircpy/example_3_250m.py", line 85, in <module>
    main()
  File "/scratch2/COASTAL/coastal/save/shared/saeed/adcircpy/example_3_250m.py", line 48, in main
    slurm = SlurmConfig(
TypeError: __init__() missing 1 required positional argument: 'partition'

@ghost
Copy link

ghost commented Apr 8, 2021 via email

@saeed-moghimi-noaa
Copy link
Author

saeed-moghimi-noaa commented Apr 8, 2021

@JaimeCalzadaNOAA Please check:

ADCIRCPY - adcprep error on 250m case see here:
/scratch2/COASTAL/coastal/save/shared/saeed/adcircpy/outputs/example_3_250m/coldstart/

Lmod has detected the following error: The following module(s) are unknown:
"netcdf/4.7.2-parallel"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "netcdf/4.7.2-parallel"

Also make sure that all modulefiles written in TCL start with the string
#%Module



_______________________________________________________________
Start Epilog v20.08.28 on node h2c48 for job 17762725 :: Thu Apr 8 17:59:48 UTC 2021
Job 17762725 (not serial) finished for user Saeed.Moghimi in partition hera with exit code 1:0
_______________________________________________________________
End Epilogue v20.08.28 Thu Apr 8 17:59:48 UTC 2021
 INFO: Processing --np
 INFO: Processing --partmesh
 File                                                      fort.14
 WAS FOUND!  Opening & Processing file

 from alloc_main1: 
  
 memory currently allocated =   1347642720  bytes
 memory high water mark     =  -1649704876  bytes
  
  
 memory currently allocated =   1441352512  bytes
 memory high water mark     =   1535062304  bytes
  
 Global Grid file read successfully.
INFO: This mesh has 7698 weir node pairs.
INFO: Maximum number of duals for any weir node is 2.
 maximum co-nodes for any node =           21
 edge count =      5385262
 
 Grid Partition Data
 METIS 4.0 will require approximately    604899540  bytes
 Total Edges Cut =      1708037
INFO: Writing mesh partition to partmesh.txt.
 INFO: METIS has partitioned nodes successfully.
  
 memory currently allocated =   1441352512  bytes
 memory high water mark     =   1535062304  bytes
  
 INFO: Processing --np
 INFO: Processing --prepall
 File                                                      fort.14
 WAS FOUND!  Opening & Processing file

 File fort.15                                                     
 WAS FOUND!  Opening & Processing file

 Elevation Station Locations contained in fort.15
 Velocity Station Locations Contained in fort.15
forrtl: severe (64): input conversion error, unit 15, file /scratch2/COASTAL/coastal/save/shared/saeed/adcircpy/outputs/example_3_250m/coldstart/fort.15
Image              PC                Routine            Line        Source             
adcprep            00000000005EFCBE  Unknown               Unknown  Unknown
adcprep            0000000000625130  Unknown               Unknown  Unknown
adcprep            00000000004070E0  presizes_mp_sizeu        1263  presizes.F
adcprep            0000000000428ECF  prepinput_                436  adcprep.F
adcprep            000000000042798B  MAIN__                    239  adcprep.F
adcprep            000000000040355E  Unknown               Unknown  Unknown
libc-2.17.so       00002B4EA7A1A555  __libc_start_main     Unknown  Unknown
adcprep            0000000000403469  Unknown               Unknown  Unknown
_______________________________________________________________
Start Epilog v20.08.28 on node h16c53 for job 17766649 :: Thu Apr 8 20:20:08 UTC 2021
Job 17766649 (not serial) finished for user Saeed.Moghimi in partition hera with exit code 64:0
_______________________________________________________________
End Epilogue v20.08.28 Thu Apr 8 20:20:08 UTC 2021

@ghost
Copy link

ghost commented Apr 8, 2021 via email

@saeed-moghimi-noaa
Copy link
Author

saeed-moghimi-noaa commented Apr 8, 2021

I am using consistent adcprep and padcirc now. I am not sure if this is related to nstcdf parallel.

@JaimeCalzadaNOAA
I pasted the Error and code here. Tomorrow, lets you share your screen and walk us through the process on Orion or Hera.

#!/bin/bash --login
#SBATCH -D .
#SBATCH -J example_3_250m.py
#SBATCH -A coastal
#SBATCH --mail-type=all
#SBATCH [email protected]
#SBATCH --output=example_3_250m.log
#SBATCH -n 1000
#SBATCH --time=08:00:00
#  #SBATCH --partition=

ulimit -s unlimited
set -e

source /scratch2/COASTAL/coastal/save/shared/saeed/ADC-WW3-NWM-NEMS/modulefiles/envmodules_intel.hera

PATH=/scratch2/COASTAL/coastal/save/shared/saeed/ADC-WW3-NWM-NEMS/ADCIRC/work/:$PATH

main() {
  SECONDS=0
  run_coldstart_phase
  if grep -Rq "ERROR: Elevation.gt.ErrorElev, ADCIRC stopping." example_3_250m.log; then
    duration=$SECONDS
    echo "ERROR: Elevation.gt.ErrorElev, ADCIRC stopping."
    echo "Wallclock time: $(($duration / 60)) minutes and $(($duration % 60)) seconds."
    exit -1
  else
    run_hotstart_phase
    duration=$SECONDS
    if grep -Rq "ERROR: Elevation.gt.ErrorElev, ADCIRC stopping." example_3_250m.log; then
      echo "ERROR: Elevation.gt.ErrorElev, ADCIRC stopping."
      echo "Wallclock time: $(($duration / 60)) minutes and $(($duration % 60)) seconds."
      exit -1
    fi
  fi
  echo "Wallclock time: $(($duration / 60)) minutes and $(($duration % 60)) seconds."
}

run_coldstart_phase() {
  rm -rf coldstart
  mkdir coldstart
  cd coldstart
  ln -sf ../fort.14
  ln -sf ../fort.13
  ln -sf ../fort.15.coldstart ./fort.15
  adcprep --np $SLURM_NTASKS --partmesh
  adcprep --np $SLURM_NTASKS --prepall
  srun padcirc 
  clean_directory
  cd ..
}

run_hotstart_phase() {
  rm -rf hotstart
  mkdir hotstart
  cd hotstart
  ln -sf ../fort.14
  ln -sf ../fort.13
  ln -sf ../fort.15.hotstart ./fort.15
  ln -sf ../coldstart/fort.67.nc
  adcprep --np $SLURM_NTASKS --partmesh
  adcprep --np $SLURM_NTASKS --prepall
  srun padcirc 
  clean_directory
  cd ..
}

clean_directory() {
  rm -rf PE*
  rm -rf partmesh.txt
  rm -rf metis_graph.txt
  rm -rf fort.13
  rm -rf fort.14
  rm -rf fort.15
  rm -rf fort.16
  rm -rf fort.80
  rm -rf fort.68.nc
}

main
Error:

> Lmod has detected the following error: The following module(s) are unknown:
"netcdf/4.7.2-parallel"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "netcdf/4.7.2-parallel"

Also make sure that all modulefiles written in TCL start with the string
#%Module



_______________________________________________________________
Start Epilog v20.08.28 on node h2c17 for job 17766614 :: Thu Apr 8 20:18:02 UTC 2021
Job 17766614 (not serial) finished for user Saeed.Moghimi in partition hera with exit code 1:0
_______________________________________________________________
End Epilogue v20.08.28 Thu Apr 8 20:18:02 UTC 2021
 INFO: Processing --np
 INFO: Processing --partmesh
 File                                                      fort.14
 WAS FOUND!  Opening & Processing file

 from alloc_main1: 
  
 memory currently allocated =   1347642720  bytes
 memory high water mark     =  -1649704876  bytes
  
  
 memory currently allocated =   1441352512  bytes
 memory high water mark     =   1535062304  bytes
  
 Global Grid file read successfully.
INFO: This mesh has 7698 weir node pairs.
INFO: Maximum number of duals for any weir node is 2.
 maximum co-nodes for any node =           21
 edge count =      5385262
 
 Grid Partition Data
 METIS 4.0 will require approximately    604899540  bytes
 Total Edges Cut =      1708037
INFO: Writing mesh partition to partmesh.txt.
 INFO: METIS has partitioned nodes successfully.
  
 memory currently allocated =   1441352512  bytes
 memory high water mark     =   1535062304  bytes
  
 INFO: Processing --np
 INFO: Processing --prepall
 File                                                      fort.14
 WAS FOUND!  Opening & Processing file

 File fort.15                                                     
 WAS FOUND!  Opening & Processing file

 Elevation Station Locations contained in fort.15
 Velocity Station Locations Contained in fort.15
forrtl: severe (64): input conversion error, unit 15, file /scratch2/COASTAL/coastal/save/shared/saeed/adcircpy/outputs/example_3_250m_v2/coldstart/fort.15
Image              PC                Routine            Line        Source             
adcprep            00000000005EFCBE  Unknown               Unknown  Unknown
adcprep            0000000000625130  Unknown               Unknown  Unknown
adcprep            00000000004070E0  presizes_mp_sizeu        1263  presizes.F
adcprep            0000000000428ECF  prepinput_                436  adcprep.F
adcprep            000000000042798B  MAIN__                    239  adcprep.F
adcprep            000000000040355E  Unknown               Unknown  Unknown
libc-2.17.so       00002ADB10157555  __libc_start_main     Unknown  Unknown
adcprep            0000000000403469  Unknown               Unknown  Unknown
_______________________________________________________________
Start Epilog v20.08.28 on node h16c53 for job 17766648 :: Thu Apr 8 20:19:38 UTC 2021
Job 17766648 (not serial) finished for user Saeed.Moghimi in partition hera with exit code 64:0
_______________________________________________________________
End Epilogue v20.08.28 Thu Apr 8 20:19:38 UTC 2021
 INFO: Processing --np
 INFO: Processing --partmesh
 File                                                      fort.14
 WAS FOUND!  Opening & Processing file

 from alloc_main1: 
  
 memory currently allocated =   1347642720  bytes
 memory high water mark     =  -1649704876  bytes
  
  
 memory currently allocated =   1441352512  bytes
 memory high water mark     =   1535062304  bytes
  
 Global Grid file read successfully.
INFO: This mesh has 7698 weir node pairs.
INFO: Maximum number of duals for any weir node is 2.
 maximum co-nodes for any node =           21
 edge count =      5385262
 
 Grid Partition Data
 METIS 4.0 will require approximately    604899540  bytes
 Total Edges Cut =      1708037
INFO: Writing mesh partition to partmesh.txt.
 INFO: METIS has partitioned nodes successfully.
  
 memory currently allocated =   1441352512  bytes
 memory high water mark     =   1535062304  bytes
  
 INFO: Processing --np
 INFO: Processing --prepall
 File                                                      fort.14
 WAS FOUND!  Opening & Processing file

 File fort.15                                                     
 WAS FOUND!  Opening & Processing file

 Elevation Station Locations contained in fort.15
 Velocity Station Locations Contained in fort.15
forrtl: severe (64): input conversion error, unit 15, file /scratch2/COASTAL/coastal/save/shared/saeed/adcircpy/outputs/example_3_250m_v2/coldstart/fort.15
Image              PC                Routine            Line        Source             
adcprep            0000000000697BCE  Unknown               Unknown  Unknown
adcprep            00000000006CE960  Unknown               Unknown  Unknown
adcprep            000000000040F010  presizes_mp_sizeu        1263  presizes.F
adcprep            000000000042CD6F  prepinput_                436  adcprep.F
adcprep            000000000042B82B  MAIN__                    239  adcprep.F
adcprep            000000000040B35E  Unknown               Unknown  Unknown
libc-2.17.so       00002B0F374E0555  __libc_start_main     Unknown  Unknown
adcprep            000000000040B269  Unknown               Unknown  Unknown
_______________________________________________________________
Start Epilog v20.08.28 on node h3c02 for job 17770446 :: Thu Apr 8 22:18:39 UTC 2021
Job 17770446 (not serial) finished for user Saeed.Moghimi in partition hera with exit code 64:0
_______________________________________________________________
End Epilogue v20.08.28 Thu Apr 8 22:18:39 UTC 2021
#! /usr/bin/env python
"""
This example recreates the Shinnecock Inlet test case with some added
improvements in order to demonstrate some of the capabilities of AdcircPy.

In contrast to example_1, this example generates input files that are separated
by a coldstart and hotstart phase.

The behaviour of this program is similar to the example_1.
"""

from datetime import datetime, timedelta
import pathlib
import tarfile
import tempfile
import urllib.request

from adcircpy import AdcircMesh, AdcircRun, Tides
from adcircpy.server import SlurmConfig
from adcircpy.forcing.winds import BestTrackForcing

PARENT = pathlib.Path(__file__).parent.absolute()
FORT14 = "/scratch2/COASTAL/coastal/save/shared/models/meshes/hsofs/250m/v1.0/fort.14"
FORT13 = "/scratch2/COASTAL/coastal/save/shared/models/meshes/hsofs/250m/v1.0/fort.13"

def main():

    # open mesh file
    mesh = AdcircMesh.open(FORT14, crs=4326)
    
    # init tidal forcing and setup requests
    tidal_forcing = Tides(tidal_source = 'TPXO' , resource = '/scratch2/COASTAL/coastal/save/shared/models/forcings/tides/h_tpxo9.v1.nc')
    tidal_forcing.use_all()

    mesh.add_forcing(tidal_forcing)
   
    # Add wind forcing to model
    #wind_forcing = BestTrackForcing('Sandy2012')
    #mesh.add_forcing(wind_forcing)

    # import fort.13
    mesh.import_nodal_attributes(FORT13)
    # activate fort.13
    for name in mesh.get_nodal_attribute_names():
        mesh.set_nodal_attribute_state(name,True,True)    
    
    # instantiate AdcircRun object.
    slurm = SlurmConfig(
        account='coastal',
        ntasks=1000,
        run_name='example_3_250m.py',
        partition='',
        walltime=timedelta(hours=8),
        mail_type='all',
        mail_user='[email protected]',
        log_filename='example_3_250m.log',
        modules=['intel/2020', 'impi/2020', 'netcdf/4.7.2-parallel'],
        path_prefix='/scratch2/COASTAL/coastal/save/shared/repositories/adcirc-cg/work/'
    )
    
    now = datetime.utcnow()
    
    driver = AdcircRun(
        mesh,
        start_date = now,
        end_date   = now + timedelta(days=10),
        spinup_time=timedelta(days=15),
        server_config=slurm
    )

    # Tweak V55 defualt
    # William suggested :TODO check from f15 again
    # tweak parmeter to get IM = 511112 and A0,B0,C0 = 0.0,1.0,0.0  for explicit scheme
    # driver.gwce_solution_scheme = 'explicit'

    # legacy (sergey)
    # tweak parmeter to get IM = 511111 and A0,B0,C0 = 0.35,0.3,0.35  for explicit scheme
    driver.gwce_solution_scheme = 'semi-implicit-legacy'
    
    # Write driver state to file.
    driver.write("outputs/example_3_250m_v2", overwrite=True)


if __name__ == '__main__':
    main()

@jreniel
Copy link

jreniel commented Apr 9, 2021

@saeed-moghimi-noaa cc. @zacharyburnettNOAA

There are three problems with your setup:

  1. If you don't wish to specify a partition, instead of doing partition='' simply remove the line or pass a null object (None), because an empty string ('') is treated as a valid object.
  2. In order to resolve the error forrtl: severe (64): input conversion error, unit 15, file [...] please check your timestep.
  3. Verify that the modules you are passing to SlurmConfig are the same ones that were used to compile the ADCIRC binaries.

To verify please resolve these in your client file and submit your job file through the debug queue with sbatch --qos=debug --time=00:30:00 slurm.job.

@ghost
Copy link

ghost commented Apr 9, 2021

@JaimeCalzadaNOAA @saeed-moghimi-noaa you can use the following commands with coupledmodeldriver to generate ADCIRC-only and NEMS+ADCIRC configurations

ADCIRC-only configuration with coupledmodeldriver:

initialize_adcirc \
  --output-directory run_adcirconly_hsofs_250m \
  --mesh-directory /scratch2/COASTAL/coastal/save/shared/models/meshes/hsofs/250m/v1.0 \
  --modeled-start-time 20121022T060000 \
  --modeled-duration 04:05:00:00 \
  --modeled-timestep 00:00:02 \
  --tidal-spinup-duration 12:06:00:00 \
  --platform HERA \
  --adcirc-processors 600 \
  --adcirc-executable /scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/ADCIRC/work/padcirc \
  --adcprep-executable /scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/ADCIRC/work/adcprep \
  --modulefile /scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/modulefiles/envmodules_intel.hera \
  --job-duration 06:00:00 \
  --forcings tidal \
  --tidal-source TPXO \
  --tidal-path /scratch2/COASTAL/coastal/save/shared/models/forcings/tides/h_tpxo9.v1.nc
generate_adcirc run_adcirconly_hsofs_250m

NEMS + ADCIRC configuration with coupledmodeldriver:

initialize_adcirc \
  --output-directory run_nemsadcirc_hsofs_250m \
  --mesh-directory /scratch2/COASTAL/coastal/save/shared/models/meshes/hsofs/250m/v1.0 \
  --modeled-start-time 20121022T060000 \
  --modeled-duration 04:05:00:00 \
  --modeled-timestep 00:00:02 \
  --tidal-spinup-duration 12:06:00:00 \
  --platform HERA \
  --adcirc-processors 598 \
  --adcirc-executable /scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/NEMS/exe/NEMS.x \
  --adcprep-executable /scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/ADCIRC/work/adcprep \
  --modulefile /scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/modulefiles/envmodules_intel.hera \
  --job-duration 06:00:00 \
  --forcings tidal,atmesh,ww3data \
  --tidal-source TPXO \
  --tidal-path /scratch2/COASTAL/coastal/save/shared/models/forcings/tides/h_tpxo9.v1.nc \
  --atmesh-path /scratch2/COASTAL/coastal/save/shared/models/forcings/hsofs/sandy/Wind_HWRF_SANDY_Nov2018_ExtendedSmoothT.nc \
  --ww3data-path /scratch2/COASTAL/coastal/save/shared/models/forcings/hsofs/sandy/ww3.HWRF.NOV2018.2012_sxy.nc \
  --nems-interval 01:00:00
generate_adcirc run_nemsadcirc_hsofs_250m

@ghost
Copy link

ghost commented Apr 12, 2021

I've updated the modeled start and end times to align with the ATMESH and WW3DATA forcings:

Sandy ATMESH:
    start_date:  22-Oct-2012 06:00:00
    stop_date:   26-Oct-2012 11:00:00
datetime.timedelta(days=4, hours=5)
Sandy WW3DATA
    start_date:  2012-10-22 06:00:00
    stop_date:   2012-11-01 23:00:00
datetime.timedelta(days=10, hours=17)

here are the most recent model runs:

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210412_hsofs_120m
/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210412_hsofs_250m
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210412_hsofs_120m
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210412_hsofs_250m

@ghost
Copy link

ghost commented Apr 12, 2021

currently running jobs:

 ➜ squeue -u Zachary.Burnett -o "%.8i %3C %4D %97Z %15j" --sort i
   JOBID CPU NODE WORK_DIR                                                                                          NAME
17908675 600 15   /scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210412_hsofs_250m/runs/run_1      ADC_HOT_RUN

17908681 600 15   /scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210412_hsofs_250m/runs/run_1 ADC_HOT_RUN

17909270 600 15   /scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210412_hsofs_120m/coldstart       ADC_COLD_RUN
17909271 1   1    /scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210412_hsofs_120m/runs/run_1      ADC_MESH_DECOMP
17909272 600 15   /scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210412_hsofs_120m/runs/run_1      ADC_HOT_RUN

17909274 600 15   /scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210412_hsofs_120m/coldstart  ADC_COLD_RUN
17909275 1   1    /scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210412_hsofs_120m/runs/run_1 ADC_MESH_DECOMP
17909276 600 15   /scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210412_hsofs_120m/runs/run_1 ADC_HOT_RUN

@saeed-moghimi-noaa
Copy link
Author

@zacharyburnettNOAA
Zach,
I do not have permission to see the content of the folders.
-Saeed

@ghost
Copy link

ghost commented Apr 12, 2021

@zacharyburnettNOAA
Zach,
I do not have permission to see the content of the folders.
-Saeed

Fixed, I gave ugo+rw

@ghost
Copy link

ghost commented Apr 13, 2021

ADCIRC only

looks like the hotstart runs of both ADCIRC-only 120m and ADCIRC-only 250m runs elapsed the 6 hour job run time and were cancelled, as shown in /scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210412_hsofs_250m/runs/run_1/ADC_HOT_RUN.err.log:

slurmstepd: error: *** STEP 17908675.0 ON h1c01 CANCELLED AT 2021-04-13T01:09:28 DUE TO TIME LIMIT ***
slurmstepd: error: *** JOB 17908675 ON h1c01 CANCELLED AT 2021-04-13T01:09:28 DUE TO TIME LIMIT ***
forrtl: error (78): process killed (SIGTERM)

I will run them again with 12 hour time limits.

Also, there are a bunch of empty core.***** files dumped to the hotstart directory.

NEMS + ADCIRC

however, the NEMS + ADCIRC coldstart run failed after a while with the following error in /scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210412_hsofs_250m/coldstart/ADC_COLD_RUN.err.log:

Attempting to use an MPI routine after finalizing MPI
Attempting to use an MPI routine after finalizing MPI
Attempting to use an MPI routine after finalizing MPI
Attempting to use an MPI routine after finalizing MPI
Attempting to use an MPI routine after finalizing MPI
...
Attempting to use an MPI routine after finalizing MPI
Attempting to use an MPI routine after finalizing MPI
Attempting to use an MPI routine after finalizing MPI
Attempting to use an MPI routine after finalizing MPI
srun: error: h23c39: tasks 520-558: Exited with exit code 1
srun: launch/slurm: _step_signal: Terminating StepId=17908679.0
srun: error: h17c02: tasks 240-279: Exited with exit code 1
srun: error: h9c31: tasks 160-199: Exited with exit code 1
srun: error: h2c53: tasks 0-39: Exited with exit code 1
srun: error: h17c50: tasks 280-319: Exited with exit code 1
srun: error: h22c03: tasks 400-439: Exited with exit code 1
srun: error: h12c52: tasks 200-239: Exited with exit code 1
srun: error: h9c01: tasks 120-159: Exited with exit code 1
srun: error: h3c39: tasks 40-79: Exited with exit code 1
srun: error: h18c48: tasks 320-359: Exited with exit code 1
srun: error: h22c45: tasks 440-479: Exited with exit code 1
srun: error: h21c39: tasks 360-399: Exited with exit code 1
srun: error: h5c06: tasks 80-119: Exited with exit code 1
srun: error: h23c45: tasks 559-597: Exited with exit code 1
srun: error: h22c46: tasks 480-519: Exited with exit code 1

@saeed-moghimi-noaa
Copy link
Author

saeed-moghimi-noaa commented Apr 13, 2021

Please compare with cases here:
/scratch2/COASTAL/coastal/noscrub/shared/Saeed.Moghimi/shared_with_zach/flo

Perhaps identical information with this run so we can compare:
/scratch2/COASTAL/coastal/noscrub/shared/Saeed.Moghimi/shared_with_zach/flo/a50_FLO_ATM2OCN_v2.2_extended/rt_20210104_h19_m40_s18r604

Take station list from here:

/scratch2/COASTAL/coastal/noscrub/shared/Saeed.Moghimi/shared_with_zach/flo/a10_FLO_OCN_SPINUP_v1.0/rt_20210104_h19_m07_s33r618/scr/fort.15.template.tide_spinup

spin up


0 0.0 0 0                ! NOUTV,TOUTSV,TOUTFV,NSPOOLV:VEL STATION OUTPUT INFO (UNIT  62)
0                        ! NSTAV - NUMBER OF ELEVATION RECORDING STATIONS, FOLLOWED BY LOCATIONS ON PROCEEDING LINES
-5 0.0 366.0 1800        ! NOUTGE,TOUTSGE,TOUTFGE,NSPOOLGE : GLOBAL ELEVATION OUTPUT INFO (UNIT  63)
-5 0.0 366.0 1800        ! NOUTGV,TOUTSGV,TOUTFGV,NSPOOLGV : GLOBAL VELOCITY  OUTPUT INFO (UNIT  64)
0                        ! NHARFR - NUMBER OF CONSTITUENTS TO BE INCLUDED IN THE HARMONIC ANALYSIS
0.0  0.0  0  0.0         ! THAS,THAF,NHAINC,FMV - HARMONIC ANALYSIS PARAMETERS
0 0 0 0                  ! NHASE,NHASV,NHAGE,NHAGV - CONTROL HARMONIC ANALYSIS AND OUTPUT TO UNITS 51,52,53,54
5 $hot_ndt_out                 ! NHSTAR,NHSINC - HOT START FILE GENERATION PARAMETERS,2days
1  0  1.E-6  25 0        ! ITITER, ISLDIA, CONVCR, ITMAX, ILUMP - ALGEBRAIC SOLUTION PARAMETERS
HSOFS V1.0.5             ! NCPROJ - PROJECT TITLE
CSDL/CMMB - NWS/NHC      ! NCINST - PROJECT INSTITUTION
CRAY CX40 on WCOSS       ! NCSOUR - PROJECT SOURCE
DEVELOPMENT              ! NCHIST - PROJECT HISTORY
http://www.adcirc.org    ! NCREF  - PROJECT REFERENCES
HSOFS V1.0.0             ! NCCOM  - PROJECT COMMENTS
CSDL/CMMD - NWS/NHC      ! NCHOST - PROJECT HOST
CF-1.0                   ! NCCONV - CONVENTIONS
[email protected]  ! NCCONT - CONTACT INFORMATION
$start_date

@ghost
Copy link

ghost commented Apr 13, 2021

added besttrack support in 6cf7f3b

@ghost
Copy link

ghost commented Apr 16, 2021

@saeed-moghimi-noaa in the directory

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210416_hsofs_250m_v1.0_besttrack/coldstart

here are the sizes of the output NetCDFs:

 ➜ du *.6*.nc
548     fort.61.nc
1005536 fort.63.nc
44380   fort.64.nc
44392   fort.67.nc
44392   fort.68.nc
44440   maxele.63.nc
44444   maxvel.63.nc

@ghost
Copy link

ghost commented Apr 16, 2021

running the command plot_fort61 from ADCIRCpy, as follows:

(CoupledModelDriver) C:\Repositories\CoupledModelDriver >
 ➜ plot_fort61 run_20210416_hsofs_250m_v1.0_besttrack\coldstart\fort.61.nc MSL

returns the following error:

Http Error: 400 Client Error: Bad Request for url: https://api.tidesandcurrents.noaa.gov:443/api/prod/datagetter?station=8459479&begin_date=20121010+01%3A
00&end_date=20121016+22%3A00&product=water_level&datum=MSL&units=metric&time_zone=gmt&format=json&application=noaa%2Fnos%2Fcsdl%2Fadcircpy
...
Http Error: 400 Client Error: Bad Request for url: https://api.tidesandcurrents.noaa.gov:443/api/prod/datagetter?station=9650593&begin_date=20121010+01%3A
00&end_date=20121016+22%3A00&product=water_level&datum=MSL&units=metric&time_zone=gmt&format=json&application=noaa%2Fnos%2Fcsdl%2Fadcircpy
Fail to allocate bitmap

going to https://api.tidesandcurrents.noaa.gov/api/prod/datagetter?station=8459479&begin_date=20121010+01%3A00&end_date=20121016+22%3A00&product=water_level&datum=MSL&units=metric&time_zone=gmt&format=json&application=noaa%2Fnos%2Fcsdl%2Fadcircpy returns the error JSON:

{"error": {"message":" Wrong Datum: No valid datum value for MSL ***station=8459479"}}

which datum should I use for these stations? The choices in adcircpy are

'MHHW', 'MHW', 'MTL', 'MSL', 'MLW', 'MLLW', 'NAVD88', 'STND'

EDIT: none of the choices worked, I'll take a look at what's wrong on Monday

@saeed-moghimi-noaa
Copy link
Author

saeed-moghimi-noaa commented Apr 16, 2021

@zacharyburnettNOAA

Try Ncveiw

module load intel
module load netcdf
module load ncview

ncview fort.61.nc

click on zeta
click inside figure
on the pop up figure change X axis from station to time

you can click on the first figure to get multiple time series

ncveiw

@ghost
Copy link

ghost commented Apr 16, 2021

running the command plot_fort61 from ADCIRCpy, as follows:

(CoupledModelDriver) C:\Repositories\CoupledModelDriver >
 ➜ plot_fort61 run_20210416_hsofs_250m_v1.0_besttrack\coldstart\fort.61.nc MSL

returns the following error:

Http Error: 400 Client Error: Bad Request for url: https://api.tidesandcurrents.noaa.gov:443/api/prod/datagetter?station=8459479&begin_date=20121010+01%3A
00&end_date=20121016+22%3A00&product=water_level&datum=MSL&units=metric&time_zone=gmt&format=json&application=noaa%2Fnos%2Fcsdl%2Fadcircpy
...
Http Error: 400 Client Error: Bad Request for url: https://api.tidesandcurrents.noaa.gov:443/api/prod/datagetter?station=9650593&begin_date=20121010+01%3A
00&end_date=20121016+22%3A00&product=water_level&datum=MSL&units=metric&time_zone=gmt&format=json&application=noaa%2Fnos%2Fcsdl%2Fadcircpy
Fail to allocate bitmap

going to https://api.tidesandcurrents.noaa.gov/api/prod/datagetter?station=8459479&begin_date=20121010+01%3A00&end_date=20121016+22%3A00&product=water_level&datum=MSL&units=metric&time_zone=gmt&format=json&application=noaa%2Fnos%2Fcsdl%2Fadcircpy returns the error JSON:

{"error": {"message":" Wrong Datum: No valid datum value for MSL ***station=8459479"}}

which datum should I use for these stations? The choices in adcircpy are

'MHHW', 'MHW', 'MTL', 'MSL', 'MLW', 'MLLW', 'NAVD88', 'STND'

EDIT: none of the choices worked, I'll take a look at what's wrong on Monday

Some stations might simply not have data.

Call the CLI using the --coops-only flag. Any stations that fails to gather COOPS data will be skipped.

@ghost
Copy link

ghost commented Apr 19, 2021

250m mesh

in the hotstart run (/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210416_hsofs_250m_v1.0_besttrack/runs/run_1), the run stops with the following error:

ERROR: check_err: NetCDF: Index exceeds dimension bound
INFO: netcdfTerminate: ADCIRC Terminating.

ADC_HOT_RUN.out.log:

 INFO: Searching for ADCIRC subdomain directories:
 INFO: Looking for './PE0000/fort.14' ...
 INFO: File './PE0000/fort.14' was found!
 INFO: The search for the subdomain directory was completed successfully.
 INFO: The ROOTDIR is  '.'.
 INFO: The INPUTDIR is './PE0000'.
 INFO: The GBLINPUTDIR is '.'.
 INFO: The GLOBALDIR is '.'.
 INFO: The LOCALDIR is './PE0000'.

 _______________________________________________________________________________


                   PROGRAM ADCIRC   VERSION v55.00-31-g7428737

     AN ADVANCED CIRCULATION MODEL FOR SHELVES, COASTAL SEAS AND ESTUARIES


       -  DEVELOPED BY

          R.A. LUETTICH, JR
            UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL
            INSTITUTE OF MARINE SCIENCES

          J.J. WESTERINK
            DEPARTMENT OF CIVIL ENGINEERING AND GEOLOGICAL SCIENCES
            UNIVERSITY OF NOTRE DAME


       -  THE ADCIRC SOURCE CODE IS COPYRIGHTED BY

          R.A. LUETTICH, JR. AND J.J. WESTERINK, 1994-2006

       NO PART OF THIS CODE MAY BE REPRODUCED OR REDISTRIBUTED
          WITHOUT THE WRITTEN PERMISSION OF THE AUTHORS



 _______________________________________________________________________________
INFO: initializeMesh: THE NEIGHBOR TABLE IS BEING COMPUTED.
INFO: initializeMesh: THE NEIGHBOR TABLE IS COMPLETED. THE MINIMUM NUMBER OF NEIGHBORS FOR ANY NODE = 3. 1+THE MAXIMUM NUMBER OF NEIGHBORS FOR ANY NODE = 8. THE PARAMETER MNEI CAN BE SET AS SMALL AS 8.
ADCIRC Version is v55.00-31-g7428737
ERROR: check_err: NetCDF: Index exceeds dimension bound
INFO: netcdfTerminate: ADCIRC Terminating.
_______________________________________________________________
Start Epilog v20.08.28 on node h1c20 for job 18046902 :: Fri Apr 16 19:04:38 UTC 2021
Job 18046902 (not serial) finished for user Zachary.Burnett in partition hera with exit code 137:0
_______________________________________________________________
End Epilogue v20.08.28 Fri Apr 16 19:04:38 UTC 2021

ADC_HOT_RUN.err.log:

application called MPI_Abort(comm=0x84000002, 322) - process 322
application called MPI_Abort(comm=0x84000002, 567) - process 567
application called MPI_Abort(comm=0x84000002, 122) - process 122
application called MPI_Abort(comm=0x84000002, 11) - process 11
application called MPI_Abort(comm=0x84000002, 14) - process 14
...
application called MPI_Abort(comm=0x84000002, 36) - process 36
application called MPI_Abort(comm=0x84000002, 37) - process 37
slurmstepd: error: *** STEP 18046902.0 ON h1c20 CANCELLED AT 2021-04-16T19:04:35 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
application called MPI_Abort(comm=0x84000002, 327) - process 327
application called MPI_Abort(comm=0x84000002, 328) - process 328
...
application called MPI_Abort(comm=0x84000002, 152) - process 152
application called MPI_Abort(comm=0x84000002, 156) - process 156
srun: error: h1c33: tasks 40-79: Killed
srun: launch/slurm: _step_signal: Terminating StepId=18046902.0
srun: error: h6c14: tasks 440-479: Killed
srun: error: h6c04: tasks 240-279: Killed
srun: error: h6c07: tasks 320-359: Killed
srun: error: h6c15: tasks 480-519: Killed
srun: error: h6c03: tasks 200-239: Killed
srun: error: h1c37: tasks 80-119: Killed
srun: error: h1c20: tasks 0-39: Killed
srun: error: h6c05: tasks 280-319: Killed
srun: error: h6c10: tasks 360-399: Killed
srun: error: h1c42: tasks 160-199: Killed
srun: error: h6c19: tasks 520-559: Killed
srun: error: h6c13: tasks 400-439: Killed
srun: error: h1c38: tasks 120-159: Killed
srun: error: h6c22: tasks 560-599: Killed

120m mesh

The 120m mesh (/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210416_hsofs_120m_v2.2_besttrack/runs/run_1) completed the run with no errors:

fort.61.nc:

image

@saeed-moghimi-noaa
Copy link
Author

As for 120m Good NEWS thanks!

For 250 m, I did not see this error before. May be @jreniel or @WPringle can take a look. -Saeed

@WPringle
Copy link
Contributor

@zacharyburnettNOAA That error is related to the hot start file. I think this usually occurs if there is no data in the fort.67.nc or fort.68.nc. So it means no data was written out into hot start during cold start run or it was deleted and new one made by adcprep before the hot start.

@ghost
Copy link

ghost commented Apr 19, 2021

@zacharyburnettNOAA That error is related to the hot start file. I think this usually occurs if there is no data in the fort.67.nc or fort.68.nc. So it means no data was written out into hot start during cold start run or it was deleted and new one made by adcprep before the hot start.

good catch, it looks like the fort.67.nc file (/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210416_hsofs_250m_v1.0_besttrack/coldstart/fort.67.nc) has a time array of length 0.

print({variable_name: variable.shape for variable_name, variable in fort67_dataset.variables.items()})
{
    'time': (0,),
    'x': (1813443,),
    'y': (1813443,),
    'element': (3564104, 3),
    'adcirc_mesh': (1,),
    'neta': (),
    'nvdll': (1,),
    'max_nvdll': (),
    'ibtypee': (1,),
    'nbdv': (186,),
    'nvel': (),
    'nvell': (186,),
    'max_nvell': (),
    'ibtype': (186,),
    'nbvv': (55274,),
    'depth': (1813443,),
    'zeta1': (0, 1813443),
    'zeta2': (0, 1813443),
    'zetad': (0, 1813443),
    'u-vel': (0, 1813443),
    'v-vel': (0, 1813443),
    'nodecode': (0, 1813443),
    'noff': (0, 3564104),
    'imhs': (),
    'iths': (),
    'iestp': (),
    'nscoue': (),
    'ivstp': (),
    'nscouv': (),
    'ipstp': (),
    'iwstp': (),
    'nscoum': (),
    'igep': (),
    'nscouge': (),
    'igvp': (),
    'nscougv': (),
    'igpp': (),
    'igwp': (),
    'nscougw': (),
}

I will look into the start and end times for output again.

@ghost
Copy link

ghost commented Apr 19, 2021

here are the last few lines of /scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210419_hsofs_250m_v1.0_besttrack/coldstart/fort.15, which produces the error:

110                                                             ! ANGINN
0 0 0 0                                                         ! NOUTE TOUTSE TOUTFE NSPOOLE
0                                                               ! NSTAE
0 0 0 0                                                         ! NOUTV TOUTSV TOUTFV NSPOOLV
0                                                               ! NSTAV
-5 0.000000 12.250000 1800                                      ! NOUTGE TOUTSGE TOUTFGE NSPOOLGE
-5 0.000000 12.250000 1800                                       ! NOUTGV TOUTSGV TOUTFGV NSPOOLGV
0                                                               ! NFREQ
0 0 0 0                                                         ! THAS THAF NHAINC FMV
0 0 0 0                                                         ! NHASE NHASV NHAGE NHAGV
5 529200                                                        ! NHSTAR NHSINC
1 0 1E-08 25                                                    ! ITITER ISLDIA CONVCR ITMAX
                                                                ! NCPROJ
                                                                ! NCINST
                                                                ! NCSOUR
                                                                ! NCHIST
                                                                ! NCREF
                                                                ! NCCOM
                                                                ! NCHOST
                                                                ! NCONV
                                                                ! NCCONT
2012-10-10 00:00                                                ! Forcing start date / NCDATE

here are the fort.67.nc timestep variables:

5 529200                                                        ! NHSTAR NHSINC

@WPringle
Copy link
Contributor

yes the NHSINC = 5929200 needs to be correct. Does that line up with the timestep and when you want to output the hot start? Hotstart output interval in seconds = NHSINC*TIMESTEP

@saeed-moghimi-noaa
Copy link
Author

saeed-moghimi-noaa commented Apr 19, 2021

Hi William @WPringle

The 250 m blows up in the cold start part.

ERR

 TIME STEP =  300043     ITERATIONS =    0     TIME =  0.60008600E+06
  ELMAX =  9.5955E+002 AT NODE  1399321  SPEEDMAX =  8.5055E+002 AT NODE  1390920  ON MYPROC =  520   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  300044     ITERATIONS =    0     TIME =  0.60008800E+06
  ELMAX =  9.8648E+002 AT NODE  1399321  SPEEDMAX =  8.9732E+002 AT NODE  1390920  ON MYPROC =  520   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  300045     ITERATIONS =    0     TIME =  0.60009000E+06
  ELMAX =  1.0094E+003 AT NODE  1399321  SPEEDMAX =  9.4873E+002 AT NODE  1390920  ON MYPROC =  520   ** WARNING: Elevation.gt.WarnElev **
 TIME STEP =  300045     ITERATIONS =    0     TIME =  0.60009000E+06
  ELMAX =  1.0094E+003 AT NODE  1399321  SPEEDMAX =  9.4873E+002 AT NODE  1390920  ON MYPROC =  520
  ** ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. **
srun: error: h10c18: tasks 120-159: Exited with exit code 1
srun: launch/slurm: _step_signal: Terminating StepId=18132011.0
srun: error: h10c05: tasks 40-79: Exited with exit code 1
srun: error: h10c21: tasks 160-199: Exited with exit code 1
srun: error: h10c40: tasks 200-239: Exited with exit code 1
srun: error: h15c28: tasks 280-319: Exited with exit code 1
srun: error: h10c42: tasks 240-279: Exited with exit code 1
srun: error: h15c37: tasks 320-359: Exited with exit code 1
srun: error: h23c19: tasks 560-599: Exited with exit code 1
srun: error: h23c03: tasks 400-439: Exited with exit code 1
srun: error: h10c12: tasks 80-119: Exited with exit code 1
srun: error: h23c17: tasks 480-519: Exited with exit code 1
srun: error: h23c16: tasks 440-479: Exited with exit code 1
srun: error: h20c41: tasks 360-399: Exited with exit code 1
srun: error: h23c18: tasks 520-559: Exited with exit code 1
srun: error: h5c27: tasks 0-39: Exited with exit code 1

fort.15

created on 2021-04-19 16:09                                     ! RUNDES
NOMAD mesh v1e MSL                                              ! RUNID
1                                                               ! NFOVER
1                                                               ! NABOUT
100                                                             ! NSCREEN
0                                                               ! IHOT
2                                                               ! ICS
111112                                                          ! IM
1                                                               ! NOLIBF
2                                                               ! NOLIFA
1                                                               ! NOLICA
1                                                               ! NOLICAT
5                                                               ! NWP
mannings_n_at_sea_floor                                        
primitive_weighting_in_continuity_equation                     
surface_canopy_coefficient                                     
surface_directional_effective_roughness_length                 
surface_submergence_state                                      
1                                                               ! NCOR
1                                                               ! NTIP
0                                                               ! NWS
1                                                               ! NRAMP
9.81                                                            ! gravitational acceleration
-3                                                              ! TAU0
2.000000                                                        ! DTDP
0                                                               ! STATIM
0                                                               ! REFTIM
12.25                                                           ! RNDAY
12.25                                                           ! DRAMP
0 1 0                                                           ! A00 B00 C00
0.01 0 0 0.01                                                   ! H0 ? ? VELMIN
-80.9048 30.2847                                                ! SLAM0 SFEA0
0.0025                                                          ! FFACTOR
10                                                              ! ESL - LATERAL EDDY VISCOSITY COEFFICIENT
0                                                               ! CORI
8                                                               ! NTIF
Q1
0.019256 6.49585E-05 0.695 0.91654 34.5375
...
1.55785958e-03 2.47790552e+01
1.15620391e-03 1.83903869e+01
7.68806691e-04 1.22285113e+01
110                                                             ! ANGINN
0 0 0 0                                                         ! NOUTE TOUTSE TOUTFE NSPOOLE
0                                                               ! NSTAE
0 0 0 0                                                         ! NOUTV TOUTSV TOUTFV NSPOOLV
0                                                               ! NSTAV
-5 0.000000 12.250000 1800                                      ! NOUTGE TOUTSGE TOUTFGE NSPOOLGE
-5 0.000000 12.250000 1800                                       ! NOUTGV TOUTSGV TOUTFGV NSPOOLGV
0                                                               ! NFREQ
0 0 0 0                                                         ! THAS THAF NHAINC FMV
0 0 0 0                                                         ! NHASE NHASV NHAGE NHAGV
5 529200                                                        ! NHSTAR NHSINC
1 0 1E-08 25                                                    ! ITITER ISLDIA CONVCR ITMAX
                                                                ! NCPROJ
                                                                ! NCINST
                                                                ! NCSOUR
                                                                ! NCHIST
                                                                ! NCREF
                                                                ! NCCOM
                                                                ! NCHOST
                                                                ! NCONV
                                                                ! NCCONT
2012-10-10 00:00                                                ! Forcing start date / NCDATE

EDIT: shortened constituent lines to make it easier to scroll through

@WPringle
Copy link
Contributor

oh right. yeah that's just a typical blowup.

I highly recommend using smagorinsky model by changing ESL from 10 to -0.2. You need to use IM = 511112, instead of 111112 when you do this. Also ICS=22 is better than ICS=2: technically ICS=2 is simply incorrect, but not that much difference for this domain.
I've been running the 250m mesh with 4.6514 sec time step using these settings and never seen a blow up.

@saeed-moghimi-noaa
Copy link
Author

saeed-moghimi-noaa commented Apr 19, 2021 via email

@ghost
Copy link

ghost commented Apr 20, 2021

@WPringle thanks, those suggestions really helped. The configuration I made from your suggestions put out a fort.67.nc that has data in it, so the hotstart run started successfully.

For reference, I used the following JSON configuration:

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210420_hsofs_250m_v1.0/configure_adcirc.json
{
  "adcirc_executable_path": "/scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/ADCIRC/work/padcirc",
  "adcprep_executable_path": "/scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/ADCIRC/work/adcprep",
  "modeled_start_time": "2012-10-22 06:00:00",
  "modeled_end_time": "2012-10-26 11:00:00",
  "modeled_timestep": 2.0,
  "fort_13_path": "/scratch2/COASTAL/coastal/save/shared/models/meshes/hsofs/250m/v1.0/fort.13",
  "fort_14_path": "/scratch2/COASTAL/coastal/save/shared/models/meshes/hsofs/250m/v1.0/fort.14",
  "tidal_spinup_duration": 1058400.0,
  "tidal_spinup_timestep": 2.0,
  "gwce_solution_scheme": "explicit",
  "use_smagorinsky": true,
  "horizontal_eddy_viscosity": null,
  "source_filename": "/scratch2/COASTAL/coastal/save/shared/repositories/ADC-WW3-NWM-NEMS/modulefiles/envmodules_intel.hera",
  "use_original_mesh": false,
  "output_surface": true,
  "surface_output_interval": 3600.0,
  "output_stations": false,
  "stations_file_path": null,
  "stations_output_interval": 360.0,
  "output_spinup": true,
  "output_elevations": true,
  "output_velocities": true,
  "output_concentrations": false,
  "output_meteorological_factors": false,
  "processors": 600,
  "nems_parameters": {}
}

to produce the following fort.15 for coldstart with generate_adcirc:

/scratch2/COASTAL/coastal/save/shared/working/zach/adcirc/run_20210420_hsofs_250m_v1.0/coldstart/fort.15
created on 2021-04-20 15:58                                     ! RUNDES                              - 32 CHARACTER ALPHANUMERIC RUN DESCRIPTION
NOMAD mesh v1e MSL                                              ! RUNID                               - 24 CHARACTER ALPANUMERIC RUN IDENTIFICATION
1                                                               ! NFOVER                              - NONFATAL ERROR OVERRIDE OPTION
1                                                               ! NABOUT                              - ABREVIATED OUTPUT OPTION PARAMETER
100                                                             ! NSCREEN                             - UNIT 6 OUTPUT OPTION PARAMETER
0                                                               ! IHOT                                - HOT START PARAMETER
2                                                               ! ICS                                 - COORDINATE SYSTEM SELECTION PARAMETER
511112                                                          ! IM                                  - MODEL SELECTION PARAMETER
1                                                               ! NOLIBF                              - BOTTOM FRICTION TERM SELECTION PARAM; before NWP==1, '2' was used
2                                                               ! NOLIFA                              - FINITE AMPLITUDE TERM SELECTION PARAMETER
1                                                               ! NOLICA                              - SPATIAL DERIVATIVE CONVECTIVE SELECTION PARAMETER
1                                                               ! NOLICAT                             - TIME DERIVATIVE CONVECTIVE TERM SELECTION PARAMETER
5                                                               ! NWP                                 - VARIABLE BOTTOM FRICTION AND LATERAL VISCOSITY OPTION PARAMETER; default 0
mannings_n_at_sea_floor
primitive_weighting_in_continuity_equation
surface_canopy_coefficient
surface_directional_effective_roughness_length
surface_submergence_state
1                                                               ! NCOR                                - VARIABLE CORIOLIS IN SPACE OPTION PARAMETER
1                                                               ! NTIP                                - TIDAL POTENTIAL OPTION PARAMETER
0                                                               ! NWS                                 - WIND STRESS AND BAROMETRIC PRESSURE OPTION PARAMETER
1                                                               ! NRAMP                               - RAMP FUNCTION OPTION
9.81                                                            ! G                                   - ACCELERATION DUE TO GRAVITY - DETERMINES UNITS
-3                                                              ! TAU0                                - WEIGHTING FACTOR IN GWCE; original, 0.005
2.000000                                                        ! DTDP                                - TIME STEP (IN SECONDS)
0                                                               ! STATIM                              - STARTING TIME (IN DAYS)
0                                                               ! REFTIM                              - REFERENCE TIME (IN DAYS)
12.25                                                           ! RNDAY                               - TOTAL LENGTH OF SIMULATION (IN DAYS)
12.25                                                           ! DRAMP                               - DURATION OF RAMP FUNCTION (IN DAYS)
0 1 0                                                           ! A00 B00 C00                         - TIME WEIGHTING FACTORS FOR THE GWCE EQUATION
0.01 0 0 0.01                                                   ! H0 NODEDRYMIN NODEWETRMP VELMIN
-80.9048 30.2847                                                ! SLAM0 SFEA0                         - CENTER OF CPP PROJECTION (NOT USED IF ICS=1, NTIP=0, NCOR=0)
0.0025                                                          ! FFACTOR
-0.2                                                            ! smagorinsky coefficient             - LATERAL EDDY VISCOSITY COEFFICIENT; IGNORED IF NWP =1
0                                                               ! CORI                                - CORIOLIS PARAMETER - IGNORED IF NCOR = 1
8                                                               ! NTIF                                - NUMBER OF TIDAL POTENTIAL CONSTITUENTS BEING FORCED starting 2008082300
...
...
...
110                                                             ! ANGINN                              - INNER ANGLE THRESHOLD
0 0 0 0                                                         ! NOUTE TOUTSE TOUTFE NSPOOLE         - ELEV STATION OUTPUT INFO (UNIT 61)
0                                                               ! NSTAE                               - TOTAL NUMBER OF ELEVATION RECORDING STATIONS
0 0 0 0                                                         ! NOUTV TOUTSV TOUTFV NSPOOLV         - VELOCITY STATION OUTPUT INFO (UNIT 62)
0                                                               ! NSTAV                               - TOTAL NUMBER OF VELOCITY RECORDING STATIONS
-5 0.000000 12.250000 1800                                      ! NOUTGE TOUTSGE TOUTFGE NSPOOLGE     - GLOBAL ELEVATION OUTPUT INFO (UNIT 63)
-5 0.000000 0.000000 1800                                       ! NOUTGV TOUTSGV TOUTFGV NSPOOLGV     - GLOBAL VELOCITY OUTPUT INFO (UNIT 64)
0                                                               ! NFREQ
0 0 0 0                                                         ! THAS THAF NHAINC FMV                - HARMONIC ANALYSIS PARAMETERS
0 0 0 0                                                         ! NHASE NHASV NHAGE NHAGV             - CONTROL HARMONIC ANALYSIS AND OUTPUT TO UNITS 51,52,53,54
5 529200                                                        ! NHSTAR NHSINC                       - HOT START FILE GENERATION PARAMETERS
1 0 1E-08 25                                                    ! ITITER ISLDIA CONVCR ITMAX          - ALGEBRAIC SOLUTION PARAMETERS
                                                                ! NCPROJ                              - PROJECT TITLE
                                                                ! NCINST                              - PROJECT INSTITUTION
                                                                ! NCSOUR                              - PROJECT SOURCE
                                                                ! NCHIST                              - PROJECT HISTORY
                                                                ! NCREF                               - PROJECT REFERENCES
                                                                ! NCCOM                               - PROJECT COMMENTS
                                                                ! NCHOST                              - PROJECT HOST
                                                                ! NCONV                               - CONVENTIONS
                                                                ! NCCONT                              - CONTACT INFORMATION
2012-10-10 00:00                                                ! NCDATE                              - forcing start date

The ESLM parameter seems to have been set automatically to -0.2 by adcircpy after I specified use_smagorinsky=true.

@ghost
Copy link

ghost commented Apr 21, 2021

basic configurations ran successfully with 250m and 120m in ADCIRC-only

focusing now on using coupled NEMS #86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants