Replicate NHC-collaboration, ensemble perturbation test case for hurricane florence

[FariborzDaneshvar-NOAA]

Directory on PW clustre: /home/Fariborz.Daneshvar/nhc_colab_test/florence_2018

Notebooks:
setup_n_run.ipynb: to create ensembles of perturbed tracks (based on OFCL track), generate SCHISM inputs, and run them.
post_ensemble.ipynb: for post processing and plotting

Follow the instructions from this issue

[saeed-moghimi-noaa]

@FariborzDaneshvar-NOAA Please update the status

[FariborzDaneshvar-NOAA]

The following issues were identified while running the setup_n_run.ipyn notebook:

1. Restarting the (NHC COLABORATION) cluster will revoke permission of the other users to access/create files (even in their home directories).

2. Absolute to relative path conversion issue during the generation of SCHISM configurations:
   The notebook worked fine with the default paths (in /lustre/hurricanes/...) for generating SCHISM files, but it failed with a path to home directory (i.e. /home/Fariborz.Daneshvar/...) with a FileNotFoundError for accessing hgrid.grd file as executing the generate_schism_configuration() function. @SorooshMani-NOAA created a ticket for this issue. Needs more investigation (in progress...)

3. Missing maxelev_###### files for some ensemble runs:
   Six out of 15 runs (for an ensemble of 15 perturbed tracks) are missing maxelev_###### files in their outputs/ directories. Based on the check_completion command, five out of these six runs (including original, 6, 7, 9, and 13) finished with 99.98% (instead of 100%) completion, while run#8 with 100% completion was also missing these outputs. So it might/not be related to the status of runs! As a result, the combine_results command did not generate maxele.63.nc file, so could not run the post_ensemble.ipynb notebook. Needs more investigation (in progress...)

[FariborzDaneshvar-NOAA]

The alternate approach was using the ondemand-storm-workflow singularity image (instead of notebooks).
I ran it for 50 ensembles on NHC_COLAB_2 cluster and unlike the previous approach (with the notebooks), all runs completed successfully.
Run_directory: /lustre/hurricanes/florence_2018_Fariborz_v1_50_ensembles

Review of outputs revealed an issue with the perturbed tracks (see issue #74 for more detail). I will re-run the model again once Soroosh resolves this issue.

I got an error for the run with BEST track (see issue #19)

[saeed-moghimi-noaa]

Great to hear that. thanks for the update! :-)

[FariborzDaneshvar-NOAA]

Update:
the perturb track issue (mentioned above) was resolved, and I ran the model with the updated singularity image:
directory: /lustre/hurricanes/florence_2018_Fariborz_v2_10_ensembles

However, review of analysis outputs revealed an inconsistency between model and surrogate validation plots (see an example below).

To ensure that it's not due to limited number of ensembles (10), I re-ran it with 39 ensembles:
New_directory: /lustre/hurricanes/florence_2018_Fariborz_v3_39_ensembles

But still got the same patterns! Will discuss it with Soroosh and William during the Ensemble Analysis meeting.

[FariborzDaneshvar-NOAA]

Update:
William pointed to the _analyze function in analyze_ensemble.py, noting that node_status_mask should be None instead of always_wet !

Soroosh updated the singularity image, and I re-ran only the analyze part of the workflow.sh, but was still getting the same results. (here is an example of new analysis with k=1, p=1, and n=0.025).

It should be noted that in all three test runs, there was only one eigenvalue!

Review of maxele.63.nc (./setup/ensemble.dir/analyze/maxele.63.nc) does not show high elevations associated with the storm track, suggesting that storm track was not included in the SCHISM runs! (see the screenshot below).

Will discuss it with the rest of the team during the Thursday tag-up meeting.

[SorooshMani-NOAA]

@FariborzDaneshvar-NOAA as we discussed, I believe this should have something to do with the track input. Let's talk with @pvelissariou1 in the next meeting. My guess is that the "faked" BEST track has some missing information.

This also could be something related to how SCHISM is compiled in the container (less likely).

Actually let's try running an actual best track as well before we ask Takis.

[pvelissariou1]

Soroosh, Fariborz hi, I can schedule a meeting on Friday if you wish so. Takis

On Wednesday, July 26, 2023, Soroosh Mani ***@***.***> wrote: @FariborzDaneshvar-NOAA as we discussed, I believe this should have

something to do with the track input. Let's talk with @pvelissariou1 in the next meeting. My guess is that the "faked" BEST track has some missing information.

This also could be something related to how SCHISM is compiled in the

container (less likely).

Actually let's try running an actual best track as well before we ask

Takis.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.<

https://ci4.googleusercontent.com/proxy/WvQeyKr0hkIPZI8yNbDLY6qMVUy9nuygjKdlDRlsp8Y8btUlLI6iyO1x6kLxirD2etlF5XDNfgiidYTaYMgQNvfXEbnLcxUHo-DhBw-jW-MDNwMCLuL2IHFx8bjeP00A2YdbkCoU9X-IGEXMi5VReCuJUWq0jO9RtLWDP7LryGKekXjkUXLQ45HdZB5VtaTPtAAolMX-2G9jena-n5lhRwDJXsGHHrX9Uut3lNggMmPqQLJuYQ=s0-d-e1-ft#https://github.com/notifications/beacon/APC7TP42WVLCRC6EHQF4ZXLXSF4GDA5CNFSM6AAAAAA2GOSP7CWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTCPZBMW.gif>Message ID: ***@***.***

…

-- Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: ***@***.***

[FariborzDaneshvar-NOAA]

Turning ON and OFF the --past-forecast flag!

Directory of a run without the flag: /lustre/hurricanes/florence_2018_Fariborz_v4_15_ensembles

@SorooshMani-NOAA, similar to the previous runs, no sign of storm track is visible in the scatter plot of maxele.63.nc:

However,
Unlike the previous runs (with the --past-forecast flag), more than one eigenvalue was found (i.e. for linear_k1_p1_n0.025):

Similar plot exist for linear_k1_p1_n0.025\, but the linear_k1_p1_n0.1\ directory is empty!

[SorooshMani-NOAA]

@FariborzDaneshvar-NOAA so in the zoomed in landfall region plot it's still missing the floodplain? This is strange. Maybe as Takis said the issue is with how I compiled SCHISM then. Let's talk about it and see what debugging approach to take.

[FariborzDaneshvar-NOAA]

@SorooshMani-NOAA , yes, here is an example:
path: /lustre/hurricanes/florence_2018_Fariborz_v4_15_ensembles/setup/ensemble.dir/analyze/linear_k1_p1_n0.025/validation_vortex_4_variable_korobov_3.png

[SorooshMani-NOAA]

@FariborzDaneshvar-NOAA using Thalassa on PW, I see a difference if I limit the max elevation colorbar so that I can see more contrast:

OFCL track (faked as BEST)	BEST track

So it seems that the issue might actually be the missing data on the OFCL track that we fake as BEST. I'd like to note here that we don't set the PaHM model in SCHISMS param.nml here and it's using the default model (i.e. GAHM)

I also see the hurricane signal in the BEST track run (original):

The fact that we have multiple eigenvalues is also a good sign, however I don't understand why we don't get the correct exceedance plots here

[FariborzDaneshvar-NOAA]

These issues will be investigated in separate tickets as follow:

• Different inundation maps for the surrogate and model of hurricane Florence
• No sign of storm track in runs with OFCL (faked as BEST) track (Resolved. See ticket updates)
• Setup and run PaHM for BEST and OFCL tracks of florence (Resolved. See ticket updates)

[FariborzDaneshvar-NOAA]

I was able to run the ondemand-storm-workflow and partially replicate Willam's paper results. However, there were some issues in the plotting routines (listed below) that will be investigated in the future.

• plot_node_map() function does not plot sensitivity plots correctly #131
• mismatch between model and validation plots generated by plot_selected_validations() #132