Aquaplanet external gravity wave #187

jeremy-lilly · 2024-02-08T21:28:48Z

This PR creates a new ocean task for testing the temporal convergence rate of time-stepping schemes in MPAS-O. The case is that of a simple external gravity wave on an aquaplant. The normal velocity is initialized to zero and the layer thickness is initialized as a Gaussian bump.

The user selects a time-stepping scheme and a series of time-steps to run the test. The first listed time-step will be used to produce a reference solution that other runs will be compared to.

The PR also implements a version of the task for LTS schemes, in particular for the new FB-LTS scheme from this PR: E3SM-Project/E3SM#6224. Convergence plots produced by this task for FB-LTS are given below.

Note that the external gravity wave case was chosen so as to have as few terms in the tendencies as possible -- because of this, both the LTS and FB-LTS schemes show their "true" order of convergence, rather than the first order convergence caused by the operator splitting that both the LTS and FB-LTS codes implement.

Checklist

User's Guide has been updated
Developer's Guide has been updated
API documentation in the Developer's Guide (api.md) has any new or modified class, method and/or functions listed
Documentation has been built locally and changes look as expected
Testing comment in the PR documents testing used to verify the changes
New tests have been added to a test suite

… point the forward step to the correct graph.info file

and a few other small edits

mark-petersen · 2024-03-11T20:17:20Z

@jeremy-lilly thanks for your work on this, it looks beautiful! Could you add a documentation page to the user's guide, similar to this?
https://github.com/E3SM-Project/polaris/blob/main/docs/users_guide/ocean/tasks/inertial_gravity_wave.md
Also, it is best if the history is just simple commits on some recent branchpoint from main. It looks like you pulled from main here. Could you fix that?

mark-petersen · 2024-03-11T20:38:33Z

Trying it out:

./configure_polaris_envs.py  --conda /usr/projects/climate/mpeterse/miconda3
source load_dev_polaris_0.3.0-alpha.1_chicoma-cpu_gnu_mpich.sh
polaris list
...
  45: ocean/spherical/icos/ext_grav_wav_local_time_step
  46: ocean/spherical/icos/ext_grav_wav_global_time_step
  47: ocean/spherical/qu/ext_grav_wav_local_time_step
  48: ocean/spherical/qu/ext_grav_wav_global_time_step

polaris setup -p ~/repos/E3SM/lts-fb-jer-pr/components/mpas-ocean -w $r/230311_polaris_ext_grav_wave -n 45 46 47 48

salloc -N 1 -t 2:0:0 --qos=debug --reservation=debug --account=t24_coastal_ocean
source load_dev_polaris_0.3.0-alpha.1_chicoma-cpu_gnu_mpich.sh
cd $r/230311_polaris_ext_grav_wave
polaris serial

This worked similarly on chicoma, perlmutter, and chrysalis. I am using a compiled executable from E3SM-Project/E3SM#6224.

Output is as follows:

(dev_polaris_0.3.0-alpha.1) pm:230311_ltsfb_test_polaris$ polaris serial
ocean/spherical/icos/ext_grav_wav_local_time_step
  * step: icos_base_mesh_120km
          execution:        SUCCESS
          runtime:          0:00:40
  * step: icos_init_120km_local_time_step
          execution:        SUCCESS
          runtime:          0:00:01
  * step: icos_init_lts_120km_local_time_step
          execution:        SUCCESS
          runtime:          0:00:15
  * step: icos_forward_120km_local_time_step_10s
          execution:        SUCCESS
          runtime:          0:01:17
  * step: icos_forward_120km_local_time_step_120s
          execution:        SUCCESS
          runtime:          0:00:13
  * step: icos_forward_120km_local_time_step_240s
          execution:        SUCCESS
          runtime:          0:00:25
  * step: icos_forward_120km_local_time_step_480s
          execution:        SUCCESS
          runtime:          0:00:44
  * step: analysis
          execution:        SUCCESS
          runtime:          0:00:07
  task execution:   SUCCESS
  task runtime:     0:03:42
ocean/spherical/icos/ext_grav_wav_global_time_step
  * step: icos_base_mesh_120km
          already completed
  * step: icos_init_120kmglobal_time_step
          execution:        SUCCESS
          runtime:          0:00:00
  * step: icos_forward_120km_global_time_step_10s
          execution:        SUCCESS
          runtime:          0:01:17
  * step: icos_forward_120km_global_time_step_120s
          execution:        SUCCESS
          runtime:          0:00:17
  * step: icos_forward_120km_global_time_step_240s
          execution:        SUCCESS
          runtime:          0:00:12
  * step: icos_forward_120km_global_time_step_480s
          execution:        SUCCESS
          runtime:          0:00:15
  * step: analysis
          execution:        SUCCESS
          runtime:          0:00:07
  task execution:   SUCCESS
  task runtime:     0:02:08
ocean/spherical/qu/ext_grav_wav_local_time_step
  * step: qu_base_mesh_120km
          execution:        SUCCESS
          runtime:          0:00:17
  * step: qu_init_120km_local_time_step
          execution:        SUCCESS
          runtime:          0:00:00
  * step: qu_init_lts_120km_local_time_step
          execution:        SUCCESS
          runtime:          0:00:15
  * step: qu_forward_120km_local_time_step_10s
          execution:        SUCCESS
          runtime:          0:01:18
  * step: qu_forward_120km_local_time_step_120s
          execution:        SUCCESS
          runtime:          0:00:12
  * step: qu_forward_120km_local_time_step_240s
          execution:        SUCCESS
          runtime:          0:00:17
  * step: qu_forward_120km_local_time_step_480s
          execution:        SUCCESS
          runtime:          0:00:08
  * step: analysis
          execution:        SUCCESS
          runtime:          0:00:06
  task execution:   SUCCESS
  task runtime:     0:02:34
ocean/spherical/qu/ext_grav_wav_global_time_step
  * step: qu_base_mesh_120km
          already completed
  * step: qu_init_120kmglobal_time_step
          execution:        SUCCESS
          runtime:          0:00:00
  * step: qu_forward_120km_global_time_step_10s
          execution:        SUCCESS
          runtime:          0:01:14
  * step: qu_forward_120km_global_time_step_120s
          execution:        SUCCESS
          runtime:          0:00:11
  * step: qu_forward_120km_global_time_step_240s
          execution:        SUCCESS
          runtime:          0:00:08
  * step: qu_forward_120km_global_time_step_480s
          execution:        SUCCESS
          runtime:          0:00:07
  * step: analysis
          execution:        SUCCESS
          runtime:          0:00:04
  task execution:   SUCCESS
  task runtime:     0:01:45
Task Runtimes:
0:03:42 PASS ocean/spherical/icos/ext_grav_wav_local_time_step
0:02:08 PASS ocean/spherical/icos/ext_grav_wav_global_time_step
0:02:34 PASS ocean/spherical/qu/ext_grav_wav_local_time_step
0:01:45 PASS ocean/spherical/qu/ext_grav_wav_global_time_step
Total runtime: 0:10:17
PASS: All passed successfully!

with plots identical to those above. Beautiful!

xylar · 2024-03-11T20:46:33Z

docs/users_guide/quick_start.md

+setting up the task (see {ref}`conda-env`). To run a task that has already
+been run, it is necessary to first delete `polaris_step_complete.log` file.


Perfect, thanks for adding this!

xylar

@jeremy-lilly, this looks great! The main thing I noticed is that documentation is missing. Please ping me to review again once that gets added.

In the meantime, a few other small comments.

xylar · 2024-03-11T20:48:03Z

polaris/ocean/convergence/convergence.cfg

@@ -4,7 +4,7 @@
 convergence_eval_time = 24.0

 # Convergence threshold below which a test fails
-convergence_thresh = 1.0
+convergence_thresh = 0.1


Hopefully, no tests are using this default but I would be curious why it was changed here.

xylar · 2024-03-11T20:52:25Z

polaris/ocean/convergence/forward.py

+            dt_btr_scaling = section.getfloat('dt_btr_scaling')
+            dt_btr = self.dt / dt_btr_scaling
+            btr_dt_str = get_time_interval_string(seconds=dt_btr *
+                                                  self.resolution)


I don't think we can make this change without also updating any test cases that already use the split time stepping to use this approach. Maybe none are so far in which case maybe this will be fine.

cbegeman

Fantastic work here! I made a few comments and suggestions. Don't feel like you have to do all the things I suggested if you don't have the time. Also, I made most of the comments before Xylar posted his, so pardon any conflicting messaging there!

I ran the non-LTS case successfully with E3SM master.

Many of the suggestions have to do with making the convergence framework changes a bit more intuitive since we originally wrote the code for space and time convergence together. It will be great to have the option to evaluate time convergence separately with your changes.

cbegeman · 2024-03-11T20:22:42Z

polaris/ocean/convergence/analysis.py

+        else:
+            for resolution in self.resolutions:
+                for dt in dts:
+                    mesh_name = resolution_to_subdir(resolution)


Suggested change

mesh_name = resolution_to_subdir(resolution)

Seems this isn't used

cbegeman · 2024-03-11T20:28:56Z

polaris/ocean/convergence/analysis.py

+                    mesh_name = resolution_to_subdir(resolution)
+                    forward = dependencies['forward'][dt]
+                    self.add_input_file(
+                        filename=f'{int(dt)}_output.nc',


Suggested change

filename=f'{int(dt)}_output.nc',

filename=f'{int(dt)}s_output.nc',

Better yet, in case there is a case with fractional seconds.

Suggested change

filename=f'{int(dt)}_output.nc',

filename=f'dt{get_time_interval_string(seconds=dt}s_output.nc',

cbegeman · 2024-03-11T20:35:07Z

polaris/ocean/convergence/analysis.py

+                error.append(error_res)
+        else:
+            for i in range(1, len(dts)):
+                mesh_name = resolution_to_subdir(resolutions[0])


Is this implying that when dts is specified, we are evaluating convergence in time only (and all steps must use the mesh at resolutions[0])? If so, I think we should put in some safeguards so that this function cannot be used incorrectly.

cbegeman · 2024-03-11T20:42:13Z

polaris/ocean/convergence/analysis.py

@@ -305,7 +382,7 @@ def compute_error(self, mesh_name, variable_name, zidx=None,

        return error

-    def exact_solution(self, mesh_name, field_name, time, zidx=None):
+    def exact_solution(self, mesh_name, field_name, time, zidx=None, dt=None):


It seems like it is somewhat of a niche case that the exact solution is just the solution at a fine time step. I would recommend overriding exact_solution just for your test case. Let me know if you need help finding an example in the code.

cbegeman · 2024-03-11T20:43:59Z

polaris/ocean/convergence/forward.py

@@ -73,13 +86,17 @@ def __init__(self, component, name, subdir, resolution, mesh, init,
        self.add_input_file(
            filename='init.nc',
            work_dir_target=f'{init.path}/initial_state.nc')
+        if graph_path is None:
+            graph_path = mesh.path
        self.add_input_file(
            filename='graph.info',


Suggested change

filename='graph.info',

I just noticed that this is no longer used

cbegeman · 2024-03-11T21:04:24Z

polaris/ocean/tasks/external_gravity_wave/forward_global_time_step.yaml

+    config_dt: {{ dt }}
+    config_time_integrator: {{ time_integrator }}
+  debug:
+    config_disable_thick_all_tend: false


We generally don't include the default options here unless we think it will lead to confusion or having a namelist option set a certain way is essential to the test. You should be able to just specify which terms are turned off (*disable* = true)

cbegeman · 2024-03-11T21:07:02Z

polaris/ocean/tasks/external_gravity_wave/forward_local_time_step.yaml

@@ -0,0 +1,81 @@
+omega:


It can be nice to have one file forward.yaml that has the config options common to both global and local time stepping and then either/both of forward_global_time_step.yaml and forward_local_time_step.yaml that contain the options that are specific to each.

cbegeman · 2024-03-11T21:09:02Z

polaris/ocean/tasks/external_gravity_wave/init.py

+
+        init_vertical_coord(config, ds)
+
+        temperature_array = temperature * xr.ones_like(ds_mesh.latCell)


Suggested change

temperature_array = temperature * xr.ones_like(ds_mesh.latCell)

temperature_array = temperature * xr.ones_like(latCell)

cbegeman · 2024-03-12T14:56:58Z

polaris/ocean/tasks/external_gravity_wave/lts_regions.py

+    center_pt = Point(lat_center, lon_center)
+    for icell in range(0, n_cells):
+        cell_pt = Point(lat_cell[icell], lon_cell[icell])
+        if distance(cell_pt, center_pt) < np.pi / 4:


Consider making the radius a function parameter

cbegeman · 2024-03-12T14:59:55Z

polaris/ocean/tasks/external_gravity_wave/lts_regions.py

+                   use_progress_bar=use_progress_bar)
+
+
+def label_mesh(mesh, graph_info, num_interface,  # noqa: C901


Consider making the function to label interface cells different from the one that sets the main regions (1,2). Since we already have an LTS test in compass that may eventually make its way to polaris, it would be nice to create some code here that could be moved to a shared directory down the line and be reused by other LTS tests.

jeremy-lilly · 2024-03-13T16:51:51Z

@xylar, @cbegeman, @mark-petersen -- thank you all for your comments and suggestions! Just wanted to let you know that I saw these and that they are on my to-do list, but I might not get to them right away. I will ping everyone when ready for a second look : ) Thanks!

jeremy-lilly and others added 20 commits January 11, 2024 16:19

First commit for external gravity wave case with fblts, still need to…

4c54bb0

… point the forward step to the correct graph.info file

Fix lts regions step, and complete region labeling routine

cadab38

graph_info is read from right folder now

cee0158

and a few other small edits

Added missing file

2637cb4

Changed number of time levels in fblts namelist

b64c87c

Fixed input read of lts regions

2367cad

Fix path for graph.info if using FB_LTS

c8fab6a

Added extra forward runs for all the necessary dts

6425017

Trying to get the analysis step to work

9438371

Fixed issue thanks to Xylar

acc02c6

Intermediate commit

52b9e7e

Stuff works

d282587

Modified forward.yaml for RK4 and dt for conv test

4714bb9

Made a lot of changes

5a95f96

Added files

6dfa0bd

Fixed issues

f4e7e8d

Small change to plot

e3a061a

Modified convergence threshold

4396252

Fix step names for runs with different dts

f999cb2

Set nVertLevels during paraview extraction of lts mesh

03c72f0

jeremy-lilly mentioned this pull request Feb 8, 2024

Implement FB-LTS Time-Stepping Scheme E3SM-Project/E3SM#6224

Merged

mark-petersen requested review from cbegeman, xylar and mark-petersen March 11, 2024 19:47

mark-petersen added ocean Related to ocean tests or analysis E3SM PR required The polaris changes won't work with the current E3SM-Project submodule and require an update labels Mar 11, 2024

xylar reviewed Mar 11, 2024

View reviewed changes

cbegeman requested changes Mar 12, 2024

View reviewed changes

jeremy-lilly force-pushed the aquaplanet-external-gravity-wave branch from 7ce4884 to 03c72f0 Compare March 15, 2024 23:11

Add documentation for ext gravity wave case

6dd12aa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Aquaplanet external gravity wave #187

Aquaplanet external gravity wave #187

jeremy-lilly commented Feb 8, 2024

mark-petersen commented Mar 11, 2024 •

edited

Loading

mark-petersen commented Mar 11, 2024

xylar Mar 11, 2024 •

edited

Loading

xylar left a comment

xylar Mar 11, 2024

xylar Mar 11, 2024

cbegeman left a comment

cbegeman Mar 11, 2024

cbegeman Mar 11, 2024

cbegeman Mar 11, 2024

cbegeman Mar 11, 2024

cbegeman Mar 11, 2024

cbegeman Mar 11, 2024

cbegeman Mar 11, 2024

cbegeman Mar 11, 2024

cbegeman Mar 11, 2024

cbegeman Mar 11, 2024

cbegeman Mar 12, 2024

cbegeman Mar 12, 2024

jeremy-lilly commented Mar 13, 2024

		setting up the task (see {ref}`conda-env`). To run a task that has already
		been run, it is necessary to first delete `polaris_step_complete.log` file.

	filename=f'{int(dt)}_output.nc',
	filename=f'{int(dt)}s_output.nc',

	filename=f'{int(dt)}_output.nc',
	filename=f'dt{get_time_interval_string(seconds=dt}s_output.nc',


		init_vertical_coord(config, ds)

		temperature_array = temperature * xr.ones_like(ds_mesh.latCell)

		use_progress_bar=use_progress_bar)


		def label_mesh(mesh, graph_info, num_interface, # noqa: C901

Aquaplanet external gravity wave #187

Are you sure you want to change the base?

Aquaplanet external gravity wave #187

Conversation

jeremy-lilly commented Feb 8, 2024

mark-petersen commented Mar 11, 2024 • edited Loading

mark-petersen commented Mar 11, 2024

xylar Mar 11, 2024 • edited Loading

Choose a reason for hiding this comment

xylar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbegeman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremy-lilly commented Mar 13, 2024

mark-petersen commented Mar 11, 2024 •

edited

Loading

xylar Mar 11, 2024 •

edited

Loading