Updated data interpolation for Antarctica mesh generation case #750

matthewhoffman · 2023-12-22T04:19:49Z

This PR adds automates interpolation of observational data from gridded datasets to Antarctic mesh within COMPASS. This takes care of the peculiarities of the current gridded compilation dataset (antarctica_8km_2020_10_20.nc), as well as using conservative remapping directly from the high-resolution BedMachineAntarctica and MeASUReS velocity datasets. There is a fairly heavy degree of pre-processing done to get the BedMachine and MeASUReS datasets ready to be used here. The pre-processing includes renaming variables, setting reasonable _FillValue and missing_value attributes, extrapolating fields to avoid interpolation ramps at ice margins, updating mask values, and raising the bed topography at Lake Vostok to ensure a flat ice surface. Those data files and processing scripts currently live here on Chicoma: /usr/projects/climate/trhille/data. Eventually that pre-processing could be integrated into a new step in COMPASS, or the processed data files could be added to the server on Anvil and downloaded as needed. However, until then, this test case provides a reproducible workflow for setting up Antarctic meshes at varying resolutions

Checklist

User's Guide has been updated
Developer's Guide has been updated
API documentation in the Developer's Guide (api.rst) has any new or modified class, method and/or functions listed
Documentation has been built locally and changes look as expected
Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

matthewhoffman · 2023-12-22T04:21:18Z

This PR is a replacement to #408 which is out of date to refactoring of the landice mesh generation workflow.

matthewhoffman · 2023-12-22T04:28:49Z

Testing

Running the default options generates a 4-20 km resolution whole-AIS mesh with 384370 cells and a 50 km gutter. On one node of Chicoma this took 41 minutes. Viewing the mesh in Paraview along with our existing production 4-20 km AIS mesh, the two meshes do not have identical cell locations but clearly have the same cell spacing.

By modifying the min and max spacing, I was able to create a consistent 2-10 km resolution mesh with 1263402 cells in a little under 2 hours. A 1-8 km mesh with 4743133 cells took 10:30 hours.

Note that a few steps are redundant, and many of the steps could be separated into a pre-processing test case; the unique operations could probably be done in about half the total time.

matthewhoffman · 2023-12-22T04:32:00Z

Note that this PR requires MPAS-Dev/MPAS-Tools#544.

The steps to update MPAS-Tools in compass to test this PR are:

Create your compass environment as normal and source the load script
Go to a separate MPAS-Tools repository and checkout Fix bugs in ESMF interpolation method MPAS-Tools#544
From the MPAS-Tools directory: python -m pip install -e conda_package
Proceed setting up test case as normal

xylar · 2023-12-22T04:39:58Z

Because of the MPAS-Tools changes, we will need to:

merge Fix bugs in ESMF interpolation method MPAS-Tools#544
make a release of MPAS-Tools
make a conda-forge package of mpas_tools
update the compass alpha version and the mpas_tools version here

That sounds like a lot but it's pretty routine. I'll do steps 2 and 3 once 1 happens.

matthewhoffman · 2024-02-05T17:48:50Z

@trhille , this was originally your PR, so it seems weird to have you review it, but look it over and test it if you'd like. I somehow lost your authorship on the main commits (maybe I ended up manually re-creating them for some reason?), but I learned how to give you "co-author" credit in the commit message so Github counts you as an author, FWIW. I rebased on main to force the stuck CI checks to run and resolved conflicts with the recent Anvil file renaming we merged last week, so it should be an easy merge.

trhille

@matthewhoffman, thanks for tackling this! I think it's looking really good. I had a few stylistic changes, and updated some comments. There's one change to the name of the gridded dataset that has to be implemented for the code to run.

This workflow is very expensive and takes a long time to run. I'm wondering if we should either put the interpolation in a separate step, or if we should include a config option to determine whether to do the full interpolation workflow. I can imagine that we'd want to turn interpolation off when with different cell spacing functions, for instance.

docs/developers_guide/landice/api.rst

trhille · 2024-02-22T17:29:14Z

docs/users_guide/landice/test_groups/antarctica.rst

+a flat ice surface there.
+
+Those data files and processing scripts currently live here on Chicoma:
+``/usr/projects/climate/trhille/data``.


Should we move this to the Fanssie project space on Perlmutter?

Thanks for copying this dir over to Perlmutter. I've updated the cfg and docs accordingly

compass/landice/tests/antarctica/mesh.py

compass/landice/mesh.py

trhille

A few more comments after finding that my 1 hr job on Chicoma only made it to the step of creating the scrip file for BedMachine.

trhille · 2024-02-22T19:25:17Z

compass/landice/mesh.py

+
+    logger = self.logger
+
+    logger.info('creating scrip file for BedMachine dataset')


This step is really slow, so we should check for an existing scrip file before proceeding. It should also write to the compass working directory instead of to the data directory if it does create a new scrip file. I'd rather have the data directory untouched.

Can you check if your compass env contains the changes to MPAS-Tools here? MPAS-Dev/MPAS-Tools#544

This step got like 100x faster after that change. I'm wondering if you'd be comfortable skipping the check for an existing file if this step only takes a minute.

Actually I think I know the answer - the conda env version for this branch would not have the MPAS-Tools update in it, because those were made at the same time as this PR. But I believe that MPAS-Tools PR is part of the current compass env. So try redoing your test after merging this branch locally into main (or rebasing) and creating an updated compass env.

I've modified to write the scrip file to the workdir.

trhille · 2024-02-22T19:27:49Z

compass/landice/mesh.py

+
+    logger = self.logger
+
+    logger.info('creating scrip file for velocity dataset')


As with the BedMachine scrip file, let's check for an existing file in the data directory before writing a huge new one. And it should write any new file to the work directory, leaving the data directory untouched. If the scrip file exists in the data directory, let's just put a symlink to it in the work directory.

I've modified to write the scrip file to the workdir.

trhille

One more change!

compass/landice/mesh.py

This commit Interpolates data from gridded datasets to an Antarctic mesh. This takes care of the peculiarities of the current gridded compilation dataset (antarctica_8km_2020_10_20.nc), as well as using conservative remapping directly from the high-resolution BedMachineAntarctica and MeASUReS velocity datasets. There is a fairly heavy degree of pre-processing done to get the BedMachine and MeASUReS datasets ready to be used here. The pre-processing includes renaming variables, setting reasonable _FillValue and missing_value attributes, extrapolating fields to avoid interpolation ramps at ice margins, updating mask values, and raising the bed topography at Lake Vostok to ensure a flat ice surface. Those data files and processing scripts currently live here on Chicoma: /usr/projects/climate/trhille/data Eventually that pre-processing could be integrated into a new step in COMPASS, or the processed data files could be added to the server on Anvil and downloaded as needed. The conservative remapping step using ESMF_RegridWeightGen requires multiple nodes in order to not run out of memory. This should probably be updated to be a separate step in the future in order to save on cost. Co-authored-by: Trevor Hillebrand <[email protected]>

This is the configuration used to create the 4km AIS mesh used in ISMIP6-2300. Co-authored-by: Trevor Hillebrand <[email protected]>

And set default cull_distance to not be too large Co-authored-by: Trevor Hillebrand <[email protected]>

corrected Co-authored-by: Trevor Hillebrand <[email protected]>

The reference gridded dataset has a funky thickness field that includes non-ice-sheet thin ice along the Antarctic Peninsula and is inconsistent with the thickness field we ultimately interpolate from using the BedMachine dataset. This leads to unusual gutter sizes in parts of the mesh and may be contributing to some strange culling artifacts. This commit adds a preprocessing function that replaces the reference gridded dataset's thickness field with one bilinearly interpolated from BedMachine. This is only used for the flood fill and mesh culling steps. Co-authored-by: Trevor Hillebrand <[email protected]>

adjusted from previous commit Co-authored-by: Trevor Hillebrand <[email protected]>

Minor corrections from Trevor Co-authored-by: Trevor Hillebrand <[email protected]>

Co-authored-by: Trevor Hillebrand <[email protected]>

trhille

I was able to run it on Perlmutter (total runtime 29:55) after transferring the data over from Chicoma.

Most fields look good:

However, observedSurfaceVelocityUncertainty is 1.0 everywhere in Antarctica.nc. I checked the Antarctica_backup.nc file, and it's 0.0 everywhere, so the issue is in the gridded dataset or the interpolation rather than the cleanup step. I checked the gridded dataset, and vErr is present, so it must be an issue with the interpolation. And indeed, the log file and the netCDF history both show that the interpolation is never performed from the MEaSURES dataset, although no error is thrown. Ah, it turns out that there is a missing check_call at the end of interp_ais_measures.

trhille · 2024-02-29T03:28:11Z

compass/landice/mesh.py

+    """
+
+    # Create a backup in case clean-up goes awry
+    backup_name = f"{fname.split('.')[:-1][0]}_backup,nc"


Suggested change

backup_name = f"{fname.split('.')[:-1][0]}_backup,nc"

backup_name = f"{fname.split('.')[:-1][0]}_backup.nc"

trhille · 2024-02-29T03:48:15Z

compass/landice/mesh.py

+            '-v', 'observedSurfaceVelocityX',
+                  'observedSurfaceVelocityY',
+                  'observedSurfaceVelocityUncertainty']
+


This interpolation is never performed! Needs another check_call(args, logger=logger)

Add missing check_call for velocity interpolation from MEaSURES dataset. Also fix small typo.

trhille · 2024-02-29T15:48:52Z

After commit 8a8bb4a, velocities and velocity uncertainty are being interpolated and the results look correct. These are from a 2–10km mesh, which ran on Perlmutter in 97:33:

trhille

@matthewhoffman, this looks ready to go to me!

matthewhoffman mentioned this pull request Dec 22, 2023

Add data interpolation to Antarctica mesh generation case #408

Closed

matthewhoffman added land ice in progress This PR is not ready for review or merging labels Dec 22, 2023

matthewhoffman assigned trhille Dec 22, 2023

matthewhoffman mentioned this pull request Jan 4, 2024

Fix bugs in ESMF interpolation method MPAS-Dev/MPAS-Tools#544

Merged

matthewhoffman force-pushed the landice/ais_data_interp branch from 0df6e35 to 8ff75a8 Compare February 5, 2024 17:45

matthewhoffman requested a review from trhille February 5, 2024 17:45

trhille requested changes Feb 22, 2024

View reviewed changes

matthewhoffman removed the in progress This PR is not ready for review or merging label Feb 28, 2024

trhille requested changes Feb 28, 2024

View reviewed changes

compass/landice/mesh.py Outdated Show resolved Hide resolved

matthewhoffman and others added 11 commits February 28, 2024 13:35

Update cfg options to create standard AIS 4km mesh

a2f4a91

This is the configuration used to create the 4km AIS mesh used in ISMIP6-2300. Co-authored-by: Trevor Hillebrand <[email protected]>

Add check that cull_distance is not too large

fdd2f50

And set default cull_distance to not be too large Co-authored-by: Trevor Hillebrand <[email protected]>

fixup esmf_regridweightgen commands

97295a6

corrected Co-authored-by: Trevor Hillebrand <[email protected]>

Adjust orientation of bedmachine to match that of gridded dataset

12b90e8

adjusted from previous commit Co-authored-by: Trevor Hillebrand <[email protected]>

Adjust default mesh spacing to match our existing 4km AIS mesh

082e2be

add docs

3f67daa

Apply suggestions from code review

6b3b0fe

Minor corrections from Trevor Co-authored-by: Trevor Hillebrand <[email protected]>

Update 8km AIS gridded dataset version

8270f19

Co-authored-by: Trevor Hillebrand <[email protected]>

Correct basal heat flux sign convention

b8649fa

Co-authored-by: Trevor Hillebrand <[email protected]>

matthewhoffman force-pushed the landice/ais_data_interp branch from 4e88dc9 to b8649fa Compare February 28, 2024 21:37

matthewhoffman added 2 commits February 28, 2024 14:34

Save scrip files in workdir instead of writing to source data dir

008e4d1

Update data path to Perlmutter

489e0b1

trhille requested changes Feb 29, 2024

View reviewed changes

Add missing check_call for velocity interpolation

8a8bb4a

Add missing check_call for velocity interpolation from MEaSURES dataset. Also fix small typo.

trhille approved these changes Feb 29, 2024

View reviewed changes

trhille merged commit 6c2ac36 into MPAS-Dev:main Feb 29, 2024
4 checks passed

andrewdnolan mentioned this pull request Mar 7, 2024

Add edge masks to default arguments of make_region_masks landice framework function #786

Merged

1 task

andrewdnolan mentioned this pull request Apr 16, 2024

Landice interpolation unification #817

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated data interpolation for Antarctica mesh generation case #750

Updated data interpolation for Antarctica mesh generation case #750

matthewhoffman commented Dec 22, 2023 •

edited

Loading

matthewhoffman commented Dec 22, 2023

matthewhoffman commented Dec 22, 2023 •

edited

Loading

matthewhoffman commented Dec 22, 2023

xylar commented Dec 22, 2023

matthewhoffman commented Feb 5, 2024

trhille left a comment

trhille Feb 22, 2024

matthewhoffman Feb 28, 2024

trhille left a comment

trhille Feb 22, 2024

matthewhoffman Feb 22, 2024

matthewhoffman Feb 22, 2024

matthewhoffman Feb 28, 2024

trhille Feb 22, 2024

matthewhoffman Feb 28, 2024

trhille left a comment

trhille left a comment

trhille Feb 29, 2024

trhille Feb 29, 2024

trhille commented Feb 29, 2024

trhille left a comment


		logger = self.logger

		logger.info('creating scrip file for BedMachine dataset')


		logger = self.logger

		logger.info('creating scrip file for velocity dataset')

	backup_name = f"{fname.split('.')[:-1][0]}_backup,nc"
	backup_name = f"{fname.split('.')[:-1][0]}_backup.nc"

Updated data interpolation for Antarctica mesh generation case #750

Updated data interpolation for Antarctica mesh generation case #750

Conversation

matthewhoffman commented Dec 22, 2023 • edited Loading

matthewhoffman commented Dec 22, 2023

matthewhoffman commented Dec 22, 2023 • edited Loading

Testing

matthewhoffman commented Dec 22, 2023

xylar commented Dec 22, 2023

matthewhoffman commented Feb 5, 2024

trhille left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trhille left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trhille left a comment

Choose a reason for hiding this comment

trhille left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trhille commented Feb 29, 2024

trhille left a comment

Choose a reason for hiding this comment

matthewhoffman commented Dec 22, 2023 •

edited

Loading

matthewhoffman commented Dec 22, 2023 •

edited

Loading