Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow reusing weights saved in a pyramid during xESFM regridding #34

Merged
merged 14 commits into from
Apr 11, 2022

Conversation

andersy005
Copy link
Member

This is my attempt at improving pyramid_regrid's performance and circumventing the ESMF errors when using multithreaded dask workers.

This PR introduces two new functionalities

  • a helper function for generating a weights pyramid. with this function, we should be able to serially generate weights without worrying about ESMF parallelism issues.
  • pyramid_regrid() accepts a new argument (weights_pyramid). with the pre-generated weights, i imagine that it's possible to turn existing loop within pyramid_regrid() into an embarrassingly parallel computation in the future.
In [1]: import xarray as xr, dask

In [2]: from ndpyramid.regrid import generate_weights_pyramid, pyramid_regrid

In [3]: ds_in = xr.tutorial.open_dataset('rasm', chunks={'time': 12})
  • weights generation
In [6]: levels = 4

In [7]: %%time
   ...: with dask.config.set({'scheduler': 'sync'}):
   ...:     weights_pyramid = generate_weights_pyramid(ds_in.isel(time=0), levels)
   ...: 

CPU times: user 3min, sys: 12 ms, total: 3min
Wall time: 3min
  • generating pyramids with pre-generated weights
In [8]: %%time
   ...: pyramida = pyramid_regrid(ds_in, levels=levels, weights_pyramid=weights_pyramid)
   ...: 
   ...: 
CPU times: user 32.6 s, sys: 32.1 ms, total: 32.6 s
Wall time: 32.6 s
  • generating pyramids without pre-generated weights
In [9]: %%time
   ...: pyramidb = pyramid_regrid(ds_in, levels=levels, weights_pyramid=None)
   ...: 
   ...: 
CPU times: user 2min 59s, sys: 0 ns, total: 2min 59s
Wall time: 2min 59s

Cc @jhamman, @norlandrhagen

@andersy005 andersy005 changed the title Allow using weights saved in a pyramid during xESFM regridding Allow reusing weights saved in a pyramid during xESFM regridding Apr 10, 2022
@andersy005 andersy005 marked this pull request as ready for review April 11, 2022 23:58
@andersy005 andersy005 merged commit 935878f into main Apr 11, 2022
@andersy005 andersy005 deleted the weights-pyramid branch April 11, 2022 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant