Lazy regridding with Linear, Nearest, and AreaWeighted #3701

bouweandela · 2020-04-28T07:41:15Z

This pull request implements lazy regridding for the Linear, Nearest, and AreaWeighted regridders.

stephenworsley · 2020-05-11T17:01:46Z

This certainly looks promising. It seems like something Iris ought to have and we deffinitely appreciate the effort to getting this in. We'll see what we can do about coming to a decision on this and getting this reviewed.

JaroCamphuijsen · 2020-07-29T10:49:08Z

@stephenworsley This would be very useful, is there a decision from the iris team already?

trexfeathers · 2020-08-21T16:31:14Z

We hope to give this the attention it needs for Iris v3.1

bjlittle · 2020-08-23T21:01:15Z

@SciTools/iris-devs Why can't this be included for iris 3.0?

It's really going to massively help the ESMValTool community, and it's an all round win for the general community.

Sadly this PR has been sitting here accruing zero value for almost 4 months ☹️

trexfeathers · 2020-08-24T08:08:08Z

SciTools/iris-devs Why can't this be included for iris 3.0?

I appreciate the frustration but we have already planned in as much as we can for delivering Iris 3.0 on time. We discussed this and several other items on 20/08, planned then allocated to 3.0 and 3.1 accordingly.

bjlittle · 2020-08-24T15:43:14Z

@bouweandela You'll need to take this PR out of draft if you're keen to get this merged

bjlittle · 2020-08-24T15:44:39Z

@trexfeathers Begs the question, when is iris 3.1 planned?

(Apologies, I'm just being divisive. I don't expect you to answer that question, but we do need a more transparent release roadmap for the community... that's my main point here)

bouweandela · 2020-08-26T15:40:50Z

The pull request is now ready for review @bjlittle. If there is anything I can help out with to get this across the finish line, please let me know!

pp-mo · 2020-09-02T18:51:16Z

lazy regridding for the Linear, Nearest, and AreaWeighted regridders

Really neat work !
We should have a good whatsnew entry to publicise it, though.

bouweandela · 2020-09-03T08:35:15Z

Thanks @pp-mo! I just added a whatsnew entry. Would this be good enough, or should it explain more?

docs/iris/src/whatsnew/contributions_3.0.0/newfeature_2020-Sep-03_lazy_regridding.txt

pp-mo · 2020-09-03T11:26:40Z

I just added a whatsnew entry. Would this be good enough

I think that's pretty much enough actually.
But looking at how it reads, I would link explicitly to the reference docs for the classes mentioned (see my suggested).
And, I think we should add a note about the lazy support in the docstrings of those classes : What you have said here about the AreaWeightedRegridder and here about the RectilinearRegridder is all fine+great, but only of use to devs as these are low-level components + don't appear in the reference docs. So, something similar where people can see it would be great !

Co-authored-by: Patrick Peglar <[email protected]>

bouweandela · 2020-09-03T14:44:17Z

lib/iris/cube.py

+                * :class:`iris.analysis.UnstructuredNearest`,
+                * :class:`iris.analysis.PointInCell`,
+                * :class:`iris.experimental.regrid.ProjectedUnstructuredLinear`,
+                * :class:`iris.experimental.regrid.ProjectedUnstructuredNearest`.


I found several more regridding schemes and added them here. Does that look right, or were they missing for a reason?

I think this is out of date.
But also, we don't want to commit to the experimental ones.
So maybe it should omit those, and amend to say "The regridding schemes in Iris currently include:" ...

Also.. if we extend this then the next section that you added "These regridding schemes support lazy regridding..." needs to change, as not all these schemes support it.

I removed the experimental regridders and changed the section in the user guide so it explicitly mentions which regridding schemes are lazy.

I did not add a section or mention of the UnstructeredNearest or PointInCell schemes to the user guide, because I would not feel qualified to write such a section and I also feel that it would be beyond the scope of this pull request.

bouweandela · 2020-09-03T15:38:10Z

@pp-mo I added a section in the Regridding chapter in the user guide and a note about whether a regridding scheme is lazy or not to all regridders I could find. Does that look better or is it too much? Is there anywhere else you would expect to see documentation of the feature?

pp-mo · 2020-09-07T13:18:46Z

Is there anywhere else you would expect to see documentation of the feature?

No, I think this looks great really.
It's true, in my view, that we could do with taking some overly deep information out of the User Guide, possibly into the "technical papers" section. It has grown over time, and I personally wish we had a more minimal User Guide like "what everyone should know".
But that's a bit of a separate project. In the meantime, what you're adding here is not really long enough to justify a technical paper, and is actually not out of keeping with the existing User Guide.

bouweandela · 2020-09-07T18:56:44Z

Thank you for reviewing again @pp-mo! If you require any further adjustments, please let me know.

bjlittle · 2020-09-08T08:55:28Z

@bouweandela We've just been experimenting with the GitHub actions/labeler to automatically label new pull requests, see #3819.

It appears to have failed for you PR (sorry about that!) which existed prior to #3819 being merged... so I was wondering whether you could rebase from master to pull in that PR to see if it makes a difference to the failure that we're seeing here in your PR.

Cheers! 😄

bouweandela · 2020-09-09T07:25:39Z

I was wondering whether you could rebase from master to pull in that PR to see if it makes a difference to the failure that we're seeing here in your PR.

@bjlittle It looks like you deleted the labeler action again from the master branch, is this question still relevant? I suspect that the action failed because of actions/labeler#36 (comment).

bjlittle · 2020-09-09T09:49:01Z

@bouweandela Apologies for the carnage... we were having major problems with that GitHub action. So much so that it was basically failing on all PRs, which is very not cool.

So we pulled it out, since it's not stable, sorry for the flip-flopping.

@pp-mo Otherwise, it's totally okay for this PR to be merged with the labeler action failing - don't let that be a barrier to this PR not being merged

pp-mo · 2020-09-09T10:55:22Z

I think this is really awesome work @bouweandela ! 👍 💐
The care for transparency and user documentation is really appreciated, too.

My only remaining problem is with the whatsnew file : since we have totally changed the whatsnew contribution scheme, merging this will re-introduce a 'contributions folder' + mess it up.

But rather than make you jump through more hoops, I will merge this and post a new PR to fix that.

pp-mo · 2020-09-09T11:02:15Z

I will merge this and post a new PR to fix that.

See #3834

* master: Rework whatsnew into new scheme. (SciTools#3834) Lazy regridding with Linear, Nearest, and AreaWeighted (SciTools#3701) Iris readme minimal (SciTools#3833)

* master: (36 commits) Rework whatsnew into new scheme. (SciTools#3834) Lazy regridding with Linear, Nearest, and AreaWeighted (SciTools#3701) Iris readme minimal (SciTools#3833) Fix sphinx logger (SciTools#3832) whatsnew entry for dependencies (SciTools#3831) Added link to scitools.org.uk on the main index. (SciTools#3830) Updated Iris install instructions (SciTools#3817) Delete labeler.yml Delete label.yml Create label.yml Delete labeler.yml Update labeler.yml Update labeler.yml Add template collapsible markdown section (SciTools#3823) Requirements re-haul (SciTools#3812) Automate pull request labels (SciTools#3819) move whatsnew contributions (SciTools#3816) Parse with packaging version (SciTools#3815) Add "What's New" entries for unpinning Cartopy, Matplotlib, Proj (SciTools#3811) tidy issue templates (SciTools#3814) ...

Lazy regridding with Linear, Nearest, AreaWeighted

127b930

trexfeathers added the Type: Enhancement label Aug 21, 2020

trexfeathers added this to the v3.1.0 milestone Aug 21, 2020

pp-mo added the Feature: ESMValTool label Aug 25, 2020

bouweandela force-pushed the lazy-regrid branch from 0f91103 to 127b930 Compare August 26, 2020 12:07

bouweandela added 2 commits August 26, 2020 17:20

Add tests

7ba9739

Add documentation

867b296

bouweandela marked this pull request as ready for review August 26, 2020 15:37

bouweandela mentioned this pull request Sep 2, 2020

AreaWeighted regrid requires a lot of memory #3808

Open

Add whatsnew message

c1851a0

pp-mo reviewed Sep 3, 2020

View reviewed changes

docs/iris/src/whatsnew/contributions_3.0.0/newfeature_2020-Sep-03_lazy_regridding.txt Outdated Show resolved Hide resolved

bouweandela and others added 2 commits September 3, 2020 13:49

Link to classes in whatsnew entry

26b2adb

Co-authored-by: Patrick Peglar <[email protected]>

Clarify laziness status and add example

56ba9b7

bouweandela commented Sep 3, 2020

View reviewed changes

Fix example output

b2a50a4

bouweandela added 2 commits September 7, 2020 20:28

Remove mention of experimental regridding schemes.

ef8206a

Clarify documentation

f524d9a

bjlittle modified the milestones: v3.1.0, v3.0.0 Sep 8, 2020

bjlittle assigned pp-mo Sep 8, 2020

pp-mo merged commit 882889c into SciTools:master Sep 9, 2020

pp-mo mentioned this pull request Sep 9, 2020

Rework whatsnew into new scheme. #3834

Merged

bouweandela deleted the lazy-regrid branch September 9, 2020 11:29

Peter9192 mentioned this pull request Sep 15, 2020

Making the multi-model statistics preprocessor lazy ESMValGroup/ESMValCore#781

Closed

rcomer mentioned this pull request Oct 2, 2020

Lazy Percentile Aggregator #3901

Merged

stephenworsley mentioned this pull request Nov 23, 2020

Sparse array approach SciTools/iris-esmf-regrid#23

Closed

stephenworsley mentioned this pull request Jun 2, 2021

Support lazy regridding SciTools/iris-esmf-regrid#80

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lazy regridding with Linear, Nearest, and AreaWeighted #3701

Lazy regridding with Linear, Nearest, and AreaWeighted #3701

bouweandela commented Apr 28, 2020 •

edited

Loading

stephenworsley commented May 11, 2020

JaroCamphuijsen commented Jul 29, 2020

trexfeathers commented Aug 21, 2020

bjlittle commented Aug 23, 2020

trexfeathers commented Aug 24, 2020

bjlittle commented Aug 24, 2020

bjlittle commented Aug 24, 2020 •

edited

Loading

bouweandela commented Aug 26, 2020

pp-mo commented Sep 2, 2020

bouweandela commented Sep 3, 2020

pp-mo commented Sep 3, 2020 •

edited

Loading

bouweandela Sep 3, 2020

pp-mo Sep 7, 2020 •

edited

Loading

pp-mo Sep 7, 2020

bouweandela Sep 7, 2020

bouweandela Sep 7, 2020

bouweandela commented Sep 3, 2020

pp-mo commented Sep 7, 2020 •

edited

Loading

bouweandela commented Sep 7, 2020

bjlittle commented Sep 8, 2020

bouweandela commented Sep 9, 2020

bjlittle commented Sep 9, 2020 •

edited

Loading

pp-mo commented Sep 9, 2020

pp-mo commented Sep 9, 2020

Lazy regridding with Linear, Nearest, and AreaWeighted #3701

Lazy regridding with Linear, Nearest, and AreaWeighted #3701

Conversation

bouweandela commented Apr 28, 2020 • edited Loading

stephenworsley commented May 11, 2020

JaroCamphuijsen commented Jul 29, 2020

trexfeathers commented Aug 21, 2020

bjlittle commented Aug 23, 2020

trexfeathers commented Aug 24, 2020

bjlittle commented Aug 24, 2020

bjlittle commented Aug 24, 2020 • edited Loading

bouweandela commented Aug 26, 2020

pp-mo commented Sep 2, 2020

bouweandela commented Sep 3, 2020

pp-mo commented Sep 3, 2020 • edited Loading

bouweandela Sep 3, 2020

Choose a reason for hiding this comment

pp-mo Sep 7, 2020 • edited Loading

Choose a reason for hiding this comment

pp-mo Sep 7, 2020

Choose a reason for hiding this comment

bouweandela Sep 7, 2020

Choose a reason for hiding this comment

bouweandela Sep 7, 2020

Choose a reason for hiding this comment

bouweandela commented Sep 3, 2020

pp-mo commented Sep 7, 2020 • edited Loading

bouweandela commented Sep 7, 2020

bjlittle commented Sep 8, 2020

bouweandela commented Sep 9, 2020

bjlittle commented Sep 9, 2020 • edited Loading

pp-mo commented Sep 9, 2020

pp-mo commented Sep 9, 2020

bouweandela commented Apr 28, 2020 •

edited

Loading

bjlittle commented Aug 24, 2020 •

edited

Loading

pp-mo commented Sep 3, 2020 •

edited

Loading

pp-mo Sep 7, 2020 •

edited

Loading

pp-mo commented Sep 7, 2020 •

edited

Loading

bjlittle commented Sep 9, 2020 •

edited

Loading