Making the multi-model statistics preprocessor lazy

[Peter9192]

The multi-model statistics preprocessor is currently sub-optimal. It operates on Numpy arrays because Iris cubes are quite strict in checking inconsistencies between the input cubes. This workaround impedes lazy evaluation and introduces some vulnerabilites (e.g. #267, #665) and a trail of ad-hoc fixes.

Inconsistent time coordinates were probably the biggest obstacle. However, starting from #677, and continuing in #685, we have already drastically reduced the complexity of time handling. Additionally, #744 proposes to move everything related to 'homogenizing' the time coordinate to the regrid_time preprocessor altogether. Moreover, Iris will soon support lazy regridding (SciTools/iris#3701), so in that case we could really benefit from lazy evaluation in multi-model statistics as well.

In #673, we have implemented a new function that computes 'multi-cube' statistics using Iris build-in functions. This already works for ensemble statistics, because in that case, the cubes are already very homogeneous.

It would be really nice if we could use this Iris 'engine' for the regular multi-model statistics preprocessor as well.

Would you be able to help out?
I might have some time, but it would be good to wait for #685, #744 and #673. In the meantime, this is a placeholder to keep the helicopter view of everything that's happening with the multi-model statistics preprocessor.

[stefsmeets]

Also related to #674

[valeriupredoi]

good call! we need to liaise with iris folk - @bjlittle - because a basic multimodel statistic (and lazy altogether) would best sit in iris. We can do it here first but migration to iris is something both us and them would benefit from 🍺

[Peter9192]

So what we did in #673 is basically: iris.cube.cubeList(...).merge_cube().collapse() with some fiddling with auxcoords and equalizing attributes. I'm not sure if that warrants a dedicated function in iris or whether it's just a good application of existing functionality. So indeed, I'm curious what @bjlittle has to say about that.

[valeriupredoi]

no, I'm talking about a dedicated iris.analysis.multimodel_statistics(args=(cubes, statistic, ...), **kwargs) function

[Peter9192]

I know 🍺. But what I'm saying is that you could also see it as a use case that is already supported by existing functionality in iris. So I'm not sure if really needs to be a dedicated function in iris. But of course I would be happy to see it migrate there eventually.

Btw, I would prefer to name it multi-cube statistics (like I did in #673), because iris is all about cubes, and different cubes are not necessarily different models.

[valeriupredoi]

I need to have a look at #673 👍

[stefsmeets]

We are working on better tests for the multimodel statistics here: #856
Might want to keep an eye on that, as I'm sure this will help with developing the code for this issue.

[Peter9192]

#685 is merged!

[schlunma]

Moving this to v2.6 to follow the corresponding PR.