read-write Xarray with nc-dataset adapter. #4835

pp-mo · 2022-06-28T11:28:20Z

Targets feature branch, just to unify testing.
This introduces a "netcdf dataset adapter" for xarray datasets, enabling to read/write xarray data as if it were a netcdf file.
ALSO combined here (and not really separate for now):

a new loading "scheme" enabling iris to load from odd objects in place of file-specs:
changes enabling the netcdf file format to read from an open dataset (instead of filepath)
changes enabling the netcdf file format to save to an open dataset (instead of filepath)

If this idea flies, it might be worth separating out the core-iris changes into a separate PR rather than doing it all at once?

Hence "WIP"

~~Also, various TO-DOs noted on this old "private PR" : pp-mo#72~~
(update: all done)

pp-mo · 2022-06-28T13:08:36Z

Status update:
Almost there with CI.
Tests are functioning.
Now just some docs-build errors.

jamesp · 2022-06-29T07:49:27Z

lib/iris/fileformats/netcdf.py

-                raise IOError(msg)
-            else:
-                raise
+        if hasattr(filename, "createVariable"):


Is it worth considering a runtime protocol to define, in code, what you mean by "supporting the netCF4.Dataset api"?

https://docs.python.org/3/library/typing.html#typing.runtime_checkable. Added in Python 3.8.

jamesp · 2022-06-29T07:49:47Z

lib/iris/fileformats/netcdf.py

@@ -1061,6 +1064,7 @@ def __init__(self, filename, netcdf_format):

        * filename (string):
            Name of the netCDF file to save the cube.
+            OR a writeable object supporting the netCF4.Dataset api.


Suggested change

OR a writeable object supporting the netCF4.Dataset api.

OR a writeable object supporting the netCDF4.Dataset api.

jamesp

Not a complete review, just some questions that popped out at me as I read the code.

This is SUPER cool :D

jamesp · 2022-06-29T08:09:18Z

lib/iris/experimental/xarray_dataset_wrapper.py

+class _XrMimic:
+    """
+    An netcdf object "mimic" wrapped around an xarray object, which will be
+    either a dim, var or dataset.


By dim do you mean DimCoord, in iris parlance?

might be good to be explicit here: var I assume is xr.Variable, dataset is xr.Dataset. dim is an object that doesn't exist in xarray, but you need it for netcdf compat?

jamesp · 2022-06-29T08:10:57Z

lib/iris/experimental/xarray_dataset_wrapper.py

+
+    def __init__(self, xr, name=None, group=None):
+        """
+        Create a mimic object wrapping a :class:`nco.Ncobj` component.


What is nco.Ncobj? Old init string text?

jamesp · 2022-06-29T08:16:29Z

lib/iris/experimental/xarray_dataset_wrapper.py

+
+    def __getitem__(self, keys):
+        if self.ndim == 0:
+            return self._xr.data


keys is ununsed in this branch of the if. Is that right? Should keys have default value? Document with a comment either way I think.

jamesp · 2022-06-29T08:18:55Z

lib/iris/experimental/xarray_dataset_wrapper.py

+        Only a limited subset of the :mod:`netCDF4` APIs are currently
+        supported : just enough to allow Iris to read and write xarray datasets
+        in place of netcdf files.


Again, this might be where a runtime protocol could help document which parts of the api are implemented so far.

pp-mo · 2022-06-29T16:16:01Z

Lazy xarray output

Here's an interesting follow-on possibility ... pp-mo#73
(keeping this feature separate for now, just to simplify the discussion).

without this, iris "streams" all data into the netcdf variables, as it would to a file,
so everything gets realised and stored in memory.

pp-mo · 2022-07-01T10:31:05Z

Important Context : things this can be used for ?

In addition to just "fixing data a bit" on either load or save,
I think this might possibly help with a number of outstanding "desired features" (all relating to netcdf) :

load chunks control : More control over lazy data creation (chunking) #3333
precise formatting round-trips : Round-trip netcdf data #4215
lazy saving : Support lazy saving #4190
output "append" mode : Add functionality to allow appending cubes to existing netcdf file #565

Generally, all these issues may be better handled in Xarray, since they are close to the file representation -- so a solution there can work with specific variables+dimensions, rather than Iris CF-representing objects.

bjlittle · 2022-07-25T15:03:16Z

Ping @dcherian

Just thought that I'd make you aware of this PR.

It's early days, but you may be able to offer some perspective on this...

pp-mo · 2022-07-25T16:05:28Z

Ping @dcherian

Thanks for making the link @bjlittle

FWIW some other comments on this that you might also like to consider ..

The current approach here is rather untidy, and grew out of an attempt to create a simple 'wrapper' object to an existing xarray dataset. But that basic approach actually fails once write support is added, since it is not possible to modify the dimensions of an existing xarray dataset, and the result is now a rather clunky "mixed" approach.

The principal limitation of this as it is, is that it expects an xarray-dataset which does not intpret cf/coords/time-data
- which is why those are all turned off in the test example.
Likewise, the xarray that you get out of the 'to_array' method will also have shortcomings in those respects + not match a "normal" load from file (e.g. in terms of time data).
[ update, 2022-08-18 :

in fact, we maybe should also be using "mask_and_scale=False" in the loading too
as I think @TomekTrzeciak also pointed out, this particular xarray encoding aspect is (perhaps uniquely?) non-reversible,
- .. in that, it loses information by (potentially) mixing up NaN values and masked points (i.e. any occurrence of fill-value)
however, the other data transformations (which this disables) definitely all can in principle be "undone" by design (the decode+encode methods)
So there is a valid route to removing those interpretations from a 'regular' dataset, as is done on saving.

]

If this "load/save xarray as a netcdf dataset" approach has some general utility, then we should put it into xarray (not Iris!),
and work on those problems, which I believe is entirely feasibble.

dcherian · 2022-07-25T17:54:59Z

Thanks @bjlittle

Is there an issue I can read about the motivation for this?

If it is to take advantage of Xarray's read/write backend options, would it be easier to use the to_iris method?

pp-mo · 2022-07-26T13:03:38Z

Thanks @bjlittle

Is there an issue I can read about the motivation for this?

You're quite right, there should probably be an issue.
I will work towards that ...
For now, I can basically say that the ideal goal is ~lossless conversion between iris, xarray and netcdf files.

If it is to take advantage of Xarray's read/write backend options, would it be easier to use the [to_iris method]

It seems to me that a big "problem" with xarray to_iris/from_iris is that it requires embedding knowledge of Iris into xarray. And also, effectively, knowledge of CF, and what parts of CF Iris supports -- all of which is really Iris' business.

Meanwhile, there is a reasonable argument that an iris.to_array/from_xarray would be a better approach, since what Iris does is mostly "extend" xarray capabilities by adding CF handling.
But that then causes essentially the same problems of injecting xarray-specifics into Iris.

And with both those solutions, there is a particular problem regarding dependencies and testing, especially for C.I. :

in order to test an xarray to/from-iris, the xarray CI needs to test against Iris, which is a costly dependency
and exactly the same is true for the "converse solution" (iris to/from_xarray)

However, this kind of "dataset adapter" approach only depends on knowledge of xarray and netCDF4.

so, as it has no Iris-specific knowledge, it really should not be here (**)
- it could properly belong in xarray,
- or in a separate package
and it could then be used by any other packages wanting to establish file-less linkage to xarray data
- does that seem feasible ?

(**) but we still need certain changes in Iris :

support reading/writing a netcdf-dataset (or similar) instead of a filepath
pass through lazy data on write, instead of realise+store

Hence, code here is really just a PoC, showing how those solutions can work together

pp-mo · 2022-10-12T15:41:15Z

I have now broken this by advancing the F-B with a mergeback.
But that's ok, I am now rationalising this work to separate concerns a bit.

So the "let iris read+write nc datasets" part is now up, in #5024,
other PRs to follow, against the updated feature-branch.

pp-mo added 3 commits June 28, 2022 12:19

Attempt to support data loading - WIP

b40166f

Add read-capable xarray wrapper.

8153e9f

Support write operations : tested equivalent to direct iris.save.

0137be4

pp-mo mentioned this pull request Jun 28, 2022

Xarray read/write via netCDF4 dataset api adapter. pp-mo/iris#72

Closed

pp-mo added 2 commits June 28, 2022 13:46

New lockfile since adding xarray.

af27d59

Testing fixes.

c07ad44

pp-mo force-pushed the nc_dataset_load branch from 5d21804 to c07ad44 Compare June 28, 2022 12:52

Fix docstrings.

09f07ff

pp-mo force-pushed the nc_dataset_load branch from b7eecb8 to 09f07ff Compare June 28, 2022 17:38

Small improvements.

0038aa9

jamesp reviewed Jun 29, 2022

View reviewed changes

pp-mo mentioned this pull request Jun 29, 2022

Support lazy saving #4190

Closed

pp-mo mentioned this pull request Aug 25, 2022

Add functionality to allow appending cubes to existing netcdf file #565

Closed

pp-mo mentioned this pull request Sep 27, 2022

Xarray bridge #4994

Closed

This was referenced Oct 12, 2022

mergeback from main #5023

Merged

Load+save things other than filepaths, and specifically support netCDF4.Dataset-s #5024

Closed

pp-mo closed this Oct 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

read-write Xarray with nc-dataset adapter. #4835

read-write Xarray with nc-dataset adapter. #4835

pp-mo commented Jun 28, 2022 •

edited

Loading

pp-mo commented Jun 28, 2022

jamesp Jun 29, 2022

jamesp Jun 29, 2022

jamesp left a comment

jamesp Jun 29, 2022

jamesp Jun 29, 2022

jamesp Jun 29, 2022

jamesp Jun 29, 2022

jamesp Jun 29, 2022

pp-mo commented Jun 29, 2022 •

edited

Loading

pp-mo commented Jul 1, 2022 •

edited

Loading

bjlittle commented Jul 25, 2022 •

edited

Loading

pp-mo commented Jul 25, 2022 •

edited

Loading

dcherian commented Jul 25, 2022

pp-mo commented Jul 26, 2022 •

edited

Loading

pp-mo commented Oct 12, 2022

	OR a writeable object supporting the netCF4.Dataset api.
	OR a writeable object supporting the netCDF4.Dataset api.

read-write Xarray with nc-dataset adapter. #4835

read-write Xarray with nc-dataset adapter. #4835

Conversation

pp-mo commented Jun 28, 2022 • edited Loading

pp-mo commented Jun 28, 2022

jamesp Jun 29, 2022

Choose a reason for hiding this comment

jamesp Jun 29, 2022

Choose a reason for hiding this comment

jamesp left a comment

Choose a reason for hiding this comment

jamesp Jun 29, 2022

Choose a reason for hiding this comment

jamesp Jun 29, 2022

Choose a reason for hiding this comment

jamesp Jun 29, 2022

Choose a reason for hiding this comment

jamesp Jun 29, 2022

Choose a reason for hiding this comment

jamesp Jun 29, 2022

Choose a reason for hiding this comment

pp-mo commented Jun 29, 2022 • edited Loading

Lazy xarray output

pp-mo commented Jul 1, 2022 • edited Loading

Important Context : things this can be used for ?

bjlittle commented Jul 25, 2022 • edited Loading

pp-mo commented Jul 25, 2022 • edited Loading

dcherian commented Jul 25, 2022

pp-mo commented Jul 26, 2022 • edited Loading

pp-mo commented Oct 12, 2022

pp-mo commented Jun 28, 2022 •

edited

Loading

pp-mo commented Jun 29, 2022 •

edited

Loading

pp-mo commented Jul 1, 2022 •

edited

Loading

bjlittle commented Jul 25, 2022 •

edited

Loading

pp-mo commented Jul 25, 2022 •

edited

Loading

pp-mo commented Jul 26, 2022 •

edited

Loading