collaborate with xugrid project?

[rabernat]

Thanks to @clyne for the great talk at the Pangeo Showcase yesterday.

Following up on that, I thought I'd help make contact between this project and the xugrid project:

• https://deltares.github.io/xugrid/
• https://github.com/Deltares/xugrid

The project was started by @Huite and includes a few other contributors such as @visr, who originally brought it to my attention.

The scope of xugrid is as follows:

Xarray extension to work with 2D unstructured grids, for data and topology stored according to UGRID conventions.

Compare this to uxarray

Uxarray will provide Xarray styled funtions to better read in and use unstructured grid datasets that follow UGRID conventions.

These scopes are nearly identical.

Hoping to start a conversation about how these teams could potentially work together.

[rajeeja]

Indeed, that would be good. I tried to clone and install their suite. It seems they are too tied to windows OS and constantly pushing changes.

…

________________________________ From: Ryan Abernathey ***@***.***> Sent: Thursday, November 11, 2021, 12:03 PM To: UXARRAY/uxarray Cc: Subscribed Subject: [UXARRAY/uxarray] collaborate with xugrid project? (Issue #4) Thanks to @clyne<https://github.com/clyne> for the great talk at the Pangeo Showcase yesterday. Following up on that, I thought I'd help make contact between this project and the xugrid project: * https://deltares.github.io/xugrid/ * https://github.com/Deltares/xugrid The project was started by @Huite<https://github.com/Huite> and includes a few other contributors such as @visr<https://github.com/visr>, who originally brought it to my attention. The scope of xugrid is as follows: Xarray extension to work with 2D unstructured grids, for data and topology stored according to UGRID conventions. Compare this to uxarray Uxarray will provide Xarray styled funtions to better read in and use unstructured grid datasets that follow UGRID conventions. These scopes are nearly identical. Hoping to start a conversation about how these teams could potentially work together. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#4>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AALF6AR6KBITZT724VBZBRTULQAOHANCNFSM5H3BAXJQ>.

[Huite]

Hi,

@rabernat Thanks for opening this.
I'd have liked to watch the Pangeo showcase live, but it was slightly late (22:00 local time for me) and (despite vaccinations) I've got a mild case of COVID-19 to contend with. I'm sure I'll be able to check it out sooner rather than later.

@rajeeja Yeah, I haven't gotten to wrapping things up neatly yet, as the README says, it's still very much a work in progress. Haven't managed to get it to pip yet either, and I have a lot of tests to write still -- I've been mostly figuring out API stuff. I think you'd be best served best by going through some of the built docs to get a high level impression: https://deltares.github.io/xugrid/

Background

To give a bit of background, I'm a geohydrologist at the Groundwater Department of Deltares The impulse for this work on unstructured grids has been the release of MODFLOW6 by the USGS: https://www.usgs.gov/software/modflow-6-usgs-modular-hydrologic-model
The MODFLOW versions we worked with before were all structured, and so both input and output data fit right with the (default) netCDF4 and Xarray data structures. I would obviously like to make working with unstructured data as easily as it is with xarray for structured data. I also would like to serve a broader goal, as there are many more potential users in Deltares, since Deltares uses unstructured hydrological models as well (network1d, mesh2d, layered 3d).

Relevant material

For the record, this where most of my previous discussion can be found:

pydata/xarray#4222
https://github.com/NOAA-ORR-ERD/gridded/issues

In general, I ended up going with the approach sketched here for now:
xgcm/xgcm#225

This does means there's a fair amount of forwarding to do. As far as I can tell, the indexes in xarray are not yet flexible enough to do the stuff I'd like to with "default" xarray accessors. I like the accessor approach -- so far -- because I can tell my colleagues to simply write .ugrid whenever they want to do something with the unstructured topology.

Also a number of notes here:
https://github.com/xarray-contrib/xoak/issues

Cooperation

In general: I'd be very happy with an open source community project. I have no demands with regards to name or branding.

Here's a number of questions that come up for me:

• Is there room for Ugrid "1D" Network topology? These networks are very useful in hydrological modeling for streams, channels, etc, and much "unstructured logic" applies.
• I would assume the priority lies with 2D mesh topology and layered 3D for now. Also because I've never seen a fully unstructured 3D UGRID file the wild. The spatial index I'm currently using is 2D, but can be easily extended to 3D: https://github.com/deltares/numba_celltree (and it's based on the NOAA one: https://github.com/NOAA-ORR-ERD/cell_tree2d)
• UGRID supports data on the faces, the edges, and the nodes. From a typical control volume finite difference perspective, this means nodes are lower in priority, but it might not really matter.
• Not everything can be written performantly with just numpy and scipy. What are acceptable (binary) dependencies to you? (That meshkernel that is a currently Windows-only dependency I believe should be an optional dependency at any rate.)

I'm still very much figuring things out -- hence the lack of tests. numba-celltree should have decent coverage though and it's on pip. I do feel like doing some benchmarking on the spatial indexes would be rather useful.

I do however have a number of obligations to fulfill within my organization, I need to deliver some features on a fairly short time scale. It seems likely to me your roadmap is somewhat different (and can afford to be more ambitious). I don't think should be an issue: I would be happy to contribute, but it also seems likely that at least on the short term I'd need to keep my own thing in the air (read: xugrid). As time progresses and this package (uxarray) matures, I could start depending on it more and more, slowly disassembling xugrid until it's finally just a shell around uxarray.

[erogluorhan]

Thanks @rabernat for bringing this to attention! Indeed, @Huite 's xarray # 4222 has already been an inspiration and was much helpful. Thanks a lot for sharing your thoughts. plans in much detail, @Huite ! (I will read through your comments in detail and share my thoughts soon)

The fact that both Raijin and SEATS groups are funded and they are committed to open-development is a great opportunity to make significant progress on this long-standing unstructured grids needs in Python ecosystem. We seem to have a good pace currently and are currently working on initial API decisions for uxarray. I believe we'd see the initial implementation in a few weeks here.

[erogluorhan]

Hi @Huite ,

As we are now getting closer to a point where UXarray is having first significant implementations done and could make an official Conda release soon, I'd like to both give a heads up and reply to your prior questions as much as I can.

UXarray Status:

Currently, we are almost finished with implementing read/write functions for not only UGRID conventions but also Exodus and SCRIP file formats in UXarray. Simultaneously, we are working on global integral - face area calculation, as the first example of analysis operators in Uxarray. We are planning to include all of these functions in an official release soon (end of April or a bit later, after performing integrated tests and bug-fixes).

We feel like we should now start prioritizing the analysis operators to be implemented. We have already identified the draft Uxarray API to have a tentative comprehensive list of analysis functions, which is fully open to community feedback to get modified/finalized! We will soon share a UXarray roadmap that shows our priorities with this draft API, which will also be open to community requests/feedback.

Your Questions/Points:

I would assume the priority lies with 2D mesh topology and layered 3D for now. Also because I've never seen a fully unstructured 3D UGRID file the wild.

Yes, exactly, we are on the same page.

Is there room for Ugrid "1D" Network topology? These networks are very useful in hydrological modeling for streams, channels, etc, and much "unstructured logic" applies.

Yes, UXarray should support 1D network topology.

Not everything can be written performantly with just numpy and scipy. What are acceptable (binary) dependencies to you? (That meshkernel that is a currently Windows-only dependency I believe should be an optional dependency at any rate.)

This is a great question, and answer will depend on future explorations for sure, but, for now, I believe Dask is a must to comply with for parallelization purposes (naturally as we extend Xarray). Other than that, numba could be quite useful to speed up the things especially when parallelization is not the case. Dependencies other than these need to be discussed case by case I think. Any Windows-only dependency would be least favorable.

I do however have a number of obligations to fulfill within my organization, I need to deliver some features on a fairly short time scale. It seems likely to me your roadmap is somewhat different (and can afford to be more ambitious). I don't think should be an issue: I would be happy to contribute, but it also seems likely that at least on the short term I'd need to keep my own thing in the air (read: xugrid). As time progresses and this package (uxarray) matures, I could start depending on it more and more, slowly disassembling xugrid until it's finally just a shell around uxarray.

Thanks a lot! UXarray has been created as the product of a collaboration between Projects Raijin and SEATS, which were separately funded by NSF and DOE, respectively (You might have already seen in our README). Even though each project has its own definition, deadlines, timeline, etc., the fact that both groups aim at providing the geoscience community with open-source unstructured grid tools in Python ecosystem has brought us together. That said, we would be very happy to collaborate with you and other projects to any extent (at anytime) for the good of community.