Add module for computing river flood footprints from GloFAS river discharge data

[peanutfun]

The module includes a data pipeline which automatically downloads GloFAS river discharge data and transforms it to flood footprints, which in turn can be transfered toHazard or RiverFlood objects.

The module introduces a few new dependencies. They have to be installed into the Jenkins test environment for the builds to succeed:

• dantro for the data pipeline
• cdsapi for downloading data from the Copernicus Data Store
• ruamel.yaml for reading YAML files. This one is already a dependency of dantro.

Major changes in CLIMADA Petals:

• Add the climada_petals.hazard.rf_glofas subpackage
• Add climada_petals.util.cds_glofas_downloader utility functions
• Add module documentation

To do:

• Tutorial
• FLOPROS database integration
• Some more tests
• Fix linter issues

[peanutfun]

@emanuel-schmid Could you have a look at why the new dependencies are not found in the checks?

[peanutfun]

@tovogt Thanks again for the thorough review! To comment on your overall thoughts:

It's really unfortunate that data needs to be written uncompressed first due to performance issues.

Yes, but this is what I came up with after months of using the module. I don't think I can do much better without further help. In my experience, using zlib is indeed horribly slow with dask, and it also does not properly support multi-process writing to a single file. You suggest calling compute first and then saving the data. However, this requires the entire data to fit into memory in the first place. Depending on the country and time frame you want to compute flood footprints for, this is not feasible given the usual memory space of a personal computer.

However, my module actually gives users the option to optimize this themselves. Setting store_intermediates to False and executing each step of RiverFloodComputation.compute themselves, they are free to call compute and store data as they see fit. I tried to find default settings that work no-matter-what. The main restriction now is drive space. You might end up writing hundreds of GB, but it actually is quite performant (with a modern SSD)

In the tutorial notebook, there are some occurrences of py:func which are not translated properly by Sphinx.

The Myst parser should support these types of references, see https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html#reference-roles

Now that I look at the doc/conf.py, we might actually still use the old nbsphinx parser for reading the notebooks. I'll try to fix that.

[tovogt]

Yes, but this is what I came up with after months of using the module. I don't think I can do much better without further help. In my experience, using zlib is indeed horribly slow with dask, and it also does not properly support multi-process writing to a single file. You suggest calling compute first and then saving the data. However, this requires the entire data to fit into memory in the first place. Depending on the country and time frame you want to compute flood footprints for, this is not feasible given the usual memory space of a personal computer.

I think it's desirable to require each individual NetCDF file to contain at most as much data as could potentially fit into (a reasonable amount of) memory. Ideally, I would even propose to have at most 4 GB of (uncompressed) data per NetCDF file. With NetCDFs, it is very easy to just split up data into several files, and then load the dataset as a multi-file-dataset, e.g. using xr.open_mfdataset. If you adhere to that, you can very well call compute for each chunk of data that's supposed to end up in an individual file. It's much faster, the data is very easy to handle, and there are practically no disadvantages. Monolithic, long-running processes that produce monolithic chunks of data, are extremely inconvenient in almost every environment. For all projects I work with, I try to split up everything into a high number of slim processes with short run times that produce comparably small chunks of data each - and it's much more convenient under almost all circumstances I can think of.

However, my module actually gives users the option to optimize this themselves. Setting store_intermediates to False and executing each step of RiverFloodComputation.compute themselves, they are free to call compute and store data as they see fit. I tried to find default settings that work no-matter-what. The main restriction now is drive space. You might end up writing hundreds of GB, but it actually is quite performant (with a modern SSD)

As I said, this is not a merge-blocker from my side, and I'm happy to go with this solution.

[tovogt]

You are using xesmf to regrid a raster with bilinear interpolation. Why don't you use rasterio.warp.reproject for that? It would avoid introducing a new dependency, it's really very powerful, and it's already used in several other places in CLIMADA.

[peanutfun]

Why don't you use rasterio.warp.reproject for that?

Simply because I am not familiar with it. I first used the xarray-internal interpolate, which is horribly slow and does not take geospatial information into account. So I switched to xesmf because it was recommended to me and is simple to use for xarray data structures. But it is also difficult to install, and poses an issue on Euler. So I would be very happy about a suggestion how to drop it and switch to another implementation, given that it is not much slower.

[tovogt]

Okay, after a closer look, I think you won't be able to have anything similar to nearest_s2d extrapolation in rasterio or similarly basic packages.

[tovogt]

Thanks @ThomasRoosli for the final cleanup. This is ready to be merged from my side.